Local AI: three machine profiles for several kinds of work
Published · updated
Running AI on your own hardware is no longer a specialist project. It’s an equipment decision, comparable to buying a file server. We’ve seen why internal AI may make sense. Here are three concrete configurations, with the prices you should expect and the conditions that decide what each machine can actually run.
The rule before the machines
Two numbers draw the first boundary:
- The amount of memory: it limits the size of the model. As an order of magnitude, the weights of a dense 8-billion-parameter model quantised to four bits need around 5–6 GB, while a 70B model needs around 40–45 GB. The runtime, the KV cache and the concurrent users need memory on top.
- Memory bandwidth: it sets the speed of generation. That’s what separates a three-second answer from a thirty-second one.
Compute, model architecture, quantisation, context, concurrency, runtime, CPU, storage and network can also dominate. So start from a measured workload, then test the model and the machine together.
Configuration 1: the network cabinet (municipality, accounting practice)
- Machine: Mac mini M4 Pro, 64 GB of unified memory
- Price: expect 2,000 to 2,500 francs in the 64 GB configuration
- Runs: quantised 30B-class models. One independent test reports 12–18 tokens per second in its configuration; reproduce the result with the model, the quantisation, the context and the runtime you actually select
- Where it lives: a shelf in the network cabinet you already have; quiet, frugal, no dedicated room
The job: for a town administration, an assistant that answers from the minutes, the regulations and the directives while citing its sources, plus local transcription of council meetings. For an accounting practice, extracting the invoice data before preparing the entries. In both cases the resident or client data can stay on site, provided the network, the logs, the telemetry, the updates, the administration and the backups genuinely stop it from leaving.
The limits are less about adapting the model than about concurrency without a dedicated GPU. The throughput may suit a small office team. But it won’t necessarily carry a public-facing service without queues or a fallback.
Configuration 2: the plant room (solar site, smart building)
- Machine: NVIDIA DGX Spark, 128 GB of unified memory
- Price: 4,699 dollars
- Runs: NVIDIA states inference support up to 200B parameters and fine-tuning up to 70B under compatible conditions; the useful throughput still depends on precision, context and concurrency
- Where it lives: the plant room, next to the building management system or the SCADA
The job: on a solar site or in an instrumented building, the machine receives a read-only copy of the data from the inverters and the meters and from the building management. It writes the operating reports. It describes the anomalies and prepares the maintenance work. It doesn’t sit in the control path: a broker or a DMZ keeps the AI away from the SCADA, with deterministic alarms and a human approval in front of anything that acts. And to run offline you also have to hold the models, the identity, the licences and the dependencies locally.
The limits: modest memory bandwidth for its class (the generation is decent, not spectacular) and an ARM ecosystem still young for some tools.
Configuration 3: the small rack (institute, laboratory, teaching)
- Machine: workstation or 4U server with an RTX PRO 6000 Blackwell: 96 GB of ECC GDDR7, 1,792 GB/s of bandwidth
- Price: a complete machine around 12,000 to 15,000 francs depending on the chassis
- Runs: the weights of a dense 70B model in FP8 may fit on one card, but the runtime, the context cache and the concurrent users eat the remaining memory; measure the actual workload
- Where it lives: a small ventilated rack; the card draws up to 600 W
The job: a research assistant that queries corpora of papers and archives while citing its sources, fine-tuning open models on a domain (including Apertus, the model from the Swiss federal institutes of technology), and a shared service for a whole institute. For teams that don’t need CUDA, the Mac Studio M3 Ultra is the quiet alternative.
Rent first, purchase afterwards
I start from a measured workload rather than from a specification sheet, then test the model and the machine together.
None of these machines should be bought on a hunch. You can rent a comparable GPU for a bounded pilot that measures quality, latency, concurrency and the effort to operate it. The price depends on the provider, the region, the storage and the traffic. Only then compare renting to buying over three years, with energy, integration, support, security, replacement, downtime and residual value in the sum. A payback of a few months isn’t automatic.
The benchmark record
For every machine, write down the model and its hash, the quantisation, the runtime and drivers, the input and output lengths, the time to first token, the throughput, the concurrent users, the peak memory, the energy, the noise, the cost and the recovery procedure. Without this record, two “tokens per second” figures can’t be compared.
Three machine profiles, one idea: size the AI to the work, and control the whole stack around the weights.
Where these prices come from
I took the prices and the specifications on 14 August 2026. The ranges aren’t quotations. Capacity, throughput and cost depend on the model, the precision, the context, the concurrency and the operations.