Local LLM on your computer: where to start

First choose one use for your local LLM

Write down a few things you want to do regularly. For tidying short notes, language quality and convenience matter. For programming you add tools, the terminal and access to the repository. For scans you need a model that understands images, or a separate OCR step.

Your own files can be connected through a RAG search. Downloading a model does not mean it knows the documents saved on your disk. The app has to read them and choose the passages for the answer. Settle from the start which folders it should not open.

What to check on the computer

On Apple Silicon the memory is shared, so we do not carry VRAM requirements over directly from NVIDIA cards. Running on a CPU is also possible for supported models, but you need to check the response time you can expect. Do not buy a computer on the number of model parameters alone.

How to choose your first local model

Start with a variant that fits with room to spare. Qwen, Gemma and Polish models come in different sizes and uses. The list of options will differ for a computer with little memory and for a workstation with a large graphics card.

Quantisation reduces the memory needed for the weights by storing them at lower precision. Check the answers, though: saving memory can change the model's behaviour. Our write-up of Q4, Q5 and W4A16 shows the differences in a specific experiment with Qwen.

Tools such as Ollama make it simpler to run supported models. For a more demanding installation you can use a different engine. The chat app, the server and the weights file are separate parts that have to fit together.

Does local mean offline?

A model can run without sending prompts to an outside provider if the computation and the components it needs work on site. The first download of the files usually needs the internet. Later web searches, cloud features of the app or history sync can still send data.

In the configuration we check the actual route the data takes. The model server does not need to be exposed to the public internet. When a phone or a second computer should use it, we set up private access and sign-in. The guide to remote access helps here.

If you are only planning to buy

Set the budget together with the task and the waiting time you can accept. A rented GPU can help you test a larger model before you buy. A smaller variant, on the other hand, may be enough on your current computer. These two trials reduce the risk of spending money on a configuration that does not fit the work.

Syntalith helps you run a local LLM: we choose the hardware and the model, install the tools, check privacy and show you how to operate it. Send us the operating system, the amount of memory, the graphics card model and a description of the use. If you do not have the hardware yet, start with the description alone.

What a local LLM costs and when it pays back

Where to start

In the free process review we choose one task and 20 to 100 documents the team has already worked on, and within 2 working days you get a written summary with a price. We roll out the local model as part of private LLMs, and the app around it as an AI app.