Local LLMs for Work: Testing Tweaks and a Setup Guide

On October 30, 2026, an article was published where the author tested various tweaks for local LLMs (large language models run on your own hardware) and compared them with tips for cloud models. Conclusion: some recommendations for the cloud do not work locally. Below is a guide on how to replicate the tests and set up a local model for real work.

What You'll Need

• A computer with a GPU (graphics card) or Apple Silicon. For models with 7–8 billion parameters, 8 GB of video memory is enough. For 13–14 billion, you need 16 GB. Check the model card for exact requirements.
• Docker (container system) or Python 3.10+.
• The ollama utility (a tool for running local LLMs) or llama.cpp (an inference engine).
• A text editor and terminal.
• A dataset with your work tasks: 20–50 examples of "request — expected response". Without it, tests are meaningless.

Step 1. Install ollama and Download a Model

Ollama simplifies running models. Installation:

curl -fsSL https://ollama.com/install.sh | sh

After installation, download a model. For example, llama3.1:8b (8 billion parameters).

ollama pull llama3.1:8b

Check that the model responds:

ollama run llama3.1:8b "Say one word: test"

If you see a response, the base is ready.

Step 2. Establish a Baseline

Collect 10–20 work requests in a file prompts.jsonl (one JSON per line). Example:

{"prompt": "Shorten the text to 10 words: ...", "expected": "..."}

Run the model on each request and save the responses. You can use a Python script:

import json, subprocess

with open('prompts.jsonl') as f:
    for line in f:
        item = json.loads(line)
        result = subprocess.run(
            ['ollama', 'run', 'llama3.1:8b', item['prompt']],
            capture_output=True, text=True
        )
        print('PROMPT:', item['prompt'])
        print('OUTPUT:', result.stdout.strip())
        print('---')

Save the output to a file. This is your reference point.

Step 3. Check Temperature and top_p

Cloud tips often say "set temperature to 0.7 for creativity". For local models, this can produce garbage. Temperature (randomness parameter) 0.7 increases diversity, but on small models hallucinations become more frequent. Start with 0.2–0.3 for facts and code.

In ollama, parameters are set via API or Modelfile. Create a Modelfile:

FROM llama3.1:8b
PARAMETER temperature 0.2
PARAMETER top_p 0.9

Build your version:

ollama create mymodel -f Modelfile
ollama run mymodel "Your request"

Compare responses with the baseline. If it's better, keep it.

Step 4. Try Quantization (Weight Compression)

Quantization reduces model size and speeds up operation, but may lower quality. Ollama uses Q4_0 (4-bit compression) by default. For tests, variants with suffixes are available, e.g., llama3.1:8b-q8_0 (8-bit).

Download and compare:

ollama pull llama3.1:8b-q8_0
ollama run llama3.1:8b-q8_0 "Your request"

If the difference in responses is not critical and speed matters more, keep Q4. If quality drops, use Q8, but check if you have enough video memory.

Step 5. Set Up a System Prompt

A system prompt (instruction for the model) in local models is often ignored if it's too long. Keep it short: 1–3 sentences. Example for text work:

You are an editor's assistant. Answer concisely, without fluff. If you don't know, say "no data".

Add it to the Modelfile:

SYSTEM """You are an editor's assistant. Answer concisely, without fluff. If you don't know, say "no data"."""

Rebuild the model and check that the response style has changed.

Step 6. Limit the Context

Context (context window) is how many tokens the model remembers. Local models often have a smaller one than cloud models. If your request is longer than the limit, the model will truncate the beginning. In ollama, the num_ctx parameter is set in the Modelfile:

PARAMETER num_ctx 4096

Set a value that fits in memory. For 8 GB of video memory with an 8b model, 4096 tokens is reasonable. Check that long requests don't lose the beginning.

Step 7. Compare with a Cloud Tip

Take any recommendation for cloud LLMs (e.g., "use temperature 0.7 and top_p 0.95") and run it on a local model. Record the result. Often it turns out that lower values work better for local models. Don't take it on faith — test on your own data.

Common Mistakes and How to Identify Them

• The model produces incoherent text. Cause: temperature too high or context too small. Solution: lower temperature to 0.2, increase num_ctx.
• Responses repeat. Cause: low temperature and top_p. Raise top_p to 0.9–0.95.
• The model ignores the system prompt. Cause: prompt too long or model too small. Shorten the instruction to one sentence.
• Not enough video memory. Cause: model or context size too large. Use Q4 quantization or reduce num_ctx.
• Slow operation. Cause: model doesn't fit in GPU and goes to CPU. Check nvidia-smi (or "System Monitor" on Mac) — GPU load should be close to 100%.

All steps are verifiable. If a tweak didn't improve on your data, don't use it. There is no data that any tweak works universally.