Why You Need a Baseline Before Fine-Tuning a Local LLM

Fine-tuning can make a small open model dramatically better at a narrow task. It can also produce a model that has lower training loss, looks convincing in a few hand-picked prompts, and is worse where it matters.

The difference is usually decided before training starts.

A baseline is the recorded performance of the unmodified model on the same task, dataset boundary, prompt template, decoding settings, and evaluation logic that will later judge the candidate. Without it, “the model improved” has no stable meaning.

The baseline is part of the experiment, not a preliminary demo

Suppose the task is extracting a typed JSON record from an invoice. A useful evaluation might include:

• valid JSON rate;
• required-field coverage;
• exact or normalized field accuracy;
• error rate for high-cost fields such as totals and dates;
• latency and peak memory on the target Mac;
• failure cases for long, noisy, or incomplete inputs.

Run that suite against the pinned base model first. Record the model revision, tokenizer, chat template, prompt, decoding parameters, dataset revision, hardware, software environment, and random seed where relevant.

That record becomes the comparison point. A later adapter should not be compared against a memory of how the base model felt.

Use train, validation, and holdout for different decisions

The split names are not bookkeeping. They separate distinct kinds of evidence:

The Hugging Face evaluation guidance distinguishes training, validation, and test responsibilities and warns that evaluating on training data can misrepresent generalization. Tensor Cortex uses “holdout” for the final protected set because the name makes its operational role explicit: it is held outside training and tuning decisions.

If you inspect the holdout after every run and then change the learning rate, prompt, or dataset based on those results, it has become another validation set. Create a fresh holdout or stop treating the reported score as unbiased.

Evaluate the task, not only model loss

Loss is useful training telemetry. It is not the product outcome.

For structured extraction, task success may be schema validity and field accuracy. For classification, use per-class precision, recall, F1, and a cost-sensitive error view. For tool selection, measure the selected tool, argument-schema validity, and execution success. For style adaptation, use a bounded rubric with blinded comparison and agreement checks.

Perplexity can still help diagnose language-model behavior, but it should not replace the task metric. The official MLX-LM LoRA guide supports separate train, validation, and test data and exposes evaluation and resume paths. Those mechanics make an experiment possible; the task-specific acceptance protocol is still your responsibility.

Freeze more than the model

A fair comparison keeps the following fixed:

1. The evaluation examples and expected outputs.
2. The prompt and chat template.
3. The decoding configuration.
4. The normalization and scoring code.
5. The target hardware and measurement method for latency and memory.
6. The failure policy for invalid, empty, or truncated output.

Changing any of these may be a valid new experiment. It is not the same comparison.

Pin the base-model revision as well. A repository name that resolves to a newer tokenizer or weights next week can make a previously reproducible baseline impossible to recover.

Track aggregate gain and individual regressions

One average score can hide a costly failure.

A candidate might improve overall field accuracy while becoming worse on invoice totals. It might raise classification F1 while collapsing minority-class recall. It might improve valid JSON rate but exceed the memory budget of the target Mac.

Use at least three acceptance layers:

• Aggregate threshold: the primary task score improves by a predeclared amount.
• Regression guard: no critical field, class, or scenario degrades beyond its allowed tolerance.
• Deployment boundary: memory, latency, package size, and format validity remain acceptable on the target device.

Write these thresholds before reviewing the candidate. Post-hoc thresholds are explanations, not independent acceptance criteria.

A compact local fine-tuning protocol

Before training:

1. Define the task and the cost of important error types.
2. Pin the dataset and base-model revisions.
3. Create train, validation, protected holdout, and stress splits.
4. Run and store the base-model baseline.
5. Declare the aggregate, regression, and device acceptance gates.

During training:

1. Use training loss for optimization telemetry.
2. Use validation results for checkpoint and configuration choices.
3. Preserve configuration, environment, seed, and checkpoints.
4. Do not inspect the protected holdout.

After training:

1. Freeze the selected checkpoint and configuration.
2. Run the candidate on the unchanged baseline and holdout protocol.
3. Inspect aggregate gains and individual regressions.
4. Measure latency and memory on the target Mac.
5. Package only if all declared gates pass; otherwise record a narrow or failed result.

What a trustworthy result looks like

The output of a fine-tuning run should be more than an adapter file. It should include:

• the base model and immutable revision;
• dataset revisions and split provenance;
• training configuration and environment;
• baseline, validation, holdout, and stress results;
• hardware, memory, latency, and runtime measurements;
• accepted regressions and known weaknesses;
• the model or adapter package and its intended runtime.

That evidence makes a negative result useful. A failed candidate can show that the dataset was too small, the method was wrong, the target model lacked capacity, or the quality gain did not justify the deployment cost.

The purpose of the baseline is not to slow training down. It is to keep compute from producing confidence without knowledge.

Primary-source packet

Sources were reviewed on 2026-08-29. The official project and documentation sources below establish tool behavior and evaluation guidance; the experiment protocol and acceptance layers are Tensor Cortex Editorial synthesis.

1. MLX-LM: Fine-Tuning with LoRA or QLoRA — official training, data-split, evaluation, adapter, and resume behavior.
2. Hugging Face Evaluate: Considerations for model evaluation — official guidance on train, validation, and test responsibilities and evaluation pitfalls.
3. Hugging Face Hub: File names and splits — official dataset split conventions for train, validation, and test data.

Scoped claim ledger and closure

No customer, model-performance, hardware-performance, compatibility, certification, partnership, or availability claim is made.