Back to Blog
Knowledge Management10 min read

When to Choose Fine-Tuning of Neural Networks over Dynamic Context Injection

Fine-tuning or RAG: how an enterprise should choose between changing the model and supplying context at query time, with research data and practical criteria.

O

Oleksandr Ivanchuk

Software Developer

A language model can be adapted to the needs of an enterprise in two fundamentally different ways: by changing the model itself through fine-tuning, or by leaving it unchanged and supplying the necessary information with every query. These approaches solve different problems, and a mistake in the choice costs months of work. In this article we examine what the research shows, which indicators make fine-tuning genuinely justified, and why in most enterprise scenarios it is not the first step.

Introduction

The request "train the model on our documents" is one of the most frequent we encounter at the start of projects. Behind it lies an understandable intuition: if a person can be taught, so can a model. However, this wording mixes two different needs. The first is for the model to know the facts of the enterprise: regulations, specifications, the history of decisions. The second is for the model to behave in a certain way: to adhere to a format, terminology, style, and classification logic.

For the first need, the principal tool today is dynamic context injection, above all Retrieval-Augmented Generation (RAG), in which relevant document fragments are found and added to the query at the moment it is executed [1]. For the second it is fine-tuning, that is, additional training of the model on examples, which changes its parameters.

We covered the fundamentals of RAG in the article on RAG technology for corporate documents. Here we focus on the opposite question: when this approach is insufficient and changing the model itself becomes a well-founded decision.

Two Ways of Adaptation: What Exactly Changes

The difference between the approaches lies in where the knowledge is stored.

Dynamic context leaves the model unchanged. The knowledge resides in an external store (a document base, a corporate system, the results of tool calls) and reaches the model only for the duration of a single query. Updating the knowledge means updating the store. The source of every answer can be cited.

Fine-tuning writes patterns directly into the model's parameters. After that the model needs no prompting to act in the required way, but what it has learned can be "forgotten" or updated only through a new training cycle. Modern methods, LoRA in particular, have made this process substantially cheaper: the authors of the method demonstrated a reduction in the number of trainable parameters by up to 10,000 times and in the GPU memory requirement by a factor of three compared with full fine-tuning [5]. However, the cost of computation is only part of the expense; the main part falls on data preparation and subsequent maintenance.

CriterionDynamic context (RAG, tools)Fine-tuning
What changesThe input data of the queryThe model's parameters
Knowledge updatesImmediately after the source is updatedA new cycle of training and validation
Source citationPossible for every answerImpossible: the knowledge is dissolved in the parameters
Access controlAt the level of documents and usersThe model knows everything it was trained on
Data requiredDocuments in their existing formHundreds to thousands of high-quality labeled examples
Cost per queryHigher: a longer contextLower: a short query, a smaller model is possible
Model dependenceThe model is easy to replaceFine-tuning is repeated for every new base model
Best suited forFacts that changeStable skills and formats

What the Research Shows

Empirical comparisons of the two approaches give a fairly consistent picture.

Microsoft researchers compared unsupervised fine-tuning and RAG on tasks that require factual knowledge. RAG consistently showed better results, both for information the model had already encountered during training and for entirely new information. The authors note separately that models find it difficult to absorb new facts through fine-tuning unless the same fact is presented in many formulations [2].

Another study, conducted on agricultural data, showed that the approaches are not mutually exclusive: fine-tuning raised accuracy by more than 6 percentage points, and adding RAG gave a further 5 points on top of that [3]. That is, the effects accumulate, and the "either/or" question is often posed incorrectly.

A third line of work concerns combining the approaches. The RAFT method involves fine-tuning the model specifically for working with retrieved documents, in particular for the ability to ignore irrelevant fragments, and consistently improves results in domain-specific scenarios [6].

Finally, it is worth remembering that the capabilities of dynamic context are themselves not fixed. Retrieval quality depends substantially on how retrieval is organized. According to Anthropic, adding explanatory context to document fragments before indexing reduces the share of failed retrievals by 49%, and in combination with reranking by 67% [4].

An important practical consequence follows from this: before concluding that "RAG cannot cope, fine-tuning is needed," it is worth checking whether the possibilities of retrieval itself have been exhausted. In our practice, in most cases where a client was dissatisfied with the quality of answers, the cause lay in the quality of document preparation and retrieval, not in the model.

Indicators That Fine-Tuning Is Justified

Based on deployment experience, I will single out the situations in which changing the model itself is a well-founded decision.

Stable behavior is needed, not new facts. A strict output format, industry terminology, an established logic for classifying requests or defects. If achieving the required behavior means adding long instructions and dozens of examples to the query every time, it is more efficient to build these patterns into the model.

The skill cannot be described in words. An experienced process engineer distinguishes a typical deviation from an emergency one, but cannot always formulate the rule. If there is an archive of thousands of such decisions, the model can learn the pattern from examples.

Strict constraints on response time or computing resources. A fine-tuned compact model is capable of performing a narrow task at the level of a large general-purpose one, but faster and more cheaply. This is especially important for edge computing and isolated production networks, where running a large model is impossible.

A large and steady flow of uniform queries. When a system processes tens of thousands of structurally identical requests per day, the savings on the length of each query begin to exceed the costs of training and maintenance.

The language or subject domain is poorly represented in the base model. Narrow industry jargon, internal designations, specific document formats with which a general-purpose model works unreliably.

The common condition for all these cases is the availability of a sufficient volume of high-quality examples and the stability of the task over time. If the rules change every quarter, a fine-tuned model becomes outdated faster than it pays for itself.

When Fine-Tuning Will Not Help

It is no less important to understand the situations in which fine-tuning is chosen by mistake.

The knowledge changes frequently. Price lists, stock balances, regulations, schedules. A model trained on yesterday's data confidently reproduces outdated information.

A source citation is required. In regulated industries, an answer without a reference to a specific document and its revision has no value. A fine-tuned model cannot provide such a reference.

Access to information is segregated. Everything the model was trained on is potentially available to everyone who uses it. It is impossible to separate knowledge by role within a single model; with dynamic context, permissions are checked at the retrieval stage. We examined this question in the article on the confidentiality of corporate data.

The data is scarce or of poor quality. A few dozen examples do not form a stable skill, and the model absorbs labeling errors just as diligently as correct answers.

The elimination of fabricated answers is expected. Fine-tuning changes the style and tendencies of the model, but does not guarantee factual accuracy. That requires sources and verification.

The limitations of the opposite approach deserve separate mention. Enlarging the context window does not make the question of information selection redundant: research shows that models make poorer use of information located in the middle of a long context than at its beginning or end [7]. Loading "all the documents at once" is not a substitute for high-quality retrieval.

The Decision Sequence We Apply

In our practice, the following logic of moving from the simple to the complex has taken shape, and it is also consistent with the recommendations of model developers [8].

  1. A clear formulation of the task and instructions. A significant share of quality problems disappears at this stage already.
  2. Dynamic context. Connecting documents and corporate systems, tuning retrieval, evaluating on a benchmark set of questions.
  3. Analysis of residual errors. If the model received the correct data but did not answer as required, this is a behavior problem, and only here does a basis for fine-tuning appear. If the necessary data did not reach the model, this is a retrieval problem.
  4. Fine-tuning for the specific residual problem, while retaining dynamic context for facts.

The third step is the key one in this sequence. Without separating errors into "the model did not know" and "the model knew but acted incorrectly," the choice between the approaches turns into guesswork. We described methods of such evaluation in the article on evaluating the quality of AI agents.

Author's Assessment and Conclusions

Fine-tuning and dynamic context are not competitors. The former builds a skill, the latter supplies knowledge. In mature systems they are combined: a model adapted to the industry's language and formats works with up-to-date data obtained at query time [3][6].

It is worth starting with dynamic context. It delivers a result faster, requires no labeled data, preserves the ability to replace the model, and allows sources to be cited. For most enterprise scenarios this is sufficient.

The basis for fine-tuning is a measured behavior problem. Not a general feeling that "the model does not sufficiently understand our specifics," but a specific class of errors that persists when the context is supplied correctly.

The main cost of fine-tuning is not computation. Preparing and maintaining the set of examples, retraining with every change in requirements or in the base model, and regression testing are ongoing costs that should be budgeted from the very beginning.

A decision about fine-tuning is a decision about data. An enterprise that does not have an orderly archive of decisions, classifications, or reference documents is not ready for fine-tuning regardless of budget. And conversely: the existence of such an archive is an asset worth using.

The choice of approach is ultimately determined neither by fashion nor by the availability of tools, but by the nature of the task: what exactly must change, what the model knows at the moment of the query or how it always acts.

References

  1. Lewis P., Perez E., Piktus A. et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." arXiv:2005.11401, 2020. https://arxiv.org/abs/2005.11401
  2. Ovadia O., Brief M., Mishaeli M., Elisha O. "Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs." arXiv:2312.05934, 2023. https://arxiv.org/abs/2312.05934
  3. Balaguer A., Benara V., Cunha R. et al. "RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture." arXiv:2401.08406, 2024. https://arxiv.org/abs/2401.08406
  4. Anthropic. "Introducing Contextual Retrieval." 2024. https://www.anthropic.com/news/contextual-retrieval
  5. Hu E., Shen Y., Wallis P. et al. "LoRA: Low-Rank Adaptation of Large Language Models." arXiv:2106.09685, 2021. https://arxiv.org/abs/2106.09685
  6. Zhang T., Patil S., Jain N. et al. "RAFT: Adapting Language Model to Domain Specific RAG." arXiv:2403.10131, 2024. https://arxiv.org/abs/2403.10131
  7. Liu N., Lin K., Hewitt J. et al. "Lost in the Middle: How Language Models Use Long Contexts." arXiv:2307.03172, 2023. https://arxiv.org/abs/2307.03172
  8. OpenAI. "Optimizing LLM Accuracy." Developer documentation. https://platform.openai.com/docs/guides/optimizing-llm-accuracy
fine-tuningmodel fine-tuningragretrieval augmented generationdynamic contextlorallmenterprise aiknowledge managementai knowledge basemodel selection

AQTeck solutions on this topic

We design and implement these systems at industrial enterprises in Ukraine and the EU.

Related Articles

Need a consultation?

Contact us for a free consultation.