What is a foundation model? Explained

What a foundation model is in plain words

Older AI systems were built for one job each. A foundation model flips that: it learns from a huge range of data first, then gets adapted to specific work.

Stanford HAI stresses that these models are not only about text. They can also be trained on images, video and other kinds of data.

IBM Research gives a sense of the payoff. With pre-trained models, a language task needed about 100 times fewer annotated examples than before.

Put simply, if you ask what is a foundation model in practice, it is the shared starting point that saves you from training everything from scratch.

Foundation model vs LLM

• Foundation model

  Broad base model, many data types.

• LLM

  A foundation model focused on language.

• Multimodal

  Combines text, images and audio.

• Frontier model

  Among the most capable at a given time.

A large language model is one kind of foundation model. IBM describes LLMs as deep learning models trained on very large amounts of data to understand and generate language.

So the foundation model vs LLM question has a simple answer: every LLM is a foundation model, but not every foundation model is an LLM. A multimodal foundation model also handles images or audio.

You will also hear frontier model. That term points to the most capable models available at a given moment, while foundation model describes the base-and-adapt design.

For the language side, see large language model. For the data side beyond text, see multimodal AI.

Small language models and when they fit

A small language model is a compact language model with far fewer parameters than the biggest ones. For comparison, GPT-3 had 175 billion parameters, and that is the scale small models avoid.

Smaller models run on lighter hardware, answer faster and can stay inside your own systems. That helps when cost, speed or data privacy matter more than broad knowledge.

A small language model fits narrow, repeated tasks such as sorting emails or extracting fields. Open-ended work that needs wide knowledge usually calls for a larger model.

Check the right size against the job before choosing. Our AI integration work starts with that fit.

How companies adapt a foundation model

Most companies never train a base model. They adapt an existing one, and there are three common ways, from lightest to heaviest.

Prompting gives clear instructions and examples. Retrieval connects the model to your own documents so answers use your facts. Fine-tuning trains the model further on your examples.

Start light and add effort only when results demand it. For the retrieval route, read retrieval-augmented generation.