Welcome to this developer-focused guide to local language models and the meaning of open-source AI.
A practical question sits behind the debate over AI openness: who defines the permissions and information that developers receive?
At first glance, this sounds like a high-level philosophical debate meant for policy-makers in Brussels or Washington, D.C. But as software engineers, cloud architects, and system administrators, this debate is actually happening directly in our terminal windows. Every time we run ollama run llama3, pull a model from Hugging Face, or integrate an LLM into our CI/CD pipelines, we are actively participating in this ecosystem.
OSI released Open Source AI Definition (OSAID) 1.0 on 28 October 2024; it is not an unfinished draft. It is a framework for assessing openness, not a replacement for each model's license. Inspect the rights and artifacts of the specific release you intend to use. OSI release announcement.
In this post, we’re going to look at why this definition matters to developers, dissect the technical components of a truly "open" model, and write some practical Python code to run and inspect a local LLM using open-source tooling.
The Technical Crisis: What is an "Open" Model anyway?
In conventional software, open-source licenses provide rights to use, inspect, modify and redistribute source code, subject to their terms. MIT, Apache 2.0 and GPL differ in their obligations; “open” does not mean that attribution or other license conditions disappear.
For a language model, distinguish at least the architecture, learned parameters and inference code. Its training and data-processing artifacts matter too. Two central components are:
- Architecture: The model structure, including layers, attention mechanisms and tensor shapes.
- Weights: Learned parameters, represented at a chosen numerical precision; their training-data scale varies by model.
If a company releases the model weights under a permissive license but hides the training data, the data filtering algorithms, and the training code, is that model truly "open source"?
Downloadable weights alone do not establish OSAID compliance. The definition calls for freedoms to use, study, modify and share, and a preferred form for modifications comprising data information, code and parameters. A model-family name is not enough to determine a particular release's rights. OSAID 1.0 requirements.
The Architecture of an Open AI Stack
Predictability, auditability and privacy matter whether inference is hosted or local. With a hosted API, review data handling, pricing, available version controls and change policies. Local inference gives more deployment control, but transfers responsibility for dependencies, access controls, logs, updates and hardware to the operator. Neither arrangement guarantees security or stable outputs.
A local stack can separate the application, inference software, model artifacts and hardware. The following is a schematic, not a compatibility matrix: GGUF and Safetensors are formats, while AWQ and GPTQ describe quantization approaches; not every engine supports every combination.
+-------------------------------------------------------------+
| Application Layer |
| (LangChain, LlamaIndex, Custom Python) |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| Inference Engine |
| (Ollama, llama.cpp, vLLM, Hugging Face) |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| Model & Quantization |
| (GGUF, Safetensors, AWQ, GPTQ files) |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| Hardware Acceleration |
| (Metal on Mac, CUDA on NVIDIA) |
+-------------------------------------------------------------+
If training transparency matters, examine concrete releases such as Ai2's OLMo 2, whose maintainers publish training data, code, recipes, evaluations and checkpoints. Availability enables inspection; it does not guarantee that a reader can afford to reproduce training or that a model is free of defects. Ai2 OLMo 2 artifacts.
Hands-On: Running and Inspecting a Local LLM
The example below illustrates local loading, metadata inspection and generation with Python and Transformers. It has not been runtime-tested for this article. It is not a secure-offline guarantee: Hub offline settings do not replace operating-system network isolation, dependency review or output validation.
Step 1: Setting up your Environment
Create a virtual environment. Install reviewed versions of PyTorch, Transformers and Safetensors using their official instructions, and record exact versions in a lockfile. Dependency installation and downloading a model are separate online preparation steps. In advance, obtain a complete, reviewed snapshot of the model at a recorded commit, including tokenizer/configuration files and Safetensors weights. Do not use a floating model revision for a reproducibility claim. Transformers setup and offline preparation; Safetensors.
mkdir local-ai-explorer
cd local-ai-explorer
python3 -m venv venv
# POSIX shell; use your platform's activation command on Windows.
source venv/bin/activate
Step 2: Writing the Inference Script
The example targets Qwen/Qwen2.5-1.5B-Instruct, whose model card reports 1.54 billion parameters. That does not establish compatibility with every laptop. CPU float32 weights alone occupy roughly 6 GB in decimal units; runtime overhead and attention caches need more. GPU or quantized configurations have different requirements. Official Qwen model card.
Save the following illustrative script as app.py. It requires an existing trusted local model directory, refuses remote custom code and requires Safetensors weights. These controls reduce particular loading risks; they are not a complete sandbox. Model loading precautions.
import os
# Set Hub controls before importing the Hugging Face libraries.
os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["HF_HUB_DISABLE_TELEMETRY"] = "1"
import argparse
from pathlib import Path
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
def main():
parser = argparse.ArgumentParser()
parser.add_argument("model_dir", help="Reviewed local Qwen2.5-1.5B-Instruct snapshot")
args = parser.parse_args()
model_dir = Path(args.model_dir).expanduser().resolve(strict=True)
if not model_dir.is_dir():
raise ValueError("model_dir must be a directory")
tokenizer = AutoTokenizer.from_pretrained(
str(model_dir), local_files_only=True, trust_remote_code=False
)
# Explicit CPU example; float32 weights need roughly 6 GB before overhead.
model = AutoModelForCausalLM.from_pretrained(
str(model_dir), local_files_only=True, trust_remote_code=False,
use_safetensors=True, torch_dtype=torch.float32
).to("cpu")
model.eval()
print(f"Device: {model.device}")
print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")
print(f"Dtype: {model.dtype}")
print(f"Model storage estimate: {model.get_memory_footprint() / 1024**2:.2f} MiB")
messages = [
{"role": "system", "content": "You are a software engineering assistant."},
{"role": "user", "content": "Explain how Python's ipaddress module validates IPv4 addresses."},
]
text = tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(text, return_tensors="pt").to("cpu")
with torch.inference_mode():
output = model.generate(**inputs, max_new_tokens=128, do_sample=False)
answer = output[0, inputs.input_ids.shape[1]:]
print(tokenizer.decode(answer, skip_special_tokens=True))
if __name__ == "__main__":
main()
Step 3: Running the Script
After reviewing the code and preparing the local snapshot, replace the example path with its actual directory. Missing local files should produce an error rather than a Hub download.
python app.py /path/to/reviewed-qwen-snapshot
This script explicitly uses CPU; it does not promise automatic CUDA or MPS acceleration. Acceleration depends on compatible software, hardware and model operations. The printed memory footprint describes model storage, not peak process RAM or total inference memory. HF_HUB_DISABLE_TELEMETRY disables supported telemetry, not downloads; HF_HUB_OFFLINE and local_files_only=True control Hub/local loading. For an actual no-network requirement, enforce and verify network isolation outside Python. Device selection; Hub environment controls.
Why Open Weights Alone Aren't Enough
Being able to run a model and count its parameters does not reveal how its training data was collected or how every output was produced. Local execution and transparency are different properties.
When you run Qwen2.5, Llama3, or Mistral, you are executing mathematical operations across learned model weights. But how were those weights calculated?
Training documentation can help investigate questions such as:
- Licensing and provenance: What licenses and sources apply to this release and its data? Neither available weights nor a generated answer settles whether a particular use is lawful.
- Bias and security: What filtering, evaluations and provenance checks were performed? Open data helps investigation but does not guarantee freedom from poisoning or bias.
- Explainability: What can evaluations and interpretability tools tell us about outputs? Even the exact corpus does not automatically explain each token choice.
OSAID 1.0 requires data information sufficient for a skilled person to build a substantially equivalent system using the same or similar data, alongside the required code and parameters. It does not demand publication of every original training example or promise bit-for-bit reproduction. Definition; OSI FAQ.
The Path Forward for Developers
As developers, we have immense power in this debate. We are the consumers, the builders, and the decision-makers inside our companies. Here is how you can take action and stay resilient against "AI vendor lock-in":
- Audit specific releases: Record the model version, license, available training artifacts, evaluation results and deployment constraints. Do not classify an entire family as legally or technically identical.
- Separate application and inference: An API boundary can ease backend changes, but compatibility, prompting, tool calling and output quality still need testing. Ollama documents compatibility with parts of the OpenAI API, not a universal drop-in replacement. Ollama compatibility.
- Support open standards: Follow OSI and relevant open AI projects. Ask vendors for specific artifacts and permissions behind their openness claims.
Which artifacts do you need before deploying a model: downloadable weights, training code, data information, or a reproducible training recipe? The answer should follow your project's goals and review requirements.
Share your experience in the comments.
Technical references checked on 28 October 2026; original publication date retained.