Hey everyone, Alex here. Welcome back to another edition of Coding with Alex on sysseder.com.
If you’ve been scrolling through Hacker News, GitHub trending, or your Mastodon feed lately, you’ve probably noticed a massive, escalating debate. It’s encapsulated by the headline: "Who gets to define the rules for AI?"
At first glance, this sounds like a high-level philosophical debate meant for policy-makers in Brussels or Washington, D.C. But as software engineers, cloud architects, and system administrators, this debate is actually happening directly in our terminal windows. Every time we run ollama run llama3, pull a model from Hugging Face, or integrate an LLM into our CI/CD pipelines, we are actively participating in this ecosystem.
Right now, the Open Source Initiative (OSI) is finalizing the official Open Source AI Definition (OSAID). This isn't just academic pedantry. How we define "open source" in the era of neural networks will dictate our freedom to modify, self-host, audit, and deploy LLMs without being locked into expensive, proprietary APIs.
In this post, we’re going to look at why this definition matters to developers, dissect the technical components of a truly "open" model, and write some practical Python code to run and inspect a local LLM using open-source tooling.
The Technical Crisis: What is an "Open" Model anyway?
In traditional software engineering, open source is straightforward. You have the source code (human-readable .py, .go, or .rs files), you compile it (or interpret it), and you run it. Under licenses like MIT, Apache 2.0, or GPL, you have the right to inspect, modify, and redistribute that code.
AI models break this paradigm completely. An LLM consists of two main parts:
- The Architecture: The structural code (often written in PyTorch or JAX) defining the layers, attention heads, and tensor shapes.
- The Weights: A massive binary blob of parameters (floating-point numbers) resulting from training on petabytes of data.
If a company releases the model weights under a permissive license but hides the training data, the data filtering algorithms, and the training code, is that model truly "open source"?
The developer community is split. On one side, companies like Meta release Llama under "open-weight" licenses, but place commercial usage restrictions and hide the exact training corpus. On the other side, purists argue that without the exact training data and pipeline, a developer cannot fully inspect, patch, or reproduce the model, violating the core tenets of open source.
The Architecture of an Open AI Stack
As developers, we care about predictability, auditability, and data privacy. When we run a closed-source API like OpenAI's GPT-4, we are sending user data to an external server, accepting silent model updates that can break our prompts overnight, and paying per token.
By moving to a local, open stack, we regain control. A modern developer's local AI architecture typically looks like this:
+-------------------------------------------------------------+
| Application Layer |
| (LangChain, LlamaIndex, Custom Python) |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| Inference Engine |
| (Ollama, llama.cpp, vLLM, Hugging Face) |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| Model & Quantization |
| (GGUF, Safetensors, AWQ, GPTQ files) |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| Hardware Acceleration |
| (Metal on Mac, CUDA on NVIDIA) |
+-------------------------------------------------------------+
To keep things truly open, transparent, and reproducible, we should target models that release both their weights and their training parameters. Excellent examples include Allen Institute for AI's OLMo or EleutherAI’s models, which provide the open datasets and training code alongside the weights.
Hands-On: Running and Inspecting a Local LLM
Let's build a practical, local setup. We are going to use Python and the Hugging Face transformers library to load an open-weight model, inspect its parameters, and run a secure, offline inference pipeline.
Step 1: Setting up your Environment
First, let's set up a virtual environment and install the required dependencies. We will need torch (PyTorch) and transformers.
mkdir local-ai-explorer
cd local-ai-explorer
python3 -m venv venv
source venv/bin/activate
pip install torch transformers accelerate
Step 2: Writing the Inference Script
We’ll write a script that loads a lightweight, highly capable, open-weights model: Qwen/Qwen2.5-1.5B-Instruct. This model is small enough to run on almost any developer laptop without requiring a massive GPU, yet is highly capable for code generation and reasoning.
Create a file named app.py and add the following code:
import os
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
# Ensure we run offline if we've already downloaded the model
# OS_HF_HUB_DISABLE_SYMLINKS=1
os.environ["HF_HUB_DISABLE_TELEMETRY"] = "1"
def main():
model_name = "Qwen/Qwen2.5-1.5B-Instruct"
print(f"[*] Loading tokenizer for {model_name}...")
tokenizer = AutoTokenizer.from_pretrained(model_name)
print(f"[*] Loading model {model_name}...")
# Using device_map="auto" to automatically utilize GPU/MPS if available
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype="auto",
device_map="auto"
)
# Let's inspect the model metadata programmatically!
print("\n--- Model Metadata & Architecture ---")
print(f"Device: {model.device}")
print(f"Total Parameters: {sum(p.numel() for p in model.parameters()):,}")
print(f"Data Type: {model.dtype}")
print(f"Memory Footprint: {model.get_memory_footprint() / (1024**2):.2f} MB")
print("-------------------------------------\n")
# Prompting the model
prompt = "Write a Python function that checks if a string is a valid IPv4 address."
messages = [
{"role": "system", "content": "You are an expert software engineer."},
{"role": "user", "content": prompt}
]
# Formatting prompt using the chat template defined in the model config
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
print("[*] Generating response (running entirely locally)...")
generated_ids = model.generate(
**model_inputs,
max_new_tokens=512,
temperature=0.7,
do_sample=True
)
# Strip the input tokens from the output to get only the generation
generated_ids = [
output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print("\n--- Model Response ---")
print(response)
print("----------------------")
if __name__ == "__main__":
main()
Step 3: Running the Script
Execute the script in your terminal:
python app.py
On a modern Apple Silicon Mac or a machine with an NVIDIA card, PyTorch will automatically leverage hardware acceleration (MPS or CUDA). Because we printed the metadata, you'll see exactly how many parameters are being computed on your machine (approx. 1.54 billion parameters) and the exact memory footprint (around 3GB of RAM).
Why Open Weights Alone Aren't Enough
While the code above runs flawlessly on your local machine, we must address the elephant in the room: black box weights.
When you run Qwen2.5, Llama3, or Mistral, you are executing mathematical operations across billions of weights. But how were those weights calculated?
Without access to the training data, we cannot answer critical questions:
- Licensing & Copyright: Was the model trained on GPL-licensed code without attribution? If it spits out a chunk of code that you use in a commercial application, are you violating copyright?
- Bias and Security: Has the model been trained to intentionally suppress certain technical answers, or does it contain hidden vulnerabilities introduced via poisoned datasets?
- Explainability: Why does the model choose token A over token B? Without the exact input corpus distribution, deep debugging is nearly impossible.
This is why the Open Source Initiative is working on the Open Source AI Definition. The current draft mandates that for an AI system to be "Open Source," it must provide enough information about its training data so that others can understand and reproduce the system's behavior, along with the complete source code used to train and run it.
The Path Forward for Developers
As developers, we have immense power in this debate. We are the consumers, the builders, and the decision-makers inside our companies. Here is how you can take action and stay resilient against "AI vendor lock-in":
- Audit Your Models: When integrating LLMs into your company's stack, categorize them. Are they proprietary APIs (OpenAI, Claude), open-weight models (Llama, Mistral), or truly open-source models (OLMo)? Know the security and legal risks of each.
- Deploy Locally: Build systems that decouple the inference engine from the application logic. Tools like
llama.cpporOllamapresent standard OpenAI-compatible HTTP endpoints, making it trivial to swap a proprietary backend for a local, open one. - Support Open Standards: Follow and contribute to organizations like the Open Source Initiative (OSI) and the Linux Foundation's AI & Data Foundation. Demand transparency from vendors claiming their models are "open source."
What are your thoughts on this? Should "open source" strictly mean 100% reproducible training pipelines, or are "open weights" enough for you to build production systems confidently?
Let me know in the comments below, or find me on Mastodon. Until next time, happy coding!
— Alex