We’ve all been there. It’s 11:30 PM, your IDE is finally closed, but you have 47 browser tabs open. You're jump-reading an RFC on HTTP/3, skimming a GitHub issue about a memory leak in a Redis client, and bookmarking three different deep-dives on Vector databases. A recent Hacker News thread struck a chord with thousands of us: "I'm not addicted to the internet or my smartphone. I'm addicted to information."
As software engineers, our brains are literally wired to seek out, categorize, and synthesize information. It’s our superpower, but it’s also our biggest cognitive bottleneck. We hoard bookmarks, star GitHub repos, and save articles to Pocket or Notion, only for those repositories of knowledge to become digital graveyards. We can't find what we saved when we actually need it because standard keyword search is too brittle.
Today, we’re going to turn this information addiction into an organized, highly queryable superpower. Instead of drowning in bookmark folders, we are going to build a self-hosted, local Semantic Search Engine for your personal markdown notes and bookmarked articles. We'll use Go, Qdrant (a high-performance vector database), and local Hugging Face embeddings so your data stays 100% private and runs entirely on your local machine.
The Architecture of Semantic Search
Before we jump into the code, let’s understand the difference between traditional keyword search and semantic search. Keyword search (like grep or basic SQL LIKE queries) looks for exact character matches. If you search for "database scaling," you might miss an article titled "Sharding Postgres for High Throughput."
Semantic search leverages Machine Learning to convert text into dense vectors (embeddings). These vectors are mathematical representations of the meaning of the text in a high-dimensional space. Words and phrases with similar semantic meanings sit close to each other in this vector space.
Our architecture will look like this:
+------------------+ +-------------------+ +-------------------+
| Personal Notes | ---> | Go Ingestion | ---> | Local Embeddings |
| & Bookmarks | | Service (CLI) | | (HuggingFace API |
+------------------+ +-------------------+ | or Local Ollama) |
| +-------------------+
v |
+-------------------+ | (Vector)
| Qdrant Vector DB | <--------------+
| (Dockerized) |
+-------------------+
We will write a Go CLI tool that scans a local directory of Markdown files, chunks the text, calls a local endpoint to generate embeddings, and upserts those vectors into a local Qdrant instance. Finally, we'll write a search query tool to find our saved knowledge using natural language.
Setting Up the Infrastructure
To keep this project light and fully local, we will run Qdrant via Docker. Qdrant is incredibly fast, written in Rust, and has a very friendly developer API.
First, spin up Qdrant in your terminal:
docker run -p 6333:6333 -p 6334:6334 \
-v $(pwd)/qdrant_storage:/qdrant/storage:z \
qdrant/qdrantNext, we need a way to generate vector embeddings locally. We will use Ollama, an amazing tool for running LLMs and embedding models locally. Download and install Ollama, then pull the lightweight, high-performance embedding model all-minilm:
ollama pull all-minilmThis model converts any text chunk into a 384-dimensional vector, which is perfect for fast, local semantic search without consuming gigabytes of VRAM.
Building the Go Ingest Engine
Let's initialize our Go project. Create a new directory and initialize the module:
mkdir info-vault && cd info-vault
go mod init info-vault
go get github.com/google/uuidWe'll start by writing our data structures and the main ingestion logic. Create a file named main.go. We want to read markdown files, split them into manageable paragraphs (chunking), generate embeddings for each chunk, and store them in Qdrant.
Step 1: Parsing and Chunking Markdown
We need to split our long articles into smaller chunks (roughly 500 characters) so that the semantic search returns the specific paragraph we need, rather than an entire 5,000-word document.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"os"
"path/filepath"
"strings"
"github.com/google/uuid"
)
type Chunk struct {
ID string
FilePath string
Content string
Vector []float32
}
// Simple chunking strategy: Split by double newline (paragraphs)
func chunkFile(filePath string) ([]Chunk, error) {
content, err := ioutil.ReadFile(filePath)
if err != nil {
return nil, err
}
paragraphs := strings.Split(string(content), "\n\n")
var chunks []Chunk
for _, para := range paragraphs {
trimmed := strings.TrimSpace(para)
if len(trimmed) < 50 { // Skip empty or extremely short lines
continue
}
chunks = append(chunks, Chunk{
ID: uuid.New().String(),
FilePath: filePath,
Content: trimmed,
})
}
return chunks, nil
}
Step 2: Generating Local Embeddings with Ollama
Now, we'll write a function to communicate with our local Ollama instance to transform our text chunks into 384-dimensional mathematical vectors.
type OllamaEmbeddingResponse struct {
Embedding []float32 `json:"embedding"`
}
type OllamaEmbeddingRequest struct {
Model string `json:"model"`
Prompt string `json:"prompt"`
}
func getEmbedding(text string) ([]float32, error) {
reqBody, err := json.Marshal(OllamaEmbeddingRequest{
Model: "all-minilm",
Prompt: text,
})
if err != nil {
return nil, err
}
resp, err := http.Post("http://localhost:11434/api/embeddings", "application/json", bytes.NewBuffer(reqBody))
if err != nil {
return nil, err
}
defer resp.Body.Close()
var result OllamaEmbeddingResponse
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
return nil, err
}
return result.Embedding, nil
}
Step 3: Storing Vectors in Qdrant
With our vectors generated, we need to send them to Qdrant. First, we must initialize a "collection" in Qdrant configured for 384-dimensional vectors using Cosine similarity. Run this setup function once, or check if the collection exists before indexing.
func initQdrantCollection() error {
url := "http://localhost:6333/collections/knowledge_base"
payload := []byte(`{
"vectors": {
"size": 384,
"distance": "Cosine"
}
}`)
req, err := http.NewRequest("PUT", url, bytes.NewBuffer(payload))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode == http.StatusOK || resp.StatusCode == http.StatusConflict {
return nil // Collection created successfully or already exists
}
return fmt.Errorf("failed to create collection, status: %d", resp.StatusCode)
}
Now, let’s write the code to upload our chunks as points to Qdrant.
type QdrantPoint struct {
ID string `json:"id"`
Vector []float32 `json:"vector"`
Payload map[string]interface{} `json:"payload"`
}
type QdrantUpsertRequest struct {
Points []QdrantPoint `json:"points"`
}
func upsertToQdrant(chunks []Chunk) error {
var points []QdrantPoint
for _, chunk := range chunks {
points = append(points, QdrantPoint{
ID: chunk.ID,
Vector: chunk.Vector,
Payload: map[string]interface{}{
"file_path": chunk.FilePath,
"content": chunk.Content,
},
})
}
reqBody, err := json.Marshal(QdrantUpsertRequest{Points: points})
if err != nil {
return err
}
url := "http://localhost:6333/collections/knowledge_base/points?wait=true"
resp, err := http.Post(url, "application/json", bytes.NewBuffer(reqBody))
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("qdrant upsert failed with status: %d", resp.StatusCode)
}
return nil
}
Step 4: Putting the Pipeline Together
Let's construct our main execution path. This CLI will read Markdown files from a directory called ./vault, parse them, generate embeddings, and save them to Qdrant.
func main() {
if len(os.Args) < 2 {
fmt.Println("Usage: go run main.go [index|search] [query/path]")
return
}
command := os.Args[1]
if command == "index" {
vaultPath := "./vault"
if len(os.Args) > 2 {
vaultPath = os.Args[2]
}
fmt.Println("Initializing Qdrant Collection...")
if err := initQdrantCollection(); err != nil {
fmt.Printf("Error: %v\n", err)
return
}
fmt.Printf("Scanning directory: %s\n", vaultPath)
var allChunks []Chunk
err := filepath.Walk(vaultPath, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if !info.IsDir() && strings.HasSuffix(info.Name(), ".md") {
fmt.Printf("Processing %s...\n", info.Name())
chunks, err := chunkFile(path)
if err != nil {
return err
}
allChunks = append(allChunks, chunks...)
}
return nil
})
if err != nil {
fmt.Printf("Error walking directory: %v\n", err)
return
}
fmt.Printf("Generating embeddings for %d chunks...\n", len(allChunks))
for i := range allChunks {
vector, err := getEmbedding(allChunks[i].Content)
if err != nil {
fmt.Printf("Error generating embedding: %v\n", err)
return
}
allChunks[i].Vector = vector
}
fmt.Println("Uploading to Qdrant...")
if err := upsertToQdrant(allChunks); err != nil {
fmt.Printf("Error uploading to Qdrant: %v\n", err)
return
}
fmt.Println("Ingestion complete! Your knowledge vault is indexed.")
} else if command == "search" {
if len(os.Args) < 3 {
fmt.Println("Please provide a search query.")
return
}
query := os.Args[2]
performSearch(query)
}
}
Querying Your Personal Knowledge Engine
Now for the magic. We need to implement the performSearch function. When you run a query like "How did I solve that Docker volume permissions error?", the app will convert your question into a vector embedding, send it to Qdrant, and retrieve the paragraphs with the closest cosine similarity.
Add this search function to your main.go:
type QdrantSearchRequest struct {
Vector []float32 `json:"vector"`
Limit int `json:"limit"`
WithPayload bool `json:"with_payload"`
}
type QdrantSearchResponse struct {
Result []struct {
Score float64 `json:"score"`
Payload struct {
Content string `json:"content"`
FilePath string `json:"file_path"`
} `json:"payload"`
} `json:"result"`
}
func performSearch(query string) {
fmt.Printf("Searching for: \"%s\"\n", query)
// 1. Vectorize the search query
queryVector, err := getEmbedding(query)
if err != nil {
fmt.Printf("Error vectorizing query: %v\n", err)
return
}
// 2. Query Qdrant
searchReq := QdrantSearchRequest{
Vector: queryVector,
Limit: 3, // Return top 3 most relevant matches
WithPayload: true,
}
reqBody, _ := json.Marshal(searchReq)
url := "http://localhost:6333/collections/knowledge_base/points/search"
resp, err := http.Post(url, "application/json", bytes.NewBuffer(reqBody))
if err != nil {
fmt.Printf("Qdrant query failed: %v\n", err)
return
}
defer resp.Body.Close()
var searchResult QdrantSearchResponse
if err := json.NewDecoder(resp.Body).Decode(&searchResult); err != nil {
fmt.Printf("Failed to parse response: %v\n", err)
return
}
// 3. Display results
fmt.Println("\n--- Search Results ---")
for i, match := range searchResult.Result {
fmt.Printf("\n[%d] Relevance Score: %.4f\n", i+1, match.Score)
fmt.Printf("Source File: %s\n", match.Payload.FilePath)
fmt.Printf("Content Highlight:\n%s\n", match.Payload.Content)
fmt.Println(strings.Repeat("-", 40))
}
}
Testing Your Local Semantic Search
Let's create a test vault directory and throw some unstructured developer wisdom into it.
mkdir vault
cat <<EOF > vault/docker_tips.md
# Docker Troubleshooting Notes
Ran into a major issue today with Postgres failing to start because of directory permissions on a mounted volume.
To fix permission errors with Docker volumes on Linux, I had to change the ownership of the host folder using chown -R 999:999 ./data. This is because the default Postgres Docker image runs as user ID 999.
EOF
cat <<EOF > vault/redis.md
# Scaling Redis Cache
Redis eviction policies are key. We hit our memory limit in production.
Switching from volatile-lru to allkeys-lru saved us from complete system crashes.
Make sure to monitor maxmemory-policy in the redis.conf file.
EOF
Now, build and run your ingestion pipeline:
go build -o vault-search main.go
./vault-search indexOnce indexed, ask a semantic question that does not share identical keywords with your source files:
./vault-search search "database out of memory crash mitigation"Even though "out of memory crash mitigation" is not explicitly in our Redis note, the vector search engine will successfully associate "memory limit," "eviction policies," and "system crashes" with your query, surfacing the vault/redis.md file as the top result!
Conclusion: Curbing the Noise
We can't easily stop our brains from craving information, but we can design smart systems to offload the cognitive burden of remembering where we saw it. By building a local semantic search engine, you bypass the cloud, ensure privacy, and create a highly responsive personal search engine custom-tuned to your personal developer stack.
What are you using to manage your information overload? Have you integrated local LLMs into your daily developer workflow? Let me know in the comments below!
Happy coding, and go clear some of those open tabs!