Showcase Project: On-device LLMs with Raspberry Pi 5 for Your Developer Portfolio

Hook: Ship an edge AI portfolio piece hiring managers can’t ignore

Hiring managers and portfolio reviewers want to see tangible evidence you can build reliable, production-minded systems — not just notebooks. If you’re a developer or ML engineer aiming to demonstrate expertise in edge AI, on-device inference, and model optimization, a polished Raspberry Pi 5 project using the AI HAT+ 2 is one of the clearest signals you can present in 2026. This article gives you a complete project blueprint, repo structure, and deployable demo app tailored to reviewers who care about latency, reproducibility, security, and maintainability.

Why this project matters in 2026

Two forces converged in late 2024–2025 and set the stage for 2026: (1) mainstream quantized runtimes and compact LLMs made accurate on-device inference feasible, and (2) hardware add-ons like the AI HAT+ 2 (released late 2025) brought dedicated acceleration and easier I/O to Raspberry Pi 5. For portfolio reviewers, this combination signals that you understand real-world trade-offs — model size vs. latency, offline inference vs. cloud, and power constraints vs. throughput.

Demonstrating an end-to-end on-device generative AI system on Pi 5 shows you can: architect for constrained hardware, optimize models, set up reproducible tooling, and present measurable results — all traits hiring teams prioritize when hiring for edge ML roles.

High-level project goals (what reviewers want to see)

• Functionality: A working demo performing text generation (or assistant tasks) fully on-device using Pi 5 + AI HAT+ 2.
• Performance metrics: Latency (cold start, tokens/sec), memory footprint, power draw, and model accuracy tradeoffs across quantization levels.
• Reproducibility: Clear setup scripts, deterministic model conversions, Docker / Nix / reproducible build.
• Code quality: Modular repo, automated tests, CI pipelines, documentation and a short walkthrough video.
• Security & privacy: Local-only inference option, threat model notes, and secure update path.

Project blueprint — scope and deliverables

Target scope for a portfolio piece (2–4 weeks of focused work):

1. Hardware proof-of-concept: Boot Pi 5, attach AI HAT+ 2, run a quantized LLM locally.
2. Model pipeline: Convert and quantize a compact open model (4-bit/8-bit) and load it with an efficient runtime (GGML-based or equivalent).
3. Demo app: Lightweight FastAPI web UI that serves a small chat interface and a local-only REST endpoint.
4. Benchmarking suite: Scripts to measure latency, memory, and power; generate a simple report (CSV + plot).
5. Repo artifacts: README with metrics, architecture diagram, setup scripts, CI (tests + lint), and a demo video (2–3 min).

Hardware & software checklist

• Raspberry Pi 5 (8GB or 16GB recommended)
• AI HAT+ 2 (AI accelerator HAT for Pi 5 released late 2025)
• MicroSD or NVMe storage (fast, 64GB+)
• Power supply and optional UPS hat for power measurements
• Ubuntu 24.04 or Raspberry Pi OS (64-bit) — use the distro that matches your runtime support
• Toolchain: build-essential, Python 3.11+, pip, git, and the chosen C++ runtime (example: llama.cpp/ggml compatibility)

Model choice & optimization strategy

For portfolio projects, choose small but capable models that showcase trade-offs clearly. In 2026, many community models optimized for edge deployment offer good conversational ability at 300M–7B parameter ranges. Your repo should document why you chose a model and the conversion steps.

Optimization steps to include

• Quantization: 8-bit or 4-bit quantization to reduce memory and improve inference speed. Include conversion scripts and test accuracy degradation.
• Pruning & distillation (optional): If you use distillation, provide before/after evals to show gains.
• Memory mapping & batching: Use mmap or on-disk formats to reduce RAM peaks and batch token generation efficiently.
• Offloading: If AI HAT+ 2 supports offload kernels, document how to use them to move expensive ops off the CPU.

Concrete repo structure (a recruiter-friendly layout)

Design the repository so a reviewer can quickly find working demos, setup scripts, and results. Below is a recommended tree and rationale.

edge-llm-pi5/
├─ README.md
├─ LICENSE
├─ hardware/
│  ├─ bills_of_materials.md
│  ├─ wiring.md
│  └─ photos/
├─ scripts/
│  ├─ setup_os.sh
│  ├─ install_runtime.sh
│  └─ convert_model.sh
├─ models/
│  ├─ original/ (links or submodule)
│  └─ converted/ (gitignored artifacts, with checksums)
├─ runtime/
│  ├─ inference_server.py
│  ├─ wrapper_c_api/ (if building runtime bindings)
│  └─ Dockerfile
├─ web-ui/
│  ├─ app.py
│  └─ static/ (js/css)
├─ benchmarks/
│  ├─ run_benchmarks.py
│  └─ results.csv
├─ docs/
│  ├─ architecture.md
│  └─ metrics.md
└─ .github/workflows/ci.yml

Key files explained

• README.md: Top-level quickstart, hardware list, and a one-minute demo GIF. Lead with the results section so reviewers see your claims immediately.
• scripts/setup_os.sh: Idempotent setup for Pi 5 OS, dependencies, and user privileges. Make it safe to re-run.
• scripts/convert_model.sh: Step-by-step, reproducible convert + quantize commands. Include the exact commit/tag for the original model and checksum for outputs.
• runtime/inference_server.py: Minimal FastAPI server exposing a POST /generate endpoint. Keep it small; heavy optimization can be in a backend C++ binary.
• benchmarks/run_benchmarks.py: Automate latency and memory measurement; output CSV/JSON and a markdown summary.
• .github/workflows/ci.yml: Linting, unit tests for conversion scripts (dry runs), and building docs. Avoid running heavy model builds in CI; use artifact publishing for large files.

Example: convert_model.sh (practical commands)

Below is a simplified, high-level conversion flow. Adapt to your chosen runtime; include exact tool versions and checksums in the repo.

#!/usr/bin/env bash
set -euo pipefail
MODEL_SOURCE="open-model-vX.Y"
OUT_DIR="../models/converted"
mkdir -p "$OUT_DIR"
# Example: download the model (or reference a submodule)
# Convert and quantize (pseudo-commands — replace with runtime-specific tools)
python3 convert.py --input "$MODEL_SOURCE" --out "$OUT_DIR/model.bin" --quantize 4
sha256sum "$OUT_DIR/model.bin" > "$OUT_DIR/model.bin.sha256"

Demo app: design and UX decisions

Make the demo purposeful and short. Hiring managers rarely want to interact for more than a few minutes. Provide three modes:

• Quick demo: A fixed 3-turn chat highlighting coherent answers and latency badge.
• Interactive endpoint: Local-only REST endpoint for testing; show curl examples.
• Automated smoke tests: Validate generation length, non-empty responses, and timing.

UX tips: include a counter showing tokens/sec and a toggle to switch quantization levels (4-bit, 8-bit) so reviewers can see trade-offs live.

Benchmarking strategy — what to measure

Measure both system and model metrics. Present them in a concise results section on README and in a dedicated docs/metrics.md.

• Cold start time: Time from server start to first token.
• Per-token latency: Average and 95th percentile times.
• Throughput: Tokens/sec at batch size 1 and higher if supported.
• Memory peak: Resident Set Size (RSS) during generation.
• Power draw: Wall power averaged during an inference run (use a USB power meter or UPS HAT).
• Quality: BLEU/ROUGE or human annotated short examples for subjective quality trade-offs.

Reproducibility & CI

Reproducibility is a huge differentiator in portfolios. Include:

• Exact package versions and system packages in a lock file.
• Scripts to reproduce conversion, with checksums and a verification step.
• Small smoke tests in CI (unit tests for helper functions, lints, formatting checks).
• Optional release artifacts: pre-built converted model stored as a GitHub Release or via file hosting; or include checksums and instructions to fetch from a canonical source.

Security, privacy, and ethical notes

Edge AI projects often claim “privacy” as a benefit. Don’t leave it as marketing copy — include a short threat model and practical controls:

• Local-only mode (no cloud access) with an environment variable to enable/disable networking.
• Signed model artifacts and checksum validation on load.
• Minimal data retention: logs should redact user inputs and rotate locally stored transcripts.
• Notes about model license and potential harmful outputs; include guardrails or rate limits.

Presenting the project to reviewers

When you add the project to your portfolio or resume, structure the entry for quick reading:

1. One-line elevator: "On-device LLM assistant on Raspberry Pi 5 with AI HAT+ 2 — 4-bit quantized 7B model, 1.2s avg token latency."
2. Two-sentence summary of the problem you solved and technical choices.
3. Bulleted results: latency, memory, power and a link to a 2-minute demo video hosted on GitHub or an unlisted YouTube link.
4. Repository link with a short instruction: "Run this on Pi 5 in 10 steps" (and include a single command for the reviewer to start the demo if hardware is attached).

Real-world example (case study)

Summary of a concise case study you can include in docs/case-study.md — adapt with your own numbers:

In December 2025 we deployed a 3B parameter model quantized to 4-bit on a Raspberry Pi 5 with AI HAT+ 2. The system averaged 1.35s per token (4 token prompt) and used 3.6GB RAM, with peak power draw of 9.2W during generation. Compared to a cloud baseline, local inference reduced latency variance and eliminated egress costs, while enabling offline operation for privacy-sensitive deployments.

Advanced strategies and 2026 trends to highlight

• Hybrid architectures: Local lightweight model for latency-critical responses and cloud fallback for heavy reasoning — show how you’d namespace and route requests.
• Adaptive quantization: Runtime switching between quantization levels for battery savings — include a demo toggle and log the state changes.
• Model personalization: On-device LoRA-style adapters stored encrypted — explain the privacy and storage strategy.
• Edge orchestration: Note how multiple Pi devices can be coordinated for distributed inference (2025/2026 edge frameworks make this easier) and provide a short design sketch.

Common pitfalls and how to avoid them

• Skipping checksums: Always include file checksums. Reviewers look for reproducibility and tamper-resistance.
• Opaque setup: If your setup requires interactive steps, provide an automated script and a manual checklist.
• Missing metrics: If you claim performance, show measurements and how you collected them.

Deliverables checklist for your repo

• README with quickstart, results, and demo video
• scripts/setup_os.sh and install_runtime.sh
• convert_model.sh with checksums
• runtime/inference_server.py (FastAPI) and a simple web UI
• benchmarks/run_benchmarks.py and sample results
• docs/architecture.md, docs/metrics.md, and hardware/bills_of_materials.md
• Small CI for linting and smoke tests
• LICENSE and CONTRIBUTING.md

Final notes — how reviewers will read your project

Reviewers scan for claims, evidence, and reproducibility. Lead with your metrics, show a 2–3 minute demo video, and make your repo easy to boot. In 2026, being able to say you shipped an on-device LLM with measurable latency, memory, and power figures — and provide the exact commands to reproduce them — is more persuasive than showing a large but vague cloud deployment.

Call to action

Ready to build this for your portfolio? Fork the starter repo (link in the README), follow the 10-step quickstart, and add your compressed benchmark report in docs/metrics.md. When you’re done, include the project on your resume and link the demo video — then send a short note to hiring managers emphasizing the measurable outcomes (latency, memory, power) and reproducibility. If you want, fork our template and share your result link — I’ll review the README and metrics checklist with practical feedback.

Related Reading

• Why Chipmakers Could Make or Break Your Next Tech Job Search
• API Checklist for Building Keyword-Driven Micro-Apps: From Intent Capture to Content Injection
• Teaching Abroad in Southern France: Where to Live, Work Permits and Local Job Boards
• DIY Toy Brand 101: How Small Makers Can Scale from Kitchen Tests to Global Sales
• How Your Mind Learns Japanese: Neuroscience Tips for Faster Vocabulary Retention