NLP · From-scratch LLM
Question Generator — SLM
A 114.1M-parameter decoder-only transformer trained from scratch on ~3B tokens, then fine-tuned to turn any text into exam-style questions.
Demo
Watch it in action
The walkthrough video will live here — script, output, and runtime demo in one take.
Recording in progress
Project details
What it does, end to end
- Full LLM pipeline end-to-end: raw corpus collection, preprocessing and tokenization, pretraining, supervised fine-tuning, inference, evaluation, and demo tooling.
- 12-layer decoder-only transformer, 768 hidden size, GQA attention (12 query heads / 4 KV heads), SwiGLU feed-forward, RoPE position encoding, 4096-token context.
- Pretrained on Wikipedia, OpenWebText, Gutenberg, and Medium for ~3.0B tokens (9,156 steps, final validation loss ≈ 2.71–2.77), then fine-tuned on 83,861 SQuAD v2 + HotpotQA samples.
- Split into question styles — short_answer and complex — selectable at inference time via an interactive CLI.
2025 — 2026
Tech stack
Built with
Architecture
How it works
Problem & solution
Why it exists
Problem
Generating good practice questions from study material usually means paying per-API-call, leaking content to third parties, or having no tool at all. Students deserve a private, one-shot generator.
Solution
Train a compact language model from scratch — own data pipeline, own architecture, own checkpoints — and fine-tune it specifically for question generation, then publish everything so it can be reproduced.
Technical details
Under the hood
- Custom object-oriented architecture in PyTorch — attention, layers, and model config are all own code, no transformer library.
- tokenizer: tiktoken r50k_base vocabulary plus 3 special tokens
- Efficient data-loading and packing for ~3B tokens; training run on an H100 GPU.
- Inference with KV-cache decoding for faster generation; torch.compile optional for predictable GPU behavior.
- Published on the Hugging Face Hub: SFT + pretrained checkpoints and both datasets.
Impacts
What changed
- Model (114M) and SFT dataset publicly available on Hugging Face — reproducible from the repo.
- Turns any lecture text into practice questions — short answering handled best, per evaluation.
- BERTScore F1 0.8945 on SQuAD v2 validation (50-sample eval).
- Used as the demo model at presentations and project showcases.
Learnings
Lessons that stuck
- 01OOP and clean module boundaries pay off in long training projects — architecture, data, training, and inference each live in their own package.
- 02Checkpoint hygiene matters: kept archived stable checkpoints as fallback so demos never broke on an experimental run.
- 03Small budgets focus the mind: 114M params, one H100, ~3B tokens — every design decision had to earn its cost.
- 04Evaluating a language model takes as much care as training it — BERTScore beats eyeballing samples.
Challenges & mistakes
What broke, and what it taught me
Mistake 01
MCQ and reasoning question quality came out uneven — short answers were strong, reasoning and MCQ weaker.
Lesson learned
Scope the model's promise: ship what it does best (short-answer generation), and document the rest honestly as limitations.
Mistake 02
Latest SFT runs sometimes regressed over archived checkpoints at demo time.
Lesson learned
Keep an archived stable checkpoint as the default demo path and treat the newest run as an experiment until re-validated.
Mistake 03
torch.compile caused unpredictable GPU behavior mid-training.
Lesson learned
Make speedups opt-in (USE_TORCH_COMPILE=1) — determinism beats best-case latency in a training run.