Skip to content
Back to projects

NLP · From-scratch LLM

Question Generator - SLM

A 114.1M-parameter decoder-only transformer trained from scratch on ~3B tokens, then fine-tuned to turn any text into exam-style questions.

Code
PythonPyTorchH100Hugging Face

2025 - 2026

What it does, end to end

  • Full LLM pipeline end-to-end: raw corpus collection, preprocessing and tokenization, pretraining, supervised fine-tuning, inference, evaluation, and demo tooling.
  • 12-layer decoder-only transformer, 768 hidden size, GQA attention (12 query heads / 4 KV heads), SwiGLU feed-forward, RoPE position encoding, 4096-token context.
  • Pretrained on Wikipedia, OpenWebText, Gutenberg, and Medium for ~3.0B tokens (9,156 steps, final validation loss ≈ 2.71–2.77), then fine-tuned on 83,861 SQuAD v2 + HotpotQA samples.
  • Split into question styles - short_answer and complex - selectable at inference time via an interactive CLI.

Tech stack

PythonPyTorchH100Hugging Face

How it works

Why it exists

Problem

Generating good practice questions from study material usually means paying per-API-call, leaking content to third parties, or having no tool at all. Students deserve a private, one-shot generator.

Solution

Train a compact language model from scratch - own data pipeline, own architecture, own checkpoints - and fine-tune it specifically for question generation, then publish everything so it can be reproduced.

Under the hood

  • Custom object-oriented architecture in PyTorch - attention, layers, and model config are all own code, no transformer library.
  • tokenizer: tiktoken r50k_base vocabulary plus 3 special tokens
  • Efficient data-loading and packing for ~3B tokens; training run on an H100 GPU.
  • Inference with KV-cache decoding for faster generation; torch.compile optional for predictable GPU behavior.
  • Published on the Hugging Face Hub: SFT + pretrained checkpoints and both datasets.

What changed

  • Model (114M) and SFT dataset publicly available on Hugging Face - reproducible from the repo.
  • Turns any lecture text into practice questions - short answering handled best, per evaluation.
  • BERTScore F1 0.8945 on SQuAD v2 validation (50-sample eval).
  • Used as the demo model at presentations and project showcases.

Lessons that stuck

  1. 01OOP and clean module boundaries pay off in long training projects - architecture, data, training, and inference each live in their own package.
  2. 02Checkpoint hygiene matters: kept archived stable checkpoints as fallback so demos never broke on an experimental run.
  3. 03Small budgets focus the mind: 114M params, one H100, ~3B tokens - every design decision had to earn its cost.
  4. 04Evaluating a language model takes as much care as training it - BERTScore beats eyeballing samples.

What broke, and what it taught me

Mistake 01

MCQ and reasoning question quality came out uneven - short answers were strong, reasoning and MCQ weaker.

Lesson learned

Scope the model's promise: ship what it does best (short-answer generation), and document the rest honestly as limitations.

Mistake 02

Latest SFT runs sometimes regressed over archived checkpoints at demo time.

Lesson learned

Keep an archived stable checkpoint as the default demo path and treat the newest run as an experiment until re-validated.

Mistake 03

torch.compile caused unpredictable GPU behavior mid-training.

Lesson learned

Make speedups opt-in (USE_TORCH_COMPILE=1) - determinism beats best-case latency in a training run.

Back to projects