Nikhil C R
Counsel Assist
Llama 3.1 8B fine-tuned with QLoRA to write structured support plans.
Problem
CounselChat has several free-form counselor answers per question. The project condenses them into a fixed plan (core issue, coping strategies, immediate actions, long-term focus) and trains an 8B model on a single consumer GPU to produce that structure directly.
Architecture
860 filtered examples (774 train / 86 held out) train LoRA adapters on the 4-bit base, merged to 16-bit and served from the Hub.
- Dataset prepdataset_prep.ipynb
Groups answers per question, generates plans with GPT-3.5, keeps only complete four-section plans.
- Fine-tuningllm_finetuning.ipynb
Unsloth + TRL on one RTX 4060 Ti, logged to Weights & Biases.
- Inferenceinference.ipynb
transformers pipeline loading the published model.
Key decisions
QLoRA on every projection
LoRA r=16, alpha=16 on q/k/v/o/gate/up/down projections with rsLoRA over a 4-bit base: 41.9M trainable parameters (0.52%).
llm_finetuning.ipynb · cells 5–6Training setup for a 16 GB GPU
lr 1e-4 linear, effective batch 16 (8 × 2 accumulation), 2 epochs (96 steps), 8-bit AdamW, bf16 where supported, eval every 10% of steps.
llm_finetuning.ipynb · cells 18–21Synthetic labels with a completeness filter
All answers to a question go to GPT-3.5 with retries and backoff; a plan is kept only if all four sections are present and none is empty or a placeholder.
dataset_prep.ipynb · cells 7–13
Evaluation
| What | How it was measured | Result |
|---|---|---|
| Validation loss | SFTTrainer on the 86-example held-out split, every ~10 steps over 96 steps, logged to W&B. Token loss only; targets are GPT-3.5-written. llm_finetuning.ipynb · cell 21 output | 1.467 → 1.219, falling at every eval |
| Training loss | Same run, steps 10 → 90. llm_finetuning.ipynb · cell 21 output | 1.387 → 1.195 |
Run it
dataset_prep.ipynb → llm_finetuning.ipynb → inference.ipynb