sutd-reranker-ft67

Finetuned cross-encoder/ms-marco-MiniLM-L6-v2 for reranking retrieved SUTD modules in the SUTD Course Recommendation Chatbot (MLOps Group 9).

Given a (job description, module passage) pair, this model predicts a relevance score used to rerank candidate modules before LLM answer generation.

Model Details

Property	Value
Base model	cross-encoder/ms-marco-MiniLM-L6-v2
Max sequence length	512
Output	Single relevance score
Loss	BinaryCrossEntropyLoss

Training Data

Finetuned on hard negatives mined by henreads/sutd-bge-large-ft67. For each of the 67 hand-annotated training jobs, the top-30 retrieved modules (excluding ground-truth positives) were used as hard negatives. After expansion this produces 1833 training rows and 180 validation rows.

A version trained on the augmented 98-pair dataset (with negatives mined by sutd-bge-large-aug98) is available at henreads/sutd-reranker-aug98.

Training Setup

• Hardware: Modal A10G (24 GB VRAM)
• Training time: ~30 seconds (0.008 GPU hours)
• Epochs: 5
• Batch size: 32
• Learning rate: 2e-5
• Hard negatives: top-30 per job via sutd-bge-large-ft67, excluding positives
• Train/val split: job-level 10% validation
• Tracking: Weights & Biases (sutd-mlops-reranker-finetune)

Evaluation

This reranker is most effective when paired with its matched embedding model. Key findings from ablation:

Config	Chat Quality (Overall 1-5)
RAG, base reranker, k=15	3.885
RAG, sutd-reranker-ft67, aug98 BGE, k=15	4.025

Important: this reranker was trained on candidates from sutd-bge-large-ft67. When used in the Hybrid pipeline (which adds PE candidates not seen during training), it underperforms the base reranker. Use henreads/sutd-reranker-aug98 for the Hybrid pipeline.

Usage

from sentence_transformers import CrossEncoder

model = CrossEncoder("henreads/sutd-reranker-ft67")

job_description = "Data Scientist at GovTech. Build ML models with Python..."
module_passage = "50.007 Machine Learning — Topics: supervised learning, neural networks..."

score = model.predict([[job_description, module_passage]])
print(score)

Project

Part of the SUTD Course Recommendation Chatbot — MLOps Group 9.
Code: github.com/henreads/sutd-mlops-group9