← All posts Community

The 70B fine-tune beating a closed model on code

Sep 8, 20263 min read

A reproducible community run with a surprisingly small training set. The thread digs into the data mix and what generalised.

The author fine-tuned an open-weights 70B model on a few thousand curated coding examples and reports scores above a popular closed model on two coding benchmarks.

Commenters reproduced the headline numbers but found the gains narrower on unseen repositories, and most of the thread is a useful discussion of which parts of the data mix mattered.

Key points

Open-weights 70B base, a few thousand curated training examples
Headline results reproduced by others in the thread
Gains are smaller on repositories outside the training distribution