The ultimate days of the AI Mathematical Olympiad’s newest competitors had been a transcontinental relay for group NVIDIA.
Each night, two group members on reverse ends of the U.S. would submit an AI reasoning mannequin to Kaggle — the net Olympics of information science and machine studying. They’d wait a tense 5 hours earlier than studying how effectively the mannequin tackled a pattern set of fifty complicated math issues.
After seeing the outcomes, the U.S. group would move the baton to teammates waking up in Armenia, Finland, Germany and Northern Eire, who would spend their day testing, modifying and optimizing completely different mannequin variations.
“Each night time I’d be so disillusioned in our rating, however then I’d get up and see the messages that got here in in a single day from teammates in Europe,” stated Igor Gitman, senior utilized scientist. “My hopes would go up and we’d strive once more.”
Whereas the group was disheartened by their lack of enchancment on the general public dataset in the course of the competitors’s last days, the actual check of an AI mannequin is how effectively it may well generalize to unseen knowledge. That’s the place their reasoning mannequin leapt to the highest of the leaderboard — appropriately answering 34 out of fifty Olympiad questions inside a five-hour time restrict utilizing a cluster of 4 NVIDIA L4 GPUs.
“We bought the magic ultimately,” stated Northern Eire-based group member Darragh Hanley, a Kaggle grandmaster and senior giant language mannequin (LLM) technologist.
Constructing a Successful Equation
The NVIDIA group competed beneath the identify NemoSkills — a nod to their use of the NeMo-Abilities assortment of pipelines for accelerated LLM coaching, analysis and inference. The seven members every contributed completely different areas of experience, spanning LLM coaching, mannequin distillation and inference optimization.
For the Kaggle problem, over 2,200 collaborating groups submitted AI fashions tasked with fixing 50 math questions — complicated issues on the Nationwide Olympiad degree, spanning algebra, geometry, combinatorics and quantity principle — inside 5 hours.
The group’s successful mannequin makes use of a mixture of pure language reasoning and Python code execution.
To finish this inference problem on the small cluster of NVIDIA L4 GPUs obtainable through Kaggle, the NemoSkills group needed to get artistic.
Their successful mannequin used Qwen2.5-14B-Base, a basis mannequin with chain-of-thought reasoning capabilities which the group fine-tuned on thousands and thousands of synthetically generated options to math issues.
These artificial options had been primarily generated by two bigger reasoning fashions — DeepSeek-R1 and QwQ-32B — and used to show the group’s basis mannequin through a type of information distillation. The top outcome was a smaller, quicker, long-thinking mannequin able to tackling complicated issues utilizing a mixture of pure language reasoning and Python code execution.
To additional enhance efficiency, the group’s resolution causes by means of a number of long-thinking responses in parallel earlier than figuring out a last reply. To optimize this course of and meet the competitors’s time restrict, the group additionally used an modern early-stopping approach.
A reasoning mannequin would possibly, for instance, be set to reply a math downside 12 completely different occasions earlier than choosing the commonest response. Utilizing the asynchronous processing capabilities of NeMo-Abilities and NVIDIA TensorRT-LLM, the group was capable of monitor and exit inference early if the mannequin had already converged on the right reply 4 or extra occasions.
TensorRT-LLM additionally enabled the group to harness FP8 quantization, a compression methodology that resulted in a 1.5x speedup over utilizing the extra generally used FP16 format. ReDrafter, a speculative decoding approach developed by Apple, was used for an additional 1.8x speedup.
The ultimate mannequin carried out even higher on the competitors’s unseen last dataset than it did on the general public dataset — an indication that the group efficiently constructed a generalizable mannequin and prevented overfitting their LLM to the pattern knowledge.
“Even with out the Kaggle competitors, we’d nonetheless be working to enhance AI reasoning fashions for math,” stated Gitman. “However Kaggle offers us the chance to benchmark and uncover how effectively our fashions generalize to a third-party dataset.”
Sharing the Wealth
The group will quickly launch a technical report detailing the methods used of their successful resolution — and plans to share their dataset and a collection of fashions on Hugging Face. The developments and optimizations they revamped the course of the competitors have been built-in into NeMo-Abilities pipelines obtainable on GitHub.
Key knowledge, know-how, and insights from this pipeline had been additionally used to coach the just-released NVIDIA Llama Nemotron Extremely mannequin.
“All through this collaboration, we used instruments throughout the NVIDIA software program stack,” stated Christof Henkel, a member of the Kaggle Grandmasters of NVIDIA, often known as KGMON. “By working intently with our LLM analysis and improvement groups, we’re capable of take what we study from the competitors on a day-to-day foundation and push these optimizations into NVIDIA’s open-source libraries.”
After the competitors win, Henkel regained the title of Kaggle World Champion — rating No. 1 among the many platform’s over 23 million customers. One other teammate, Finland-based Ivan Sorokin, earned the Kaggle Grandmaster title, held by simply over 350 individuals all over the world.
For his or her first-place win, the group additionally received a $262,144 prize that they’re directing to the NVIDIA Basis to help charitable organizations.
Meet the total group — Igor Gitman, Darragh Hanley, Christof Henkel, Ivan Moshkov, Benedikt Schifferer, Ivan Sorokin and Shubham Toshniwal — within the video beneath:
Pattern math questions within the featured visible above are from the 2025 American Invitational Arithmetic Examination. Discover the total set of questions and options on the Artwork of Downside Fixing wiki.
