Question about reproducibility of benchmarks (Table 3 of Technical Report)? | 关于技术报告表 3 中基准测试可复现性的问题

#9
by pandemo - opened

Hi MiMo team, thank you so much for open-sourcing such impressive models and sharing your research!

A quick question about reproducing the Table 3 results in the technical report: which agentic harnesses(eg Claude Code, Codex...etc) were used for the Code Agent, General Agent, and Cybersecurity benchmarks? Were MiMo Code or mimoagent used for any of them?

I couldn’t find a benchmark-to-harness mapping /methodology in the report. Any details you can share would be deeply appreciated by the research community🙏!


MiMo 团队好!非常感谢你们开源如此出色的模型,并分享研究成果!

想请教一个关于复现技术报告表 3 结果的问题:Code Agent、General Agent 和 Cybersecurity 这几类基准测试分别使用了哪些智能体运行框架(例如 Claude Code、Codex 等)?其中是否有测试使用了 MiMo Code 或 mimoagent?

我在报告中没有找到各项基准测试与所用框架的对应关系或具体评测方法。如果能分享一些相关信息,研究社区会非常感激🙏

mimo-table3

Suggestion: try a bunch of them, and if they don't work, it was something with either the prompting or an agent change.

Sign up or log in to comment