Data-efficient evaluation and co-optimized recommendation of large language models for practical tasks.
Choosing the right LLM for a task is expensive: full evaluation across benchmarks does not scale. This project develops methods to evaluate models with far less data (via subset selection driven by rank-correlation prediction) and to co-optimize recommendation and evaluation, so that model selection is both cheap and accurate — including for scientific use cases.
Best Theme Paper Award; Oral presentation (top 1% of accepted papers).
@inproceedings{saranathan2025sublime,title={SubLIME: Subset Selection via Rank Correlation Prediction for Data-Efficient LLM Evaluation},author={Saranathan, Gayathri and Xu, Cong and Alam, Mahammad Parwez and Kumar, Tarun and Foltin, Martin and Wong, Soon Yee and Bhattacharya, Suparna},booktitle={Annual Meeting of the Association for Computational Linguistics (ACL)},year={2025},}
ICLR-W
Co-optimizing Recommendation and Evaluation for LLM Selection
Tarun Kumar, Cong Xu, Arpit Shah, and 3 more authors
In ICLR 2025 Workshop on Foundation Models in the Wild, 2025
@inproceedings{kumar2025cooptimizing,title={Co-optimizing Recommendation and Evaluation for LLM Selection},author={Kumar, Tarun and Xu, Cong and Shah, Arpit and Diallo, Baradji and Foltin, Martin and Bhattacharya, Suparna},booktitle={ICLR 2025 Workshop on Foundation Models in the Wild},year={2025},}
2024
CUG
System for Recommendation and Evaluation of Large Language Models for Practical Tasks in Science
Cong Xu, Tarun Kumar, Martin Foltin, and 7 more authors
@inproceedings{xu2024llmrecommendation,title={System for Recommendation and Evaluation of Large Language Models for Practical Tasks in Science},author={Xu, Cong and Kumar, Tarun and Foltin, Martin and Justine, Annmary and Serebryakov, Sergey and Shah, Arpit and Mishra, Ashish and Nayak, Gyanaranjan and Bhattacharya, Suparna and Faraboschi, Paolo},booktitle={Cray User Group (CUG) Conference},year={2024},}