
ResearchResearch
New Multilingual Medical AI Benchmark Reveals Language and Vision Gaps
Researchers have developed EuropeMedQA, a multilingual and multimodal medical examination dataset drawn from official regulatory exams in Italy, France, Spain, and Portugal. The dataset is designed to evaluate how well large language models perform on non-English medical tasks that include both text and images, addressing a known gap where LLM performance drops significantly outside English-centric benchmarks. The study follows FAIR data principles and includes an automated translation pipeline, with evaluation using zero-shot prompting on contemporary multimodal models to assess cross-lingual transfer and visual reasoning capabilities.
by Francesco Andrea Causio, Vittorio De Vita, Olivia Riccomi, Michele Ferramola, Federico Felizzi, Alessandro Tosi, Antonio Cristiano, Lorenzo De Mori, Chiara Battipaglia, Melissa Sawaya, Luigi De Angelis, Marcello Di Pumpo, Alessandra Piscitelli, Pietro Eric Risuleo, Alessia Longo, Giulia Vojvodic, Mariapia Vassalli, Bianca Destro Castaniti, Nicol\`o Scarsi, Manuel Del Medicoยท ArXiv (cs.AI)
Source