❌

Normal view

Received — 19 December 2025 ⏭ Favourites

Small Language Models for Efficient Agentic Tool Calling: Outperforming Large Models with Targeted Fine-tuning

✇cs.AI, q-bio.NC updates on arXiv.org
By: Polaris Jhandi · Owais Kazi · Shreyas Subramanian · Neel Sendas
19 December 2025 at 13:00
arXiv:2512.15943v1 Announce Type: new Abstract: As organizations scale adoption of generative AI, model cost optimization and operational efficiency have emerged as critical factors determining sustainability and accessibility. While Large Language Models (LLMs) demonstrate impressive capabilities across diverse tasks, their extensive computational requirements make them cost-prohibitive for routine enterprise use. This limitation motivates the exploration of Small Language Models (SLMs), which can deliver comparable performance in targeted applications while drastically reducing infrastructure overhead (Irugalbandara et al., 2023). In this work, we investigate the feasibility of replacing LLM-driven workflows with optimized SLMs. We trained a domain-adapted SLM to execute representative tasks traditionally handled by LLMs, such as document summarization, query answering, and structured data interpretation. As part of the experiment, we investigated the fine-tuning of facebook/opt-350m model (single epoch only) using the Hugging Face TRL (Transformer Reinforcement Learning), specifically the Supervised Fine-Tuning (SFT) trainer. The OPT-350M model was released by Meta AI in 2022 as part of the OPT (Open Pretrained Transformer) family of models. Similar studies demonstrate that even models at the 350M parameter scale can meaningfully contribute to instruction-tuning pipelines (Mekala et al., 2024). Experimental results demonstrated that our fine-tuned SLM achieves exceptional performance with a 77.55\% pass rate on ToolBench evaluation, significantly outperforming all baseline models including ChatGPT-CoT (26.00\%), ToolLLaMA-DFS (30.18\%), and ToolLLaMA-CoT (16.27\%). These findings emphasize that thoughtful design and targeted training of SLMs can significantly lower barriers to adoption, enabling cost-effective, large-scale integration of generative AI into production systems.

Distributional AGI Safety

✇cs.AI, q-bio.NC updates on arXiv.org
By: Nenad Toma\v{s}ev · Matija Franklin · Julian Jacobs · S\'ebastien Krier · Simon Osindero
19 December 2025 at 13:00
arXiv:2512.16856v1 Announce Type: new Abstract: AI safety and alignment research has predominantly been focused on methods for safeguarding individual AI systems, resting on the assumption of an eventual emergence of a monolithic Artificial General Intelligence (AGI). The alternative AGI emergence hypothesis, where general capability levels are first manifested through coordination in groups of sub-AGI individual agents with complementary skills and affordances, has received far less attention. Here we argue that this patchwork AGI hypothesis needs to be given serious consideration, and should inform the development of corresponding safeguards and mitigations. The rapid deployment of advanced AI agents with tool-use capabilities and the ability to communicate and coordinate makes this an urgent safety consideration. We therefore propose a framework for distributional AGI safety that moves beyond evaluating and aligning individual agents. This framework centers on the design and implementation of virtual agentic sandbox economies (impermeable or semi-permeable), where agent-to-agent transactions are governed by robust market mechanisms, coupled with appropriate auditability, reputation management, and oversight to mitigate collective risks.

Emergent Bias and Fairness in Multi-Agent Decision Systems

✇cs.AI, q-bio.NC updates on arXiv.org
By: Maeve Madigan · Parameswaran Kamalaruban · Glenn Moynihan · Tom Kempton · David Sutton · Stuart Burrell
19 December 2025 at 13:00
arXiv:2512.16433v1 Announce Type: cross Abstract: Multi-agent systems have demonstrated the ability to improve performance on a variety of predictive tasks by leveraging collaborative decision making. However, the lack of effective evaluation methodologies has made it difficult to estimate the risk of bias, making deployment of such systems unsafe in high stakes domains such as consumer finance, where biased decisions can translate directly into regulatory breaches and financial loss. To address this challenge, we need to develop fairness evaluation methodologies for multi-agent predictive systems and measure the fairness characteristics of these systems in the financial tabular domain. Examining fairness metrics using large-scale simulations across diverse multi-agent configurations, with varying communication and collaboration mechanisms, we reveal patterns of emergent bias in financial decision-making that cannot be traced to individual agent components, indicating that multi-agent systems may exhibit genuinely collective behaviors. Our findings highlight that fairness risks in financial multi-agent systems represent a significant component of model risk, with tangible impacts on tasks such as credit scoring and income estimation. We advocate that multi-agent decision systems must be evaluated as holistic entities rather than through reductionist analyses of their constituent components.

AI4EOSC: a Federated Cloud Platform for Artificial Intelligence in Scientific Research

✇cs.AI, q-bio.NC updates on arXiv.org
By: Ignacio Heredia · \'Alvaro L\'opez Garc\'ia · Germ\'an Molt\'o · Amanda Calatrava · Valentin Kozlov · Alessandro Costantini · Viet Tran · Mario David · Daniel San Mart\'in · Marcin P{\l}\'ociennik · Marta Obreg\'on Ruiz · Sa\'ul Fernandez · Judith S\'ainz-Pardo D\'iaz · Miguel Caballer · Caterina Alarc\'on Mar\'in · Stefan Dlugolinsky · Martin \v{S}eleng · Lisana Berberi · Khadijeh Alibabaei · Borja Esteban Sanchis · Pedro Castro · Giacinto Donvito · Diego Aguirre · Sergio Langarita · Vicente Rodriguez · Leonhard Duda · Andr\'es Heredia Canales · Susana Rebolledo Ruiz · Jo\~ao Machado · Giang Nguyen · Fernando Aguilar G\'omez · Jaime D\'iez
19 December 2025 at 13:00
arXiv:2512.16455v1 Announce Type: cross Abstract: In this paper, we describe a federated compute platform dedicated to support Artificial Intelligence in scientific workloads. Putting the effort into reproducible deployments, it delivers consistent, transparent access to a federation of physically distributed e-Infrastructures. Through a comprehensive service catalogue, the platform is able to offer an integrated user experience covering the full Machine Learning lifecycle, including model development (with dedicated interactive development environments), training (with GPU resources, annotation tools, experiment tracking, and federated learning support) and deployment (covering a wide range of deployment options all along the Cloud Continuum). The platform also provides tools for traceability and reproducibility of AI models, integrates with different Artificial Intelligence model providers, datasets and storage resources, allowing users to interact with the broader Machine Learning ecosystem. Finally, it is easily customizable to lower the adoption barrier by external communities.

From Facts to Conclusions : Integrating Deductive Reasoning in Retrieval-Augmented LLMs

✇cs.AI, q-bio.NC updates on arXiv.org
By: Shubham Mishra · Samyek Jain · Gorang Mehrishi · Shiv Tiwari · Harsh Sharma · Pratik Narang · Dhruv Kumar
19 December 2025 at 13:00
arXiv:2512.16795v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) grounds large language models (LLMs) in external evidence, but fails when retrieved sources conflict or contain outdated or subjective information. Prior work address these issues independently but lack unified reasoning supervision. We propose a reasoning-trace-augmented RAG framework that adds structured, interpretable reasoning across three stages : (1) document-level adjudication, (2) conflict analysis, and (3) grounded synthesis, producing citation-linked answers or justified refusals. A Conflict-Aware Trust-Score (CATS) pipeline is introduced which evaluates groundedness, factual correctness, refusal accuracy, and conflict-behavior alignment using an LLM-as-a-Judge. Our 539-query reasoning dataset and evaluation pipeline establish a foundation for conflict-aware, interpretable RAG systems. Experimental results demonstrate substantial gains over baselines, most notably with Qwen, where Supervised Fine-Tuning improved End-to-End answer correctness from 0.069 to 0.883 and behavioral adherence from 0.074 to 0.722.

Constitutional Law and AI Governance: Constraints on Model Licensing and Research Classification

✇cs.AI, q-bio.NC updates on arXiv.org
By: Alex Mark · Aaron Scher
19 December 2025 at 13:00
arXiv:2509.05361v2 Announce Type: replace-cross Abstract: Transformative AI systems may pose unprecedented catastrophic risks, but the U.S. Constitution places significant constraints on the government's ability to govern this technology. This paper examines how the First Amendment, administrative law, and the Fourteenth Amendment shape the legal vulnerability of two regulatory proposals: model licensing and AI research classification. While the First Amendment may provide some degree of protection for model algorithms or outputs, this protection does not foreclose regulation. Policymakers must also consider administrative legal requirements, due to both agency review and authority. Finally, while substantive due process and equal protection pose minimal obstacles, procedural due process requires the government to clearly define when developers vest a legal interest in their models. Given this analysis, effective AI governance requires careful implementation to avoid these legal challenges.

First, do NOHARM: towards clinically safe large language models

✇cs.AI, q-bio.NC updates on arXiv.org
By: David Wu · Fateme Nateghi Haredasht · Saloni Kumar Maharaj · Priyank Jain · Jessica Tran · Matthew Gwiazdon · Arjun Rustagi · Jenelle Jindal · Jacob M. Koshy · Vinay Kadiyala · Anup Agarwal · Bassman Tappuni · Brianna French · Sirus Jesudasen · Christopher V. Cosgriff · Rebanta Chakraborty · Jillian Caldwell · Susan Ziolkowski · David J. Iberri · Robert Diep · Rahul S. Dalal · Kira L. Newman · Kristin Galetta · J. Carl Pallais · Nancy Wei · Kathleen M. Buchheit · David I. Hong · Ernest Y. Lee · Allen Shih · Vartan Pahalyants · Tamara B. Kaplan · Vishnu Ravi · Sarita Khemani · April S. Liang · Daniel Shirvani · Advait Patil · Nicholas Marshall · Kanav Chopra · Joel Koh · Adi Badhwar · Liam G. McCoy · David J. H. Wu · Yingjie Weng · Sumant Ranji · Kevin Schulman · Nigam H. Shah · Jason Hom · Arnold Milstein · Adam Rodman · Jonathan H. Chen · Ethan Goh
19 December 2025 at 13:00
arXiv:2512.01241v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are routinely used by physicians and patients for medical advice, yet their clinical safety profiles remain poorly characterized. We present NOHARM (Numerous Options Harm Assessment for Risk in Medicine), a benchmark using 100 real primary care-to-specialist consultation cases to measure frequency and severity of harm from LLM-generated medical recommendations. NOHARM covers 10 specialties, with 12,747 expert annotations for 4,249 clinical management options. Across 31 LLMs, potential for severe harm from LLM recommendations occurs in up to 22.2% (95% CI 21.6-22.8%) of cases, with harm of omission accounting for 76.6% (95% CI 76.4-76.8%) of errors. Safety performance is only moderately correlated (r = 0.61-0.64) with existing AI and medical knowledge benchmarks. The best models outperform generalist physicians on safety (mean difference 9.7%, 95% CI 7.0-12.5%), and a diverse multi-agent approach improves safety compared to solo models (mean difference 8.0%, 95% CI 4.0-12.1%). Therefore, despite strong performance on existing evaluations, widely used AI models can produce severely harmful medical advice at nontrivial rates, underscoring clinical safety as a distinct performance dimension necessitating explicit measurement.
Received — 18 December 2025 ⏭ Favourites

A Decision-Theoretic Approach for Managing Misalignment

✇cs.AI, q-bio.NC updates on arXiv.org
By: Daniel A. Herrmann · Abinav Chari · Isabelle Qian · Sree Sharvesh · B. A. Levinstein
18 December 2025 at 13:00
arXiv:2512.15584v1 Announce Type: new Abstract: When should we delegate decisions to AI systems? While the value alignment literature has developed techniques for shaping AI values, less attention has been paid to how to determine, under uncertainty, when imperfect alignment is good enough to justify delegation. We argue that rational delegation requires balancing an agent's value (mis)alignment with its epistemic accuracy and its reach (the acts it has available). This paper introduces a formal, decision-theoretic framework to analyze this tradeoff precisely accounting for a principal's uncertainty about these factors. Our analysis reveals a sharp distinction between two delegation scenarios. First, universal delegation (trusting an agent with any problem) demands near-perfect value alignment and total epistemic trust, conditions rarely met in practice. Second, we show that context-specific delegation can be optimal even with significant misalignment. An agent's superior accuracy or expanded reach may grant access to better overall decision problems, making delegation rational in expectation. We develop a novel scoring framework to quantify this ex ante decision. Ultimately, our work provides a principled method for determining when an AI is aligned enough for a given context, shifting the focus from achieving perfect alignment to managing the risks and rewards of delegation under uncertainty.

DrugRAG: Enhancing Pharmacy LLM Performance Through A Novel Retrieval-Augmented Generation Pipeline

✇cs.AI, q-bio.NC updates on arXiv.org
By: Houman Kazemzadeh · Kiarash Mokhtari Dizaji · Seyed Reza Tavakoli · Farbod Davoodi · MohammadReza KarimiNejad · Parham Abed Azad · Ali Sabzi · Armin Khosravi · Siavash Ahmadi · Mohammad Hossein Rohban · Glolamali Aminian · Tahereh Javaheri
18 December 2025 at 13:00
arXiv:2512.14896v1 Announce Type: cross Abstract: Objectives: To evaluate large language model (LLM) performance on pharmacy licensure-style question-answering (QA) tasks and develop an external knowledge integration method to improve their accuracy. Methods: We benchmarked eleven existing LLMs with varying parameter sizes (8 billion to 70+ billion) using a 141-question pharmacy dataset. We measured baseline accuracy for each model without modification. We then developed a three-step retrieval-augmented generation (RAG) pipeline, DrugRAG, that retrieves structured drug knowledge from validated sources and augments model prompts with evidence-based context. This pipeline operates externally to the models, requiring no changes to model architecture or parameters. Results: Baseline accuracy ranged from 46% to 92%, with GPT-5 (92%) and o3 (89%) achieving the highest scores. Models with fewer than 8 billion parameters scored below 50%. DrugRAG improved accuracy across all tested models, with gains ranging from 7 to 21 percentage points (e.g., Gemma 3 27B: 61% to 71%, Llama 3.1 8B: 46% to 67%) on the 141-item benchmark. Conclusion: We demonstrate that external structured drug knowledge integration through DrugRAG measurably improves LLM accuracy on pharmacy tasks without modifying the underlying models. This approach provides a practical pipeline for enhancing pharmacy-focused AI applications with evidence-based information.

MedChat: A Multi-Agent Framework for Multimodal Diagnosis with Large Language Models

✇cs.AI, q-bio.NC updates on arXiv.org
By: Philip R. Liu · Sparsh Bansal · Jimmy Dinh · Aditya Pawar · Ramani Satishkumar · Shail Desai · Neeraj Gupta · Xin Wang · Shu Hu
18 December 2025 at 13:00
arXiv:2506.07400v3 Announce Type: replace-cross Abstract: The integration of deep learning-based glaucoma detection with large language models (LLMs) presents an automated strategy to mitigate ophthalmologist shortages and improve clinical reporting efficiency. However, applying general LLMs to medical imaging remains challenging due to hallucinations, limited interpretability, and insufficient domain-specific medical knowledge, which can potentially reduce clinical accuracy. Although recent approaches combining imaging models with LLM reasoning have improved reporting, they typically rely on a single generalist agent, restricting their capacity to emulate the diverse and complex reasoning found in multidisciplinary medical teams. To address these limitations, we propose MedChat, a multi-agent diagnostic framework and platform that combines specialized vision models with multiple role-specific LLM agents, all coordinated by a director agent. This design enhances reliability, reduces hallucination risk, and enables interactive diagnostic reporting through an interface tailored for clinical review and educational use. Code available at https://github.com/Purdue-M2/MedChat.

Multimodal Foundation Models for Early Disease Detection

✇cs.AI, q-bio.NC updates on arXiv.org
By: Md Talha Mohsin · Ismail Abdulrashid
18 December 2025 at 13:00
arXiv:2510.01899v2 Announce Type: replace-cross Abstract: Healthcare data now span EHRs, medical imaging, genomics, and wearable sensors, but most diagnostic models still process these modalities in isolation. This limits their ability to capture early, cross-modal disease signatures. This paper introduces a multimodal foundation model built on a transformer architecture that integrates heterogeneous clinical data through modality-specific encoders and cross-modal attention. Each modality is mapped into a shared latent space and fused using multi-head attention with residual normalization. We implement the framework using a multimodal dataset that simulates early-stage disease patterns across EHR sequences, imaging patches, genomic profiles, and wearable signals, including missing-modality scenarios and label noise. The model is trained using supervised classification together with self-supervised reconstruction and contrastive alignment to improve robustness. Experimental evaluation demonstrates strong performance in early-detection settings, with stable classification metrics, reliable uncertainty estimates, and interpretable attention patterns. The approach moves toward a flexible, pretrain-and-fine-tune foundation model that supports precision diagnostics, handles incomplete inputs, and improves early disease detection across oncology, cardiology, and neurology applications.

A novel statistical feature selection framework for biomarker discovery and cancer classification via multiomics integration

✇(Multiomics OR Omics) AND (Lung OR gastric OR Hepatocellular)
By: Moshira S Ghaleb · Maryam N Al-Berry · Hala M Ebied · Mohamed F Tolba
17 December 2025 at 19:00

BMC Med Res Methodol. 2025 Dec 17. doi: 10.1186/s12874-025-02713-z. Online ahead of print.

ABSTRACT

BACKGROUND: Early cancer diagnosis is essential for improving prognosis and guiding treatment. However, the high dimensionality and complexity of omics data present major challenges. Computational approaches that extract stable biomarkers and enable reliable classification across cancer types and stages are needed.

METHODS: A novel feature selection method, sDCFE (synergistic Discriminative Cluster-based Feature Extraction), was developed by extending Fisher-like variance analysis with a median absolute deviation (MAD) regularization term and a cluster separation component to enhance robustness and interpretability. Features selected by sDCFE were compared with those obtained from XGBoost, and the intersected set of 82 genes was evaluated through functional enrichment (KEGG, Reactome, GO BP), survival analysis (Kaplan-Meier, Cox regression), and biomarker novelty assessment against six external resources. Hybrid classification models integrating XGBoost, sDCFE, and deep learning were applied to pancancer classification, and the framework was further extended to lung squamous cell carcinoma (LUSC) staging using RNA-seq and methylation data.

RESULTS: The overlap between sDCFE and XGBoost yielded 82 candidate biomarkers enriched in cancer-related pathways, including cell cycle regulation, immune signalling, and DNA repair. Novelty assessment stratified these genes into established, emerging, and novel categories. Six genes-HFE2, LOC339674, SERINC2, SFTA3, SOX2OT, and ACPP-emerged as the most promising candidates, supported by enrichment and survival associations across multiple cancers. The hybrid model achieved near-perfect pancancer classification on TCGA (accuracy = 99.3%, MCC = 0.992, AUC = 1.0) and demonstrated strong generalizability on PCAWG (accuracy = 94%, MCC = 0.929, AUC = 0.997). In the LUSC staging task, multiomics integration improved classification performance: the CNN-based model reached 84% accuracy, while logistic regression applied to sDCFE-ranked features achieved 88.5% accuracy with superior calibration, highlighting the robustness of the selected features.

CONCLUSION: sDCFE provides a principled extension of Fisher-like methods, enabling stable and interpretable biomarker selection. When combined with XGBoost and deep learning, the framework achieves highly accurate and biologically grounded cancer classification across both cancer types and stages. The identification of novel and prognostic biomarkers, including HFE2, LOC339674, SERINC2, SFTA3, SOX2OT, and ACPP, underscores its translational potential. These results position the framework as a promising precision oncology tool to support early diagnosis, risk stratification, and treatment decision-making.

PMID:41408184 | DOI:10.1186/s12874-025-02713-z

Enhancing Transparency and Traceability in Healthcare AI: The AI Product Passport

✇cs.AI, q-bio.NC updates on arXiv.org
By: A. Anil Sinaci · Senan Postaci · Dogukan Cavdaroglu · Machteld J. Boonstra · Okan Mercan · Kerem Yilmaz · Gokce B. Laleci Erturkmen · Folkert W. Asselbergs · Karim Lekadir
17 December 2025 at 13:00
arXiv:2512.13702v1 Announce Type: cross Abstract: Objective: To develop the AI Product Passport, a standards-based framework improving transparency, traceability, and compliance in healthcare AI via lifecycle-based documentation. Materials and Methods: The AI Product Passport was developed within the AI4HF project, focusing on heart failure AI tools. We analyzed regulatory frameworks (EU AI Act, FDA guidelines) and existing standards to design a relational data model capturing metadata across AI lifecycle phases: study definition, dataset preparation, model generation/evaluation, deployment/monitoring, and passport generation. MLOps/ModelOps concepts were integrated for operational relevance. Co-creation involved feedback from AI4HF consortium and a Lisbon workshop with 21 diverse stakeholders, evaluated via Mentimeter polls. The open-source platform was implemented with Python libraries for automated provenance tracking. Results: The AI Product Passport was designed based on existing standards and methods with well-defined lifecycle management and role-based access. Its implementation is a web-based platform with a relational data model supporting auditable documentation. It generates machine- and human-readable reports, customizable for stakeholders. It aligns with FUTURE-AI principles (Fairness, Universality, Traceability, Usability, Robustness, Explainability), ensuring fairness, traceability, and usability. Exported passports detail model purpose, data provenance, performance, and deployment context. GitHub-hosted backend/frontend codebases enhance accessibility. Discussion and Conclusion: The AI Product Passport addresses transparency gaps in healthcare AI, meeting regulatory and ethical demands. Its open-source nature and alignment with standards foster trust and adaptability. Future enhancements include FAIR data principles and FHIR integration for improved interoperability, promoting responsible AI deployment.

Graph AI generates neurological hypotheses validated in molecular, organoid, and clinical systems

✇cs.AI, q-bio.NC updates on arXiv.org
By: Ayush Noori · Joaqu\'in Polonuer · Katharina Meyer · Bogdan Budnik · Shad Morton · Xinyuan Wang · Sumaiya Nazeen · Yingnan He · I\~naki Arango · Lucas Vittor · Matthew Woodworth · Richard C. Krolewski · Michelle M. Li · Ninning Liu · Tushar Kamath · Evan Macosko · Dylan Ritter · Jalwa Afroz · Alexander B. H. Henderson · Lorenz Studer · Samuel G. Rodriques · Andrew White · Noa Dagan · David A. Clifton · George M. Church · Sudeshna Das · Jenny M. Tam · Vikram Khurana · Marinka Zitnik
17 December 2025 at 13:00
arXiv:2512.13724v1 Announce Type: cross Abstract: Neurological diseases are the leading global cause of disability, yet most lack disease-modifying treatments. We present PROTON, a heterogeneous graph transformer that generates testable hypotheses across molecular, organoid, and clinical systems. To evaluate PROTON, we apply it to Parkinson's disease (PD), bipolar disorder (BD), and Alzheimer's disease (AD). In PD, PROTON linked genetic risk loci to genes essential for dopaminergic neuron survival and predicted pesticides toxic to patient-derived neurons, including the insecticide endosulfan, which ranked within the top 1.29% of predictions. In silico screens performed by PROTON reproduced six genome-wide $\alpha$-synuclein experiments, including a split-ubiquitin yeast two-hybrid system (normalized enrichment score [NES] = 2.30, FDR-adjusted $p

Criminal Liability in AI-Enabled Autonomous Vehicles: A Comparative Study

✇cs.AI, q-bio.NC updates on arXiv.org
By: Sahibpreet Singh · Manjit Singh
17 December 2025 at 13:00
arXiv:2512.14330v1 Announce Type: cross Abstract: AI revolutionizes transportation through autonomous vehicles (AVs) but introduces complex criminal liability issues regarding infractions. This study employs a comparative legal analysis of primary statutes, real-world liability claims, and academic literature across the US, Germany, UK, China, and India; jurisdictions selected for their technological advancement and contrasting regulatory approaches. The research examines the attribution of human error, AI moral agency, and the identification of primary offenders in AV incidents. Findings reveal fragmented regulatory landscapes: India and the US rely on loose networks of state laws, whereas the UK enacted the pioneering Automated and Electric Vehicles Act 2018. Germany enforces strict safety standards, distinguishing liability based on the vehicle's operating mode, while China similarly aims for a stringent liability regime. The study concludes that globally harmonized legal standards are essential to foster technological innovation while ensuring minimum risk and clear liability attribution.

A Multicenter Benchmark of Multiple Instance Learning Models for Lymphoma Subtyping from HE-stained Whole Slide Images

✇cs.AI, q-bio.NC updates on arXiv.org
By: Rao Muhammad Umer · Daniel Sens · Jonathan Noll · Christian Matek · Lukas Wolfseher · Rainer Spang · Ralf Huss · Johannes Raffler · Sarah Reinke · Wolfram Klapper · Katja Steiger · Kristina Schwamborn · Carsten Marr
17 December 2025 at 13:00
arXiv:2512.14640v1 Announce Type: cross Abstract: Timely and accurate lymphoma diagnosis is essential for guiding cancer treatment. Standard diagnostic practice combines hematoxylin and eosin (HE)-stained whole slide images with immunohistochemistry, flow cytometry, and molecular genetic tests to determine lymphoma subtypes, a process requiring costly equipment, skilled personnel, and causing treatment delays. Deep learning methods could assist pathologists by extracting diagnostic information from routinely available HE-stained slides, yet comprehensive benchmarks for lymphoma subtyping on multicenter data are lacking. In this work, we present the first multicenter lymphoma benchmarking dataset covering four common lymphoma subtypes and healthy control tissue. We systematically evaluate five publicly available pathology foundation models (H-optimus-1, H0-mini, Virchow2, UNI2, Titan) combined with attention-based (AB-MIL) and transformer-based (TransMIL) multiple instance learning aggregators across three magnifications (10x, 20x, 40x). On in-distribution test sets, models achieve multiclass balanced accuracies exceeding 80% across all magnifications, with all foundation models performing similarly and both aggregation methods showing comparable results. The magnification study reveals that 40x resolution is sufficient, with no performance gains from higher resolutions or cross-magnification aggregation. However, on out-of-distribution test sets, performance drops substantially to around 60%, highlighting significant generalization challenges. To advance the field, larger multicenter studies covering additional rare lymphoma subtypes are needed. We provide an automated benchmarking pipeline to facilitate such future research.

COMMA: A Communicative Multimodal Multi-Agent Benchmark

✇cs.AI, q-bio.NC updates on arXiv.org
By: Timothy Ossowski · Danyal Maqbool · Jixuan Chen · Zefan Cai · Tyler Bradshaw · Junjie Hu
17 December 2025 at 13:00
arXiv:2410.07553v5 Announce Type: replace Abstract: The rapid advances of multimodal agents built on large foundation models have largely overlooked their potential for language-based communication between agents in collaborative tasks. This oversight presents a critical gap in understanding their effectiveness in real-world deployments, particularly when communicating with humans. Existing agentic benchmarks fail to address key aspects of inter-agent communication and collaboration, particularly in scenarios where agents have unequal access to information and must work together to achieve tasks beyond the scope of individual capabilities. To fill this gap, we introduce COMMA: a novel puzzle benchmark designed to evaluate the collaborative performance of multimodal multi-agent systems through language communication. Our benchmark features a variety of multimodal puzzles, providing a comprehensive evaluation across four key categories of agentic capability in a communicative collaboration setting. Our findings reveal surprising weaknesses in state-of-the-art models, including strong proprietary models like GPT-4o and reasoning models like o4-mini. Many chain of thought reasoning models such as R1-Onevision and LLaVA-CoT struggle to outperform even a random baseline in agent-agent collaboration, indicating a potential growth area in their communication abilities.

A Knowledge Graph-based Retrieval-Augmented Generation Framework for Algorithm Selection in the Facility Layout Problem

✇cs.AI, q-bio.NC updates on arXiv.org
By: Nikhil N S (Indian Institute of Science · Bengaluru · India) · Bilal Muhammed (TCS Research and Innovation · Tata Consultancy Services Ltd) · Soban Babu Beemaraj (TCS Research and Innovation · Tata Consultancy Services Ltd) · Amol Dilip Joshi (TCS Research and Innovation · Tata Consultancy Services Ltd)
17 December 2025 at 13:00
arXiv:2509.18054v2 Announce Type: replace-cross Abstract: Selecting a solution algorithm for the Facility Layout Problem (FLP), an NP-hard optimization problem with multiobjective trade-off, is a complex task that requires deep expert knowledge. The performance of a given algorithm depends on the specific characteristics of the problem, such as the number of facilities, objectives, and constraints. This creates a need for a data-driven recommendation method to guide algorithm selection in automated design systems. This paper introduces a new recommendation method to make this expertise accessible, based on a Knowledge Graph-Based Retrieval-Augmented Generation (KG-RAG) framework. In this framework, a domain-specific knowledge graph (KG) is constructed from the literature. The method then employs a multifaceted retrieval mechanism to gather relevant evidence from this KG using three distinct approaches: precise graph-based search, flexible vector-based search, and cluster-based high-level search. The retrieved evidence is utilized by a Large Language Model (LLM) to generate algorithm recommendations based on data-driven reasoning. This KG-RAG framework is tested on a use case consisting of six problems comprising of complex multi-objective and multi-constraint FLP case. The results are compared with the Gemini 1.5 Flash chatbot. The results show that KG-RAG achieves an average reasoning score of 4.7 out of 5 compared to 3.3 for the baseline chatbot.

Beyond Task Completion: An Assessment Framework for Evaluating Agentic AI Systems

✇cs.AI, q-bio.NC updates on arXiv.org
By: Sreemaee Akshathala · Bassam Adnan · Mahisha Ramesh · Karthik Vaidhyanathan · Basil Muhammed · Kannan Parthasarathy
17 December 2025 at 13:00
arXiv:2512.12791v2 Announce Type: replace-cross Abstract: Recent advances in agentic AI have shifted the focus from standalone Large Language Models (LLMs) to integrated systems that combine LLMs with tools, memory, and other agents to perform complex tasks. These multi-agent architectures enable coordinated reasoning, planning, and execution across diverse domains, allowing agents to collaboratively automate complex workflows. Despite these advances, evaluation and assessment of LLM agents and the multi-agent systems they constitute remain a fundamental challenge. Although various approaches have been proposed in the software engineering literature for evaluating conventional software components, existing methods for AI-based systems often overlook the non-deterministic nature of models. This non-determinism introduces behavioral uncertainty during execution, yet existing evaluations rely on binary task completion metrics that fail to capture it. Evaluating agentic systems therefore requires examining additional dimensions, including the agent ability to invoke tools, ingest and retrieve memory, collaborate with other agents, and interact effectively with its environment. These challenges emerged during our ongoing industry collaboration with MontyCloud Inc., when we deployed an agentic system in production. These limitations surfaced during deployment, highlighting practical gaps in the current evaluation methods and the need for a systematic assessment of agent behavior beyond task outcomes. Informed by these observations and established definitions of agentic systems, we propose an end-to-end Agent Assessment Framework with four evaluation pillars encompassing LLMs, Memory, Tools, and Environment. We validate the framework on a representative Autonomous CloudOps use case, where experiments reveal behavioral deviations overlooked by conventional metrics, demonstrating its effectiveness in capturing runtime uncertainties.
❌