❌

Normal view

Received — 30 December 2025 ⏭ Favourites

DarkPatterns-LLM: A Multi-Layer Benchmark for Detecting Manipulative and Harmful AI Behavior

✇cs.AI, q-bio.NC updates on arXiv.org
By: Sadia Asif · Israel Antonio Rosales Laguan · Haris Khan · Shumaila Asif · Muneeb Asif
30 December 2025 at 13:00
arXiv:2512.22470v1 Announce Type: new Abstract: The proliferation of Large Language Models (LLMs) has intensified concerns about manipulative or deceptive behaviors that can undermine user autonomy, trust, and well-being. Existing safety benchmarks predominantly rely on coarse binary labels and fail to capture the nuanced psychological and social mechanisms constituting manipulation. We introduce \textbf{DarkPatterns-LLM}, a comprehensive benchmark dataset and diagnostic framework for fine-grained assessment of manipulative content in LLM outputs across seven harm categories: Legal/Power, Psychological, Emotional, Physical, Autonomy, Economic, and Societal Harm. Our framework implements a four-layer analytical pipeline comprising Multi-Granular Detection (MGD), Multi-Scale Intent Analysis (MSIAN), Threat Harmonization Protocol (THP), and Deep Contextual Risk Alignment (DCRA). The dataset contains 401 meticulously curated examples with instruction-response pairs and expert annotations. Through evaluation of state-of-the-art models including GPT-4, Claude 3.5, and LLaMA-3-70B, we observe significant performance disparities (65.2\%--89.7\%) and consistent weaknesses in detecting autonomy-undermining patterns. DarkPatterns-LLM establishes the first standardized, multi-dimensional benchmark for manipulation detection in LLMs, offering actionable diagnostics toward more trustworthy AI systems.

Why AI Safety Requires Uncertainty, Incomplete Preferences, and Non-Archimedean Utilities

✇cs.AI, q-bio.NC updates on arXiv.org
By: Alessio Benavoli · Alessandro Facchini · Marco Zaffalon
30 December 2025 at 13:00
arXiv:2512.23508v1 Announce Type: new Abstract: How can we ensure that AI systems are aligned with human values and remain safe? We can study this problem through the frameworks of the AI assistance and the AI shutdown games. The AI assistance problem concerns designing an AI agent that helps a human to maximise their utility function(s). However, only the human knows these function(s); the AI assistant must learn them. The shutdown problem instead concerns designing AI agents that: shut down when a shutdown button is pressed; neither try to prevent nor cause the pressing of the shutdown button; and otherwise accomplish their task competently. In this paper, we show that addressing these challenges requires AI agents that can reason under uncertainty and handle both incomplete and non-Archimedean preferences.

Interpretable Link Prediction in AI-Driven Cancer Research: Uncovering Co-Authorship Patterns

✇cs.AI, q-bio.NC updates on arXiv.org
By: Shahab Mosallaie · Andrea Schiffauerova · Ashkan Ebadi
30 December 2025 at 13:00
arXiv:2512.22181v1 Announce Type: cross Abstract: Artificial intelligence (AI) is transforming cancer diagnosis and treatment. The intricate nature of this disease necessitates the collaboration of diverse stakeholders with varied expertise to ensure the effectiveness of cancer research. Despite its importance, forming effective interdisciplinary research teams remains challenging. Understanding and predicting collaboration patterns can help researchers, organizations, and policymakers optimize resources and foster impactful research. We examined co-authorship networks as a proxy for collaboration within AI-driven cancer research. Using 7,738 publications (2000-2017) from Scopus, we constructed 36 overlapping co-authorship networks representing new, persistent, and discontinued collaborations. We engineered both attribute-based and structure-based features and built four machine learning classifiers. Model interpretability was performed using Shapley Additive Explanations (SHAP). Random forest achieved the highest recall for all three types of examined collaborations. The discipline similarity score emerged as a crucial factor, positively affecting new and persistent patterns while negatively impacting discontinued collaborations. Additionally, high productivity and seniority were positively associated with discontinued links. Our findings can guide the formation of effective research teams, enhance interdisciplinary cooperation, and inform strategic policy decisions.

LLM-Guided Exemplar Selection for Few-Shot Wearable-Sensor Human Activity Recognition

✇cs.AI, q-bio.NC updates on arXiv.org
By: Elsen Ronando · Sozo Inoue
30 December 2025 at 13:00
arXiv:2512.22385v1 Announce Type: cross Abstract: In this paper, we propose an LLM-Guided Exemplar Selection framework to address a key limitation in state-of-the-art Human Activity Recognition (HAR) methods: their reliance on large labeled datasets and purely geometric exemplar selection, which often fail to distinguish similar weara-ble sensor activities such as walking, walking upstairs, and walking downstairs. Our method incorporates semantic reasoning via an LLM-generated knowledge prior that captures feature importance, inter-class confusability, and exemplar budget multipliers, and uses it to guide exemplar scoring and selection. These priors are combined with margin-based validation cues, PageRank centrality, hubness penalization, and facility-location optimization to obtain a compact and informative set of exemplars. Evaluated on the UCI-HAR dataset under strict few-shot conditions, the framework achieves a macro F1-score of 88.78%, outperforming classical approaches such as random sampling, herding, and $k$-center. The results show that LLM-derived semantic priors, when integrated with structural and geometric cues, provide a stronger foundation for selecting representative sensor exemplars in few-shot wearable-sensor HAR.

MedGemma vs GPT-4: Open-Source and Proprietary Zero-shot Medical Disease Classification from Images

✇cs.AI, q-bio.NC updates on arXiv.org
By: Md. Sazzadul Islam Prottasha · Nabil Walid Rafi
30 December 2025 at 13:00
arXiv:2512.23304v1 Announce Type: cross Abstract: Multimodal Large Language Models (LLMs) introduce an emerging paradigm for medical imaging by interpreting scans through the lens of extensive clinical knowledge, offering a transformative approach to disease classification. This study presents a critical comparison between two fundamentally different AI architectures: the specialized open-source agent MedGemma and the proprietary large multimodal model GPT-4 for diagnosing six different diseases. The MedGemma-4b-it model, fine-tuned using Low-Rank Adaptation (LoRA), demonstrated superior diagnostic capability by achieving a mean test accuracy of 80.37% compared to 69.58% for the untuned GPT-4. Furthermore, MedGemma exhibited notably higher sensitivity in high-stakes clinical tasks, such as cancer and pneumonia detection. Quantitative analysis via confusion matrices and classification reports provides comprehensive insights into model performance across all categories. These results emphasize that domain-specific fine-tuning is essential for minimizing hallucinations in clinical implementation, positioning MedGemma as a sophisticated tool for complex, evidence-based medical reasoning.

Generating Verifiable Chain of Thoughts from Exection-Traces

✇cs.AI, q-bio.NC updates on arXiv.org
By: Shailja Thakur · Vaibhav Saxena · Rohan Kulkarni · Shivdeep Singh · Parameswaran Selvam · Hima Patel · Hiroshi Kanayama
30 December 2025 at 13:00
arXiv:2512.00127v2 Announce Type: replace-cross Abstract: Teaching language models to reason about code execution remains a fundamental challenge. While Chain-of-Thought (CoT) prompting has shown promise, current synthetic training data suffers from a critical weakness: the reasoning steps are often plausible-sounding explanations generated by teacher models, not verifiable accounts of what the code actually does. This creates a troubling failure mode where models learn to mimic superficially convincing but logically flawed reasoning patterns. We address this by grounding CoT generation directly in program execution traces. Our pipeline instruments code to capture its dynamic behavior, then narrates these execution traces into natural language and factually-grounded rationales that are verifiable by design. This execution-grounded approach ensures every reasoning step reflects what the program computes, eliminating logical hallucinations at the source. We evaluate our method on code reasoning tasks, code generation and explanation tasks from HumanEval. Models trained on our bi-directional trace-grounded data achieve substantial improvements on reasoning tasks, with gains of up to 30 points on output prediction and 28 points on input prediction over base models, alongside competitive explanation and code generation performance. https://github.ibm.com/IBM-Research-AI/Verified-Code-CoT

Metabolic signatures in gastroenteropancreatic neuroendocrine neoplasms: unraveling diagnostic and prognostic insights

✇(Multiomics OR Omics) AND (Lung OR gastric OR Hepatocellular)
By: Reydson Alcides de Lima-Souza · Tayná Figueiredo Maciel · Moisés Willian Aparecido Gonçalves · Fernanda Cristina Poscai Ribeiro · Rodrigo A Maioral · João Figueira Scarini · Heloisa P Soraes · Gary Chris Fillmore · Erika Said Abu Egal
29 December 2025 at 19:00

Front Endocrinol (Lausanne). 2025 Dec 11;16:1676021. doi: 10.3389/fendo.2025.1676021. eCollection 2025.

ABSTRACT

Gastroenteropancreatic neuroendocrine neoplasms (GEP-NENs) are a heterogeneous group of tumors characterized by diverse biological behaviors and variable clinical outcomes. Recent advances have highlighted the important role of metabolic reprogramming in tumorigenesis, progression, and therapeutic resistance in GEP-NENs. In this review, we synthesize the current evidence on metabolic biomarkers and altered metabolic pathways-particularly those involving glucose, lipid, and amino acid metabolism. Key biomarkers such as GLUT-1, FASN, and enzymes involved in ferroptosis, cholesterol biosynthesis, and amino acid catabolism demonstrate strong associations with tumor aggressiveness, hypoxia, and mTOR signaling. Moreover, metabolomic profiling and functional studies suggest that metabolic markers may inform prognosis and predict response to targeted therapies such as Everolimus. Although promising, the clinical translation of these markers is still limited and requires further validation in large, subtype-specific cohorts. Our findings highlight the importance of integrating metabolic profiling into the diagnostic and therapeutic landscape of GEP-NENs. Future research should prioritize biomarker standardization, multi-omics integration, and the development of metabolism-based therapeutic strategies tailored to tumor subtype and differentiation grade.

PMID:41458541 | PMC:PMC12738315 | DOI:10.3389/fendo.2025.1676021

Received — 28 December 2025 ⏭ Favourites

Evaluating Peer Online Forums to Support Health: Ethical and Practical Challenges

✇Journal of Medical Internet Research
By: Fiona Lobban · Neil Caton · Zoe Glossop · Jade Haines · Gemma Hayward · Connor Heapy · Rose Johnston · Steve Jones · Chris Lodge · Karen Machin · Paul Marshall · Tamara Rakic · Paul Rayson · Heather Robinson · Elena Semino · John Vidler
25 December 2025 at 01:30
Many people use peer online forums to seek support for health-related problems. More research is needed to understand the impacts of forum use, and how these are generated. However, there are significant ethical and practical challenges with the methods available to do the required research. We examine the key challenges associated with conducting each of the most commonly used online data collection methods: surveys, interviews, forum post analysis; and triangulation of these methods. Based on our learning from the Improving Peer Online Forums (iPOF) study, an inter-disciplinary realist informed mixed methods evaluation of peer online forums, we outline strategies that can be used to address key issues pertaining to assessing important outcomes, facilitating participation, validating participants (users who consent to take part in one or more parts of the study), protecting anonymity, gaining consent, managing risk, multi-stakeholder engagement, and triangulation. We share this learning to support researchers, reviewers, and ethics committees faced with deciding how best to address these challenges. We highlight the need for ongoing open, transparent discussion to ensure the research field keeps pace with evolving technology design and societal attitudes to online data use.

Context matching is not reasoning when performing generalized clinical evaluation of generative language models

✇npj Digital Medicine
By: Andrew Wen · Qiuhao Lu · Yu-Neng Chuang · Guanchu Wang · Jiayi Yuan · Jiamu Zhang · Liwei Wang · Sunyang Fu · Kurt D. Miller · Heling Jia · Steven D. Bedrick · William R. Hersh · Kirk E. Roberts · Xia Hu · Hongfang Liu
27 December 2025 at 08:00

npj Digital Medicine, Published online: 27 December 2025; doi:10.1038/s41746-025-02253-2

Context matching is not reasoning when performing generalized clinical evaluation of generative language models

Cancer in a drop: Liquid biopsy highlights from the World Conference on Lung Cancer (WCLC) 2025

✇MRD
By: Nadia Ghazali · Ana Ortega-Franco · Diego de Miguel-Perez · Erick F Saldanha · Konstantinos Venetis · Pasquale Pisapia · Surbhi Singhal · Benjamin A Bleiberg · Carolina Reduzzi · Roberto Borea
24 December 2025 at 19:00

J Liq Biopsy. 2025 Nov 29;10:100449. doi: 10.1016/j.jlb.2025.100449. eCollection 2025 Dec.

ABSTRACT

The role of liquid biopsy in oncological care continues to expand, with multiple studies presented at the International Association for the Study of Lung Cancer (IASLC) 2025 World Conference on Lung Cancer (WCLC 2025). This review summarizes recent advances in liquid biopsy for thoracic oncology, encompassing both non-small cell lung cancer (NSCLC) and small cell lung cancer (SCLC). In early detection and screening, proteomic profiling has identified potential biomarkers predictive of future lung cancer risk. The integration of proteomics with clinical and imaging data can improve pulmonary nodule malignancy prediction. In resectable NSCLC, tumour-informed whole-genome sequencing (WGS) assay demonstrated high sensitivity for minimal residual disease (MRD) detection, with MRD clearance following neoadjuvant osimertinib or chemo-immunotherapy associated with favorable outcomes. In advanced NSCLC, longitudinal liquid biopsy analyses reveal dynamic subclonal evolution driving early treatment resistance. Circulating tumor DNA (ctDNA) clearance following targeted therapy in MET exon 14 skipping and BRAF-mutated tumors was associated with improved clinical outcomes. Emerging biomarkers such as ctDNA tumour fraction and circulating microRNA signatures are promising for radiotherapy stratification and prediction of immunotherapy-related toxicities. In SCLC, MRD monitoring enables earlier detection of disease progression and supports ctDNA-guided selection of patients for consolidation immunotherapy following chemotherapy. Overall, these advances demonstrate the expanding role of liquid biopsy in improving early detection, guiding treatment, and improving disease monitoring in lung cancer.

PMID:41438843 | PMC:PMC12720026 | DOI:10.1016/j.jlb.2025.100449

Received — 19 December 2025 ⏭ Favourites
  • ✇cs.AI, q-bio.NC updates on arXiv.org
  • AI Needs Physics More Than Physics Needs AI Peter Coveney · Roger Highfield
    arXiv:2512.16344v1 Announce Type: new Abstract: Artificial intelligence (AI) is commonly depicted as transformative. Yet, after more than a decade of hype, its measurable impact remains modest outside a few high-profile scientific and commercial successes. The 2024 Nobel Prizes in Chemistry and Physics recognized AI's potential, but broader assessments indicate the impact to date is often more promotional than technical. We argue that while current AI may influence physics, physics has signific
     
  • 🏷️
  • ↗️

AI Needs Physics More Than Physics Needs AI

✇cs.AI, q-bio.NC updates on arXiv.org
By: Peter Coveney · Roger Highfield
19 December 2025 at 13:00
arXiv:2512.16344v1 Announce Type: new Abstract: Artificial intelligence (AI) is commonly depicted as transformative. Yet, after more than a decade of hype, its measurable impact remains modest outside a few high-profile scientific and commercial successes. The 2024 Nobel Prizes in Chemistry and Physics recognized AI's potential, but broader assessments indicate the impact to date is often more promotional than technical. We argue that while current AI may influence physics, physics has significantly more to offer this generation of AI. Current architectures - large language models, reasoning models, and agentic AI - can depend on trillions of meaningless parameters, suffer from distributional bias, lack uncertainty quantification, provide no mechanistic insights, and fail to capture even elementary scientific laws. We review critiques of these limits, highlight opportunities in quantum AI and analogue computing, and lay down a roadmap for the adoption of 'Big AI': a synthesis of theory-based rigour with the flexibility of machine learning.

Distributional AGI Safety

✇cs.AI, q-bio.NC updates on arXiv.org
By: Nenad Toma\v{s}ev · Matija Franklin · Julian Jacobs · S\'ebastien Krier · Simon Osindero
19 December 2025 at 13:00
arXiv:2512.16856v1 Announce Type: new Abstract: AI safety and alignment research has predominantly been focused on methods for safeguarding individual AI systems, resting on the assumption of an eventual emergence of a monolithic Artificial General Intelligence (AGI). The alternative AGI emergence hypothesis, where general capability levels are first manifested through coordination in groups of sub-AGI individual agents with complementary skills and affordances, has received far less attention. Here we argue that this patchwork AGI hypothesis needs to be given serious consideration, and should inform the development of corresponding safeguards and mitigations. The rapid deployment of advanced AI agents with tool-use capabilities and the ability to communicate and coordinate makes this an urgent safety consideration. We therefore propose a framework for distributional AGI safety that moves beyond evaluating and aligning individual agents. This framework centers on the design and implementation of virtual agentic sandbox economies (impermeable or semi-permeable), where agent-to-agent transactions are governed by robust market mechanisms, coupled with appropriate auditability, reputation management, and oversight to mitigate collective risks.

AI4EOSC: a Federated Cloud Platform for Artificial Intelligence in Scientific Research

✇cs.AI, q-bio.NC updates on arXiv.org
By: Ignacio Heredia · \'Alvaro L\'opez Garc\'ia · Germ\'an Molt\'o · Amanda Calatrava · Valentin Kozlov · Alessandro Costantini · Viet Tran · Mario David · Daniel San Mart\'in · Marcin P{\l}\'ociennik · Marta Obreg\'on Ruiz · Sa\'ul Fernandez · Judith S\'ainz-Pardo D\'iaz · Miguel Caballer · Caterina Alarc\'on Mar\'in · Stefan Dlugolinsky · Martin \v{S}eleng · Lisana Berberi · Khadijeh Alibabaei · Borja Esteban Sanchis · Pedro Castro · Giacinto Donvito · Diego Aguirre · Sergio Langarita · Vicente Rodriguez · Leonhard Duda · Andr\'es Heredia Canales · Susana Rebolledo Ruiz · Jo\~ao Machado · Giang Nguyen · Fernando Aguilar G\'omez · Jaime D\'iez
19 December 2025 at 13:00
arXiv:2512.16455v1 Announce Type: cross Abstract: In this paper, we describe a federated compute platform dedicated to support Artificial Intelligence in scientific workloads. Putting the effort into reproducible deployments, it delivers consistent, transparent access to a federation of physically distributed e-Infrastructures. Through a comprehensive service catalogue, the platform is able to offer an integrated user experience covering the full Machine Learning lifecycle, including model development (with dedicated interactive development environments), training (with GPU resources, annotation tools, experiment tracking, and federated learning support) and deployment (covering a wide range of deployment options all along the Cloud Continuum). The platform also provides tools for traceability and reproducibility of AI models, integrates with different Artificial Intelligence model providers, datasets and storage resources, allowing users to interact with the broader Machine Learning ecosystem. Finally, it is easily customizable to lower the adoption barrier by external communities.

Plausibility as Failure: How LLMs and Humans Co-Construct Epistemic Error

✇cs.AI, q-bio.NC updates on arXiv.org
By: Claudia Vale Oliveira (DigiMedia · University of Aveiro · Aveiro · Portugal) · Nelson Zagalo (DigiMedia · University of Aveiro · Aveiro · Portugal) · Filipe Silva (DigiMedia · University of Aveiro · Aveiro · Portugal) · Anabela Brandao (DigiMedia · University of Aveiro · Aveiro · Portugal) · Syeda Faryal Hussain Khurrum (DigiMedia · University of Aveiro · Aveiro · Portugal) · Joaquim Santos (DigiMedia · University of Aveiro · Aveiro · Portugal)
19 December 2025 at 13:00
arXiv:2512.16750v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as epistemic partners in everyday reasoning, yet their errors remain predominantly analyzed through predictive metrics rather than through their interpretive effects on human judgment. This study examines how different forms of epistemic failure emerge, are masked, and are tolerated in human AI interaction, where failure is understood as a relational breakdown shaped by model-generated plausibility and human interpretive judgment. We conducted a three round, multi LLM evaluation using interdisciplinary tasks and progressively differentiated assessment frameworks to observe how evaluators interpret model responses across linguistic, epistemic, and credibility dimensions. Our findings show that LLM errors shift from predictive to hermeneutic forms, where linguistic fluency, structural coherence, and superficially plausible citations conceal deeper distortions of meaning. Evaluators frequently conflated criteria such as correctness, relevance, bias, groundedness, and consistency, indicating that human judgment collapses analytical distinctions into intuitive heuristics shaped by form and fluency. Across rounds, we observed a systematic verification burden and cognitive drift. As tasks became denser, evaluators increasingly relied on surface cues, allowing erroneous yet well formed answers to pass as credible. These results suggest that error is not solely a property of model behavior but a co-constructed outcome of generative plausibility and human interpretive shortcuts. Understanding AI epistemic failure therefore requires reframing evaluation as a relational interpretive process, where the boundary between system failure and human miscalibration becomes porous. The study provides implications for LLM assessment, digital literacy, and the design of trustworthy human AI communication.

Constitutional Law and AI Governance: Constraints on Model Licensing and Research Classification

✇cs.AI, q-bio.NC updates on arXiv.org
By: Alex Mark · Aaron Scher
19 December 2025 at 13:00
arXiv:2509.05361v2 Announce Type: replace-cross Abstract: Transformative AI systems may pose unprecedented catastrophic risks, but the U.S. Constitution places significant constraints on the government's ability to govern this technology. This paper examines how the First Amendment, administrative law, and the Fourteenth Amendment shape the legal vulnerability of two regulatory proposals: model licensing and AI research classification. While the First Amendment may provide some degree of protection for model algorithms or outputs, this protection does not foreclose regulation. Policymakers must also consider administrative legal requirements, due to both agency review and authority. Finally, while substantive due process and equal protection pose minimal obstacles, procedural due process requires the government to clearly define when developers vest a legal interest in their models. Given this analysis, effective AI governance requires careful implementation to avoid these legal challenges.

First, do NOHARM: towards clinically safe large language models

✇cs.AI, q-bio.NC updates on arXiv.org
By: David Wu · Fateme Nateghi Haredasht · Saloni Kumar Maharaj · Priyank Jain · Jessica Tran · Matthew Gwiazdon · Arjun Rustagi · Jenelle Jindal · Jacob M. Koshy · Vinay Kadiyala · Anup Agarwal · Bassman Tappuni · Brianna French · Sirus Jesudasen · Christopher V. Cosgriff · Rebanta Chakraborty · Jillian Caldwell · Susan Ziolkowski · David J. Iberri · Robert Diep · Rahul S. Dalal · Kira L. Newman · Kristin Galetta · J. Carl Pallais · Nancy Wei · Kathleen M. Buchheit · David I. Hong · Ernest Y. Lee · Allen Shih · Vartan Pahalyants · Tamara B. Kaplan · Vishnu Ravi · Sarita Khemani · April S. Liang · Daniel Shirvani · Advait Patil · Nicholas Marshall · Kanav Chopra · Joel Koh · Adi Badhwar · Liam G. McCoy · David J. H. Wu · Yingjie Weng · Sumant Ranji · Kevin Schulman · Nigam H. Shah · Jason Hom · Arnold Milstein · Adam Rodman · Jonathan H. Chen · Ethan Goh
19 December 2025 at 13:00
arXiv:2512.01241v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are routinely used by physicians and patients for medical advice, yet their clinical safety profiles remain poorly characterized. We present NOHARM (Numerous Options Harm Assessment for Risk in Medicine), a benchmark using 100 real primary care-to-specialist consultation cases to measure frequency and severity of harm from LLM-generated medical recommendations. NOHARM covers 10 specialties, with 12,747 expert annotations for 4,249 clinical management options. Across 31 LLMs, potential for severe harm from LLM recommendations occurs in up to 22.2% (95% CI 21.6-22.8%) of cases, with harm of omission accounting for 76.6% (95% CI 76.4-76.8%) of errors. Safety performance is only moderately correlated (r = 0.61-0.64) with existing AI and medical knowledge benchmarks. The best models outperform generalist physicians on safety (mean difference 9.7%, 95% CI 7.0-12.5%), and a diverse multi-agent approach improves safety compared to solo models (mean difference 8.0%, 95% CI 4.0-12.1%). Therefore, despite strong performance on existing evaluations, widely used AI models can produce severely harmful medical advice at nontrivial rates, underscoring clinical safety as a distinct performance dimension necessitating explicit measurement.
Received — 18 December 2025 ⏭ Favourites

DrugRAG: Enhancing Pharmacy LLM Performance Through A Novel Retrieval-Augmented Generation Pipeline

✇cs.AI, q-bio.NC updates on arXiv.org
By: Houman Kazemzadeh · Kiarash Mokhtari Dizaji · Seyed Reza Tavakoli · Farbod Davoodi · MohammadReza KarimiNejad · Parham Abed Azad · Ali Sabzi · Armin Khosravi · Siavash Ahmadi · Mohammad Hossein Rohban · Glolamali Aminian · Tahereh Javaheri
18 December 2025 at 13:00
arXiv:2512.14896v1 Announce Type: cross Abstract: Objectives: To evaluate large language model (LLM) performance on pharmacy licensure-style question-answering (QA) tasks and develop an external knowledge integration method to improve their accuracy. Methods: We benchmarked eleven existing LLMs with varying parameter sizes (8 billion to 70+ billion) using a 141-question pharmacy dataset. We measured baseline accuracy for each model without modification. We then developed a three-step retrieval-augmented generation (RAG) pipeline, DrugRAG, that retrieves structured drug knowledge from validated sources and augments model prompts with evidence-based context. This pipeline operates externally to the models, requiring no changes to model architecture or parameters. Results: Baseline accuracy ranged from 46% to 92%, with GPT-5 (92%) and o3 (89%) achieving the highest scores. Models with fewer than 8 billion parameters scored below 50%. DrugRAG improved accuracy across all tested models, with gains ranging from 7 to 21 percentage points (e.g., Gemma 3 27B: 61% to 71%, Llama 3.1 8B: 46% to 67%) on the 141-item benchmark. Conclusion: We demonstrate that external structured drug knowledge integration through DrugRAG measurably improves LLM accuracy on pharmacy tasks without modifying the underlying models. This approach provides a practical pipeline for enhancing pharmacy-focused AI applications with evidence-based information.

Leveraging LLMs for Structured Data Extraction from Unstructured Patient Records

✇cs.AI, q-bio.NC updates on arXiv.org
By: Mitchell A. Klusty · Elizabeth C. Solie · Caroline N. Leach · W. Vaiden Logan · Lynnet E. Richey · John C. Gensel · David P. Szczykutowicz · Bryan C. McLellan · Emily B. Collier · Samuel E. Armstrong · V. K. Cody Bumgardner
17 December 2025 at 13:00
arXiv:2512.13700v1 Announce Type: new Abstract: Manual chart review remains an extremely time-consuming and resource-intensive component of clinical research, requiring experts to extract often complex information from unstructured electronic health record (EHR) narratives. We present a secure, modular framework for automated structured feature extraction from clinical notes leveraging locally deployed large language models (LLMs) on institutionally approved, Health Insurance Portability and Accountability Act (HIPPA)-compliant compute infrastructure. This system integrates retrieval augmented generation (RAG) and structured response methods of LLMs into a widely deployable and scalable container to provide feature extraction for diverse clinical domains. In evaluation, the framework achieved high accuracy across multiple medical characteristics present in large bodies of patient notes when compared against an expert-annotated dataset and identified several annotation errors missed in manual review. This framework demonstrates the potential of LLM systems to reduce the burden of manual chart review through automated extraction and increase consistency in data capture, accelerating clinical research.
❌