❌

Normal view

Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievals

arXiv:2502.16101v3 Announce Type: replace Abstract: Retrieval-augmented generation (RAG) has shown impressive capabilities in mitigating hallucinations in large language models (LLMs). However, LLMs struggle to maintain consistent reasoning when exposed to misleading or conflicting evidence, especially in real-world domains such as politics, where information is polarized or selectively framed. Mainstream RAG benchmarks evaluate models under clean retrieval settings, where systems generate answers from gold-standard documents, or under synthetically perturbed settings, where documents are artificially injected with noise. These assumptions fail to reflect real-world conditions, often leading to an overestimation of RAG system performance. To address this gap, we introduce RAGuard, the first benchmark to evaluate the robustness of RAG systems against misleading retrievals. Unlike prior benchmarks that rely on synthetic noise, our fact-checking dataset captures naturally occurring misinformation by constructing its retrieval corpus from Reddit discussions. It categorizes retrieved evidence into three types: supporting, misleading, and unrelated, providing a realistic and challenging testbed for assessing how well RAG systems navigate different types of evidence. Our experiments reveal that, when exposed to potentially misleading retrievals, all tested LLM-powered RAG systems perform worse than their zero-shot baselines (i.e., no retrieval at all), while human annotators consistently perform better, highlighting LLMs' susceptibility to noisy environments. To our knowledge, RAGuard is the first benchmark to systematically assess the robustness of the RAG against misleading evidence. We expect this benchmark to drive future research toward improving RAG systems beyond idealized datasets, making them more reliable for real-world applications. The dataset is available at https://huggingface.co/datasets/UCSC-IRKM/RAGuard.

The Impact of Digital Health Interventions on Psychological Health, Self-Efficacy, and Quality of Life in Patients With End-Stage Kidney Disease: Systematic Review and Meta-Analysis

Background: End-stage kidney disease (ESKD) imposes a significant global health burden, with patients often experiencing poor quality of life (QoL) due to psychological distress and low self-efficacy. Digital health interventions (DHIs) offer potential to address these challenges. However, their effects in this population remain inconsistent, and a comprehensive synthesis of the evidence is lacking. Objective: To assess the impact of DHIs on the psychological health, self-efficacy, and QoL of ESKD patients, and to evaluate engagement, adherence and satisfaction with these interventions. Methods: A comprehensive search was conducted across six electronic databases (PubMed, Web of Science, Cochrane Library, PsycINFO, Embase, and CINAHL) up to January 21, 2025. Randomized controlled trials (RCTs) examining DHIs effects on psychological health, self-efficacy, or QoL in ESKD patients were included. Two reviewers independently screened studies, extracted data, and assessed the risk of bias using the RoB 2 tool. Meta-analysis was performed using Review Manager 5.4, with subgroup analyses by treatment modality, intervention type, and duration. Evidence quality was assessed using the GRADE approach. Results: 23 RCTs involving 2407 ESKD patients from 12 countries were included. DHIs significantly improved depression (SMD: -0.41, 95% CI: [-0.63, -0.19], P=.003) and overall QoL (SMD: 0.55, 95% CI: [0.07, 1.03], P=.03). While DHIs did not significantly improve overall self-efficacy (SMD: 0.56, 95% CI: [-0.06, 1.18], P=.08), a benefit was observed in hemodialysis patients (SMD: 0.59, 95% CI: [0.34, 0.83], P<.001 engagement was favorable with completion rates above adherence of and generally positive patient feedback on dhis. application-based interventions improved self-efficacy ci: p overall qol telemedicine depression video-based due to high heterogeneity risk bias evidence quality rated as low for moderate general anxiety very stress self-efficacy. conclusions: dhis can significantly improve the psychological health eskd patients particularly when tailored needs delivered through interactive platforms such applications telemedicine. suggest good acceptability in clinical practice. however warrants cautious interpretation. future research should involve more high-quality rcts design that address unique elderly peritoneal dialysis kidney transplant recipients. trial: prospero crd42024629357 https:>

Association between pace of biological aging and cancer and the modulating role of physical activity: a national cross-sectional study

Cancer remains a serious public health problem impeding gains in life expectancy. Epigenetic clocks, derived from sets of DNA methylation CpGs and mathematical algorithms, have demonstrated a remarkable abilit...

Ferroptosis in lung cancer: a novel pathway regulating cell death and a promising target for drug therapy

Cell Death Discovery, Published online: 01 April 2023; doi:10.1038/s41420-023-01407-z

Ferroptosis in lung cancer: a novel pathway regulating cell death and a promising target for drug therapy
❌