A new test from OpenAI aims to understand how close AI is to outperforming humans at economically valuable work.

Background: Large language models (LLMs) hold promise for supporting clinical tasks, particularly in data-driven and technical disciplines such as radiation oncology. While prior evaluation studies have focused on examination-style settings for evaluating LLMs, their performance in real-life clinical scenarios remains unclear. In the future, LLMs might be used as general AI assistants to answer questions arising in clinical practice. It is unclear how well a modern LLM, locally executed within the infrastructure of a hospital, would answer such questions compared with clinical experts. Objective: This study aimed to assess the performance of a locally deployed, state-of-the-art medical LLM in answering real-world clinical questions in radiation oncology compared with clinical experts. The aim was to evaluate the overall quality of answers, as well as the potential harmfulness of the answers if used for clinical decision-making. Methods: Physicians from 10 departments of European hospitals collected questions arising in the clinical practice of radiation oncology. Fifty of these questions were answered by 3 senior radiation oncology experts with at least 10 years of work experience, as well as the LLM Llama3-OpenBioLLM-70B (Ankit Pal and Malaikannan Sankarasubbu). In a blinded review, physicians rated the overall answer quality on a 5-point Likert scale (quality), assessed whether an answer might be potentially harmful if used for clinical decision-making (harmfulness), and determined if responses were from an expert or the LLM (recognizability). Comparisons between clinical experts and LLMs were then made for quality, harmfulness, and recognizability. Results: There were no significant differences between the quality of the answers between LLM and clinical experts (mean scores of 3.38 vs 3.63; median 4.00, IQR 3.00-4.00 vs median 3.67, IQR 3.33-4.00; P=.26; Wilcoxon signed rank test). The answers were deemed potentially harmful in 13% of cases for the clinical experts compared with 16% of cases for the LLM (P=.63; Fisher exact test). Physicians correctly identified whether an answer was given by a clinical expert or an LLM in 78% and 72% of cases, respectively. Conclusions: A state-of-the-art medical LLM can answer real-life questions from the clinical practice of radiation oncology similarly well as clinical experts regarding overall quality and potential harmfulness. Such LLMs can already be deployed within the local hospital environment at an affordable cost. While LLMs may not yet be ready for clinical implementation as general AI assistants, the technology continues to improve at a rapid pace. Evaluation studies based on real-life situations are important to better understand the weaknesses and limitations of LLMs in clinical practice. Such studies are also crucial to define when the technology is ready for clinical implementation. Furthermore, education for health care professionals on generative AI is needed to ensure responsible clinical implementation of this transforming technology.

OpenAI is continuing to build out massive AI data centers to train and serve increasingly powerful AI models.

Background: Artificial intelligence (AI) and Large Language models (LLMs) are increasing in sophistication and are being integrated into many disciplines. The potential for LLMs to augment clinical decisions is an evolving area of research. Objective: This study compared the responses of over 1000 kidney specialist physicians (nephrologists) to outputs of commonly used LLMs using a questionnaire determining when a kidney biopsy should be performed. Methods: This research group completed a large online questionnaire for nephrologists to determine when a kidney biopsy should be performed. The questionnaire was co-designed with patient participation, refined through multiple iterations, then piloted locally before international dissemination. It was the largest international study in the field and demonstrated variation between human clinicians in biopsy propensity relating to human factors such as sex and age, as well as systemic factors such as country, job seniority and technical proficiency. The same questions were put to both human doctors and LLMs in an identical order in a single session. Eight commonly used LLMs were interrogated: Chat GPT 3.5, Mistral Hugging Face, Perplexity, Microsoft Co-pilot, Llama 2, GPT 4.0, MedLM and Claude 3. The most common response given by clinicians (human mode) to each question was taken as the baseline for comparison. Questionnaire responses to the indications and contraindications for biopsy generated a score (0-44) reflecting biopsy propensity, in which a higher score was used as a surrogate marker for an increased tolerance of potential associated risks. Results: The ability of LLMs to reproduce human expert consensus varied widely with some models demonstrating a balanced approach to risk in a similar manner to humans, whilst other models reported outputs at either end of the spectrum for risk tolerance. In terms of agreement with the human mode, Chat GPT 3.5 and GPT 4.0 (Open AI) had the highest levels of alignment, with the human mode selected in 6/11 questions. The total biopsy propensity score generated from the human mode was 23/44. Both Open AI models produced similar propensity scores between 22 and 24, however Llama 2 and MS Co-pilot also reported scores within this range, but with poorer response alignment to the human mode at only 2/11 questions. The most risk averse model in this study was MedLM with a propensity score of 11 and the least risk averse model was Claude 3 with a score of 34. Conclusions: LLM outputs demonstrated a modest ability to replicate human clinical decision making in this study, however the performance varied widely between LLM models. Questions with more uniform human responses produced LLM outputs with greater alignment, whereas in questions with low levels of human consensus there was poor output alignment. This may limit the practical use of LLMs in real world clinical practice.

Single-cell transcriptomics analysis and multimodal profiling (STAMP) by imaging enables single-cell analysis of cells in suspension without the need for sequencing. The markedly reduced costs and flexible experimental designs support the profiling of millions of cells or the large-scale multiplexing of conditions, perturbations, and sample types.

npj Digital Medicine, Published online: 30 August 2025; doi:10.1038/s41746-025-01945-z

Enabling secure and self determined health data sharing and consent management

We developed a plain text modeling language—a cell behavior hypothesis grammar—to easily build virtual cell models and connect them to data, helping scientists to unlock the hidden dynamics of tissues. We provide examples showing how to use them in virtual experiments exploring how cancer responds to the cells in its environment and how the brain forms layers in development.

Gemini is Google’s long-promised, next-gen generative AI model family.

© 2024 TechCrunch. All rights reserved. For personal use only.

OpenAI used the subreddit, r/ChangeMyView, to create a test for measuring the persuasive abilities of its AI reasoning models. The company revealed this in a system card — a document outlining how an AI system works — that was released along with its new “reasoning” model, o3-mini, on Friday. Millions of Reddit users are members […]

© 2024 TechCrunch. All rights reserved. For personal use only.

Nature, Published online: 15 January 2025; doi:10.1038/d41586-025-00052-z

The incoming US president is expected to gut support for research on the environment and infectious diseases, but could buoy work in artificial intelligence, quantum research and space exploration.

Transposable element exonization by unannotated splicing events produces stable protein isoforms with acquired functions that are subject to evolutionary selection.

Cancer biomarkers have transformed oncology, enabling treatments tailored to each tumor’s unique profile. This review highlights the field’s progress due to advancements in understanding cancer biology, testing methods, and understanding of the immune microenvironment to advance precision oncology for improved patient outcomes.

Nature, Published online: 23 February 2024; doi:10.1038/d41586-024-00568-w

Debate over figure connecting genes, race and ethnicity reignites concerns among geneticists about how to represent human diversity.

Nature, Published online: 08 February 2024; doi:10.1038/s41586-024-07051-6

Author Correction: Genotyping, sequencing and analysis of 140,000 adults from Mexico City

Nature, Published online: 07 December 2023; doi:10.1038/d41586-023-03898-3

The new bargaining group will represent thousands of early-career scientists at the US National Institutes of Health.

Nature, Published online: 06 December 2023; doi:10.1038/d41586-023-03821-w

One organ in a person’s body can age faster than the rest — with implications for health and mortality.

Nature, Published online: 25 October 2023; doi:10.1038/d41586-023-03272-3

A neural-network-based artificial intelligence outperforms ChatGPT at quickly folding new words into its lexicon, a key aspect of human intelligence.

Oncogene, Published online: 24 October 2023; doi:10.1038/s41388-023-02857-6

Translation of tissue-based artificial intelligence into clinical practice: from discovery to adoption

Nature, Published online: 21 June 2023; doi:10.1038/s41586-023-06254-7

A murine colorectal cancer (CRC) model shows that mutant KRAS-STAT4-mediated upregulation of Y chromosome KDM5D contributes to the sex differences in KRAS-mutant CRC, providing an actionable therapeutic strategy for metastasis risk reduction for men afflicted with KRAS-mutant CRC.

Nature, Published online: 05 April 2023; doi:10.1038/d41586-023-00998-y

First peer-reviewed analysis of the Chinese swabs confirms animal DNA was present in samples that tested positive for SARS-CoV-2.