Large Language Models in Rheumatology-Centered Research: a Scoping Review of Model Openness and Reporting Practices

Abstract

• Most rheumatology studies using LLMs relied exclusively on closed/proprietary models • Reproducibility-relevant details were frequently underreported • We propose a review-derived checklist to improve reporting transparency • Our findings may inform future consensus-based reporting efforts To systematically identify and characterize original rheumatology research using large language models (LLMs), quantify reliance on closed/proprietary versus open systems, and assess reporting practices relevant to reproducibility (model versions, prompts, inference settings, and availability of code/data), in order to identify recurring reporting gaps and inform pragmatic considerations for improving transparency. We searched PubMed/MEDLINE for English-language peer-reviewed original research published from 1 November 2022 to 23 January 2026. Two reviewers independently screened title/abstract and full texts, with a third resolving disagreements. Data were extracted using an LLM-assisted workflow and then independently verified against source articles by the authors. Extracted items included study characteristics, model families and openness, access mode, versioning/timing, and transparency indicators (prompts, code, data). Of 185 screened records, 63 studies were included. Most were research (n = 26/63; 41.27%) or education-focused (n = 20/63; 31.75%). Studies predominantly used closed/proprietary LLMs (n = 50/63; 79.37%), with limited exclusive use of open-weight models (n = 4/63; 6.35%) and some hybrid use (n = 9/63; 14.29%). OpenAI models were most common (n = 55/63; 87.30%). Reporting was heterogeneous: interaction language was often not reported (n = 42/63; 66.67%); access mode was reported in (n = 19/63; 30.16%); output generation date in (n = 33/63; 52.38%). Prompts were shared in (n = 48/63; 76.19%), but code was publicly available in (n = 4/63; 6.35%), and data in (n = 13/63; 20.63%). LLM-based rheumatology research largely depends on closed models with inconsistent reporting of reproducibility-critical details and minimal code sharing. Clearer reporting practices and, in the longer term, consensus-based standards will be needed to strengthen the methodological robustness of this field.

Ähnliche Arbeiten