201 Using a large language model to identify behavioral and social science research at the University of Michigan

Abstract

Objectives/Goals: To tailor support for behavioral and social science research (BSSR), it is helpful to understand this broad category. As a first step to characterizing BSSR at University of Michigan (UM), our goal is to use a large language model (LLM) to identify health-related BSSR from a database of funded studies. Methods/Study Population: We are using a private, secure version of Open AI ChatGPT-4.1 LLM to evaluate whether studies are health-related BSSR or not based on the study team’s (1) abstract, (2) objectives, and (3) key words entered into UM’s internal research proposal system for all studies funded over the last 10 years (i.e., 2014 – 2024). The model included a prompt with the National Institutes of Health (NIH) definition of BSSR and was instructed to categorize the research as being BSSR, not BSSR, or possibly BSSR. Results from the LLM were compared for consistency with assessments made by human-made decision rules. Results/Anticipated Results: Preliminary results from a pilot sample of 150 studies funded in 2024 (~5%) reviewed by ChatGPT for BSSR status revealed: 6 false positives, 128 true negatives, 14 true positives, and 2 false negatives. Accuracy = 95%, precision = 70%, recall (sensitivity) = 88%, specificity = 96%, and F1 Score = 0.78. ChatGPT and human-based decisions differed primarily on cognition and vehicle technology studies and studies for which information was sparse. Further refinements to the prompt are being made and applied to the remaining studies funded in 2024 before expanding to the entire sample of studies from the last 10 years. Discussion/Significance of Impact: It is time and cost prohibitive for a human to review and classify all funded studies at UM as BSSR or not. Using LLMs offers the potential to make this classification work feasible with limited resources, which is a necessary step toward understanding the state of BSSR.

Ähnliche Arbeiten