Artificial intelligence versus human researcher performance for systematic literature searches: a study focusing on the surgical management of base of thumb arthritis

INTRODUCTION

In an era of digital transformation, traditional literature search methods are being supplemented and replaced by artificial intelligence (AI)-based platforms^[1,2]. These include software such as Elicit, Consensus, and ChatGPT, which have been proposed as valuable tools for expediting information retrieval and facilitating the dissemination of medical information. In this domain, ChatGPT has received considerable commentary on its potential in academia across a range of topics, from osteoarthritis to cosmetic surgery, with major concerns about its ability to correctly identify the source of its knowledge, albeit surprisingly accurately. Although an interesting avenue to explore, the comparative efficiency and accuracy of different chatbots in locating and sourcing information compared with traditional human-initiated searches have not been explored^[3-5].

Trapeziometacarpal osteoarthritis (TMCJ OA) is a common condition among the elderly that significantly limits thumb movement and functionality necessary for everyday tasks^[6]. Management of TMCJ OA begins medically in mild cases, progressing to operative intervention only when anti-inflammatory and pain relief prove insufficient. Multiple surgical and non-surgical treatment modalities are available, but their comparative effectiveness is unclear, especially surgical ones^[7]. This gap in the literature leaves healthcare professionals and patients in a predicament during the decision-making process, and a systematic review and meta-analysis is likely an effective means to summarize information and facilitate a consensus in the plastics and orthopedics community.

With this in mind, we carried out a comparative study that scrutinized the performance of Elicit, Consensus, and ChatGPT with manual human literature search methods for the management of TMCJ OA. The primary outcomes were the ability to identify publications with higher-level evidence, as well as the number of publications and their relevance. The outcomes of interest specific to TMCJ OA were also investigated to inform the potential role and value of AI for conducting systematic reviews.

METHODS

The current study adhered to the Cochrane Handbook of Systematic Reviews of Interventions and the preferred reporting items for systematic reviews and meta-analyses (PRISMA) statement guidelines throughout all stages^[8]. The study was registered on PROSPERO, the International Prospective Systematic Review (CRD420431089). The primary objective of this study was to evaluate the performance of AI-based platforms (Elicit, Consensus and ChatGPT) against human experts for conducting a literature search for a systematic review on the base of thumb arthritis treatments. Institutional ethical approval was not required since this study did not involve human subjects. These three AI platforms were selected for their prominence and widespread adoption in the research community at the time of the study. Elicit, developed by Ought, was chosen for its specialized focus on scientific literature search and summarization. Consensus, created by Consensus.app, was selected for its ability to aggregate and analyze scientific papers. ChatGPT, an advanced language model by OpenAI, was included due to its versatility in understanding and generating human-like text across various domains, including scientific literature. The standard ChatGPT version was used to minimize any potential bias and maintain methodological consistency with other studies.

Risk of bias assessment

The methodological quality of each study was assessed using Cochrane risk-of-bias (ROB) tool for randomized trials [Figure 1]. The RoB tool addresses the following biases: random sequence generation, bias due to deviations from intended interventions, bias due to incomplete outcome data, bias in the measurement of the outcome, and selective reporting. The items were assessed as “low risk”, “high risk”, or “some concerns”. We used the original RoB tool rather than the updated RoB 2 tool, as our research team had extensive experience with the original tool, ensuring consistent and accurate assessments, and wanted to maintain comparability with other systematic reviews in our area of research that predominantly used the original tool. We acknowledge that the RoB 2 tool offers a more nuanced approach, particularly for assessing bias in subjective outcomes and open-label studies. However, our use of the original RoB tool may have resulted in slightly more conservative bias assessments. This conservative approach strengthens the reliability of our findings, as it is less likely to underestimate potential biases in the included studies.

Figure 1. Risk of bias of all included studies.

RESULTS

The manual search executed by human authors yielded 6,018 initial results, followed by 4,980 results from Elicit, 3,436 from Consensus, and lastly, only 6 from ChatGPT, Table 1. Elicit found 205 RCTs, while the manual search found 63, Consensus returned 42, and ChatGPT identified one [Figures 2-4]. For prospective studies, the manual search yielded 1,852 results, followed by 1,123 from Elicit, 963 from Consensus, and one from ChatGPT. Elicit’s broader selection of RCTs stems from its indiscriminate inclusion of all studies discussing base of thumb arthritis regardless of comparison with surgical management strategies, and its search focused largely on non-surgical management. Lastly, Elicit had the highest false positives at 94%, followed by consensus at 76%, human researchers at 43%, and ChatGPT at 0%, Table 1.

Figure 2. PRISMA figure of consensus platform search. PRISMA: Preferred reporting items for systematic reviews and meta-analyses.

Figure 3. PRISMA figure of Elicit platform search. PRISMA: Preferred reporting items for systematic reviews and meta-analyses.

Figure 4. PRISMA figure of manual search. PRISMA: Preferred reporting items for systematic reviews and meta-analyses.

DISCUSSION

This case study is the first to explore the comparative performance between human-initiated and AI-initiated literature searches. These findings demonstrate AI platforms currently have poor proficiency for use in academia, especially ChatGPT, which performed poorly across all domains and outcomes. Although Elicit came the closest to mimicking human precision of the initial search, manual searches were far superior to all AI literature search engines in terms of the number of studies identified and their specificity to the subject of TMCJ OA. AI engines also overlooked studies extracted from the manual search and lacked precision in the subject of the search, evidenced by high false positive identification rates.

Interestingly, the average age of participants across the 23 included studies was 49.83 years, notably younger than the typical patient population seen in most CMC1 (first carpometacarpal joint) osteoarthritis publications. This relatively young cohort raises essential questions about the generalisability of the study results to the broader TMCJ OA population, which typically presents in older adults. Including younger patients may reflect a trend toward earlier surgical intervention, possibly due to increased awareness or changes in treatment paradigms. Further investigation is warranted to understand the implications of this age discrepancy for treatment outcomes and long-term prognosis for TMCJ OA patients.

Upon inspecting the number of relevant studies produced, Elicit was the most comprehensive AI search engine, albeit only surpassing Consensus by a single article. However, most RCTs identified by Elicit were tangential and addressed various topics beyond management strategies. As this methodology has never been applied since the inception of large language models (LLM), these findings cannot be discussed and contextualized in other studies. Despite the promise of AI to replicate laborious manual tasks, the results herein are disappointing and suggest that LLMs currently have no applicability in relieving the burdensome process of literature searching and screening. This study shows that LLMs could do a disservice to the scientific community by excluding publications typically deemed important and including irrelevant ones in initial searches. This misalignment with the topic of discussion led to a 94% false positive rate within the search, compared to a human false positive rate of 43%. While Consensus elicited nearly 1,500 fewer studies than Elicit, it included nine of the ten studies identified by Elicit, yielding a significantly lower false positive rate of 73%. Although Elicit identified the most publications overall, its search was the least precise and most inefficient of all AI search engines. No AI-driven engine could identify studies not included in the human search, indicating that human searches were the most precise and had a very low false negative rate.

Concerning primary outcomes, Elicit emerged as the sole AI search engine capable of identifying RCTs addressing all relevant primary outcomes. However, Consensus failed to uncover any studies focused on quality of life, although AI search engines could identify more than one study, each discussing the range of motion. ChatGPT exhibited the least effective performance, locating only one study addressing two of the six primary outcomes, and finding a volume of studies that was small in comparison to manual searches by authors and AI search engines^[32,33]. Overall, AI search engines were inferior to manual searching, highlighting a shortcoming in their algorithms for sourcing comprehensive, high-quality literature relevant to the research topic^[2]. The indiscriminate data retrieval by AI search engines in this study points to a potential for them to produce erroneous information outputs, due to a lack of precision and hierarchical structure during information gathering and organization^[2]. Peering into the mind of an algorithm, it is clear from these results that these deficits could account for the erroneous or outdated responses sometimes reported in previous studies. Therefore, for AI to be a viable tool in academic literature searches, substantial improvements are needed in categorization, publication filtering, bias detection, database integration, and ethical data handling.

This study explores the use of AI for literature searches, highlighting the significant improvements required for AI tools to be feasibly incorporated into literature searches for the creation of academic content. These improvements may be grouped into a few main broad categories that should be considered. Paramount among these is ensuring “reproducibility”, which is the cornerstone of academic research and literature searches, as exemplified by the dual-reviewer approach outlined in the PRISMA guidelines. Current AI tools fall short in accuracy and comprehensiveness. Additionally, users may ask AI search engines the same question multiple times and receive different answers informed by different sources^[34,35,36]. A future AI system must be able to recognize and understand context, academic language, and abbreviations to meet the reproducibility standard. Moreover, AI must develop a nuanced understanding of academic context, hierarchy, and the goals of a literature search to match or surpass human researchers in precision and thoroughness^[37].

Secondly, AI should transcend simple keyword identification to gain a deeper semantic understanding of academic papers. This includes comprehending study objectives, methodologies employed, and resultant conclusions. Such advancements would enhance AI’s ability to categorize, filter, and rank search results based on criteria such as relevance, currency, citation frequency, and the publishing journal’s impact factor. Thirdly, improvements are needed to discern and neutralize potential biases, including those related to geographic location, authorship, or publication prestige, to ensure fair data representation. Given the dynamic nature of academia, with its continuous generation of novel knowledge and methodologies, AI platforms must be equipped for accessible, ongoing learning and enhancement.

In conclusion, This study found AI tools such as Elicit, Consensus, and ChatGPT were inaccurate and lacked comprehension compared with human-initiated literature searches. These tools need to evolve beyond simple keyword identification toward a nuanced understanding of academic hierarchy and context. Therefore, AI’s integration into academic literature searches demands substantial enhancements in its understanding of academic context and hierarchy, fulfilling the crucial reproducibility criterion and aligning it with the rigorous standards of human-conducted research.