From unstructured to structured: opportunities, risks and engineering practices for LLM-based data extraction in chemistry and materials

INTRODUCTION

The paradigm of chemical materials discovery is shifting toward datadriven approaches, where highthroughput screening, machine learning and artificial intelligence rely on large volumes of highquality structured data^[1-3]. Yet the vast majority of accumulated experimental knowledge, such as synthesis conditions, characterization results, property measurements, resides in unstructured natural language texts, including millions of research articles, patents and supporting information files^{[4, 5]}. This “textual data warehouse” stands in stark contrast to the needs of datadriven chemical materials discovery workflows. Manual curation is prohibitively slow, and traditional natural language processing tools, often rulebased or trained on narrow corpora, suffer from poor generalization and require weeks of reengineering for each new task. In contrast to traditional methods that rely on predefined rules or task-specific training data, LLM-based approaches offer greater flexibility and generalization through in-context learning. However, they also introduce new challenges related to reliability, interpretability, and error control [Figure 1].

Figure 1. Comparison of data extraction paradigms.

Large language models (LLMs) have emerged as a gamechanger. Their ability to perform incontext learning from a few examples^[6,7] and to follow complex instructions enables researchers to prototype data extraction pipelines in days rather than months^[8,9]. By treating LLMs as versatile “data engines,” we can now convert unstructured text into structured formats, such as tables, databases, knowledge graphs and so on, at an unprecedented scale^[10-13]. This transformation promises to accelerate the construction of comprehensive experimental databases that fuel downstream applications, from catalyst screening to property prediction.

Several recent reviews have surveyed the emerging role of LLMs and AI-driven frameworks in chemical and materials science. For instance, Zhang et al.^[14] summarize the use of LLMs and related AI tools for knowledge extraction and data-driven research, while Zhang et al.^[15] present the concept of a “digital materials ecosystem,” highlighting integrated workflows that combine databases, AI agents, and automated experimentation. These works collectively emphasize the growing importance of structured data extraction and AI-enabled discovery pipelines.

Despite these advances, these studies primarily focus on model capabilities, system architectures, or high-level workflows, often under idealized settings, with comparatively less attention paid to the practical challenges encountered in real-world deployment. Consequently, the systematic risks inherent to LLM-based data extraction remain insufficiently examined. These risks manifest in multiple forms. LLMs may hallucinate facts, make misassociations between conditions and values, propagate errors in multi-step workflows, or quietly turn uncertainties into certainties. Recognizing these risks and developing engineering practices to mitigate them is essential if we are to build reliable, reusable experimental databases. This perspective addresses this gap by examining LLM-based data extraction from an engineering practice perspective, with a focus on identifying systematic risks and developing practical mitigation strategies.

In this perspective, we first dissect the typical LLMbased extraction workflow, including preprocessing, model interaction and postprocessing. We then expose the hidden traps encountered during largescale extraction, informed by practical experience. Finally, we propose a set of guiding principles and future directions to steer the field toward trustworthy, scalable and impactful data extraction.

LLM AS A DATA ENGINE: FROM UNSTRUCTURED TEXT TO STRUCTURED DATA

A successful extraction pipeline consists of three interconnected stages, namely preprocessing, LLM interaction and postprocessing [Figure 2]. Each stage requires careful choices that balance accuracy, cost and robustness.

Figure 2. Three-stage workflow for LLM-driven data extraction. LLM: Large language model.

LLM interaction: prompting, fine-tuning and beyond

The core decision is how to query the LLM. Prompt engineering is the first resort. A system prompt defines the model’s persona (“You are a chemistry expert”), while the user prompt contains the task and the text^[20]. Zeroshot prompts are simple but may lack domain knowledge. Fewshot prompts provide a handful of inputoutput examples, enabling incontext learning^[21]. The selection of examples, including both positive and negative cases, strongly influences performance. For complex tasks, chainofthought prompting decomposes the extraction into reasoning steps, improving accuracy^[22].

When prompting falls short, finetuning on a small and curated dataset can adapt a general LLM to specialized chemical entities and relations^[23]. Parameterefficient methods like LoRA (low-rank adaptation)^[24] make fine-tuning feasible with modest computational resources. Recent work shows that a 7B-parameter model finetuned on chemistryspecific tasks can outperform much larger general models^[25].

Beyond text, chemical data often reside in figures, spectra, and table images rather than plain text. Recent work has shown that combining image-text alignment with LLM-based reasoning enables the extraction of structured information directly from such multimodal sources^[26] [Figure 3]. From a workflow perspective, this extends the three-stage paradigm discussed above: in preprocessing, figures and tables are parsed using vision models or OCR tools; in model interaction, cross-modal alignment associates visual elements with textual context; and in postprocessing, consistency checks integrate information across modalities. In practice, hybrid workflows, using vision-language models (VLMs) for initial interpretation and specialized tools (e.g., OSRA for structure diagrams) for verification, offer a pragmatic compromise.

Figure 3. Schematic diagram and evaluation methods of the DIVE workflow^[26].

SYSTEMIC RISKS AND HIDDEN TRAPS IN LARGE-SCALE EXTRACTION

Despite the promise, deploying LLMs for bulk extraction reveals several insidious pitfalls that can undermine database quality [Table 1].

Risk 1: Condition-value mismatch - the most insidious error. LLMs often correctly extract a numeric value (e.g., “product selectivity”) but fail to capture or misassign its experimental conditions (temperature, doping level, measurement method). This occurs because models excel at recognizing isolated facts but struggle with implicit contextual relationships. Downstream models trained on such mismatched data will learn spurious correlations.

Risk 2: Error accumulation in multi-step workflows. When processing thousands of documents at one time, even with a single-step accuracy rate of 99%, the final accuracy rate may drop sharply after the accumulation of multiple tasks (such as identifying compounds → extracting conditions → associating performance) [Figure 4A]. The autoregressive nature of LLMs means a mistake early on propagates through all subsequent generations.

Figure 4. (A) Schematic illustration of error accumulation in large-scale extraction; (B) Schematic illustration of LLM “sycophancy” bias. LLM: Large language model.

Risk 3: “Ingratiation” bias - turning uncertainty into certainty. When the source text says “approximately 400 °C” or “a possible intermediate,” LLMs tend to output “400 °C” or “intermediate: present,” erasing ambiguity. Without explicit instructions to preserve uncertainty, the systematically over-confident predictions from LLMs create a false sense of precision in the database [Figure 4B].

Risk 4: Ambiguous instructions lead to ineffective reasoning. Overly long or vague prompts waste the LLM’s context window on irrelevant details. In extreme cases, the model may overthink and hallucinate data to satisfy an unclear request. Concise, explicit instructions are essential.

Risk 5: Literature bias and data contamination. Scientific literature overrepresents positive results. Databases built solely from publications will lack failed experiment results, biasing any subsequent analysis. Moreover, LLMs may have been trained on a portion of the test corpus, leading to inflated benchmark scores that do not reflect true generalization.

THE ROAD AHEAD: ENGINEERING PRACTICES AND RESEARCH FRONTIERS

To harness LLMs responsibly, we propose four guiding principles.

Principle 1: Decompose tasks with divide and conquer. Complex extraction goals should be broken into a series of focused subtasks, each handled by a dedicated prompt or model call. This modularity reduces error rates and simplifies debugging.

Principle 2: Multi-layer validation. Build validation at four levels: (1) syntactic level- constrained decoding ensures format compliance; (2) semantic level - chemical rules (atom conservation, charge balance) flag implausible extractions; (3) consistency level- crossrecord checks (e.g., large property variations for the same material) trigger human review; (4) Uncertainty quantification: where feasible, the model should output confidence scores for low-certainty fields.

Principle 3: Clear division of labor - LLMs for what they do best. It is important to delineate what LLMs are best suited for, such as entity recognition, relation extraction and information summarization, and what should be delegated to deterministic postprocessing tools, such as unit conversion, name standardization and numerical computation. Anything that can be handled deterministically with conventional code should not be delegated to probabilistic LLMs. A mechanism can be established that combines automatic extraction with periodic verification by experts. Recent work demonstrates that such role separation can be effectively implemented through modular pipelines combined with deterministic tool-based postprocessing, with embedding-based matching scores providing a practical way to quantitatively assess extraction quality^[26]. LLMs serve as powerful “postdoctoral researchers,” whereas human scientists remain the ultimate “principal investigators.” This hybrid approach maximizes reliability and reduces cost.

Principle 4: Appeal for standardized reporting at the source. Encourage journals and authors to present key data (synthesis parameters, performance metrics) in machine-readable tables within the main text, rather than burying them in unstructured paragraphs. FAIR data principles should become standard practice. Standardized reporting drastically lowers the barrier for subsequent extraction.

CONCLUSION AND OUTLOOK

Large language models have unlocked the ability to convert the vast unstructured literature of chemistry and materials into structured, actionable data. By following a systematic workflow, including preprocessing, careful LLM interaction and rigorous postprocessing, we can accelerate the construction of highquality experimental databases. Yet this promise comes with profound risks, namely mismatched conditions, error cascades, overconfidence and hidden biases. Realizing the full potential of LLMs requires not only technical innovation but also a disciplined engineering mindset encompassing task decomposition, multilayer validation, strategic leveraging of each tool's strengths, and advocacy for improved data-reporting practices in scientific publishing. When these principles are embraced, LLMs will become trusted partners in scientific discovery, transforming text into insight and powering the next wave of datadriven materials innovation.

Looking forward, several research frontiers will shape the next generation of extraction systems:

Multimodal agents that can seamlessly navigate text, figures, and tables, invoking specialized tools (e.g., crystallographic databases) when needed.

Crossdocument integration to link methods described in one paper with results reported in another, enabling the reconstruction of complete experimental workflows.

Lowresource benchmarks that assess model performance on rare entities, ambiguous descriptions, and outofdistribution samples, providing a realistic measure of generalization.