Large Language Models and the Threat of Hallucinated Packages

The rapid evolution of Large Language Models (LLMs) is reshaping how developers approach coding. However, this power comes with an alarming drawback: a tendency for these models to hallucinate non-existent software packages. This issue could potentially fuel a wave of supply chain attacks, as recent research reveals a staggering rate of hallucinations in generated code.

The potential dangers of LLMs in software development.

The Extent of the Hallucination Problem

In one of the most comprehensive studies on this issue, researchers conducted tests across 30 different scenarios, revealing that 19.7% of the 2.23 million code samples generated included references to non-existent packages. This amounts to 440,445 instances of hallucinated packages across tested models, specifically focusing on Python and JavaScript.

What’s even more concerning is the discovery of 205,474 unique examples of hallucinated package names. As developers increasingly rely on LLMs to automate their coding processes, the probability of falling prey to malware-laden hallucinations rises exponentially.

The underlying issue lies in the multifaceted libraries in languages like Python and JavaScript that allow developers to assemble robust applications quickly. A previous study revealed the existence of 245,000 malicious packages within open-source repositories, indicating the already critical state of software supply chains.

Understanding Hallucination and its Consequences

Hallucination is a phenomenon where LLMs generate plausible but entirely fabricated responses. This extends to code generation, where an LLM might suggest a package that doesn’t even exist. Developers who trust these outputs without verification risk including these imaginary packages in their projects, potentially introducing vulnerabilities.

“Unsuspecting users, who trust the LLM output, may not scrutinize the validity of these hallucinated packages in the generated code,” researchers assert. “This resulting insecure open-source code also has the potential of being included in the dependency chain of other packages and code, leading to a cascading effect where vulnerabilities are propagated across numerous codebases.”

An example of code vulnerable to hallucination issues.

Variability Among Models

Not all LLMs are created equal when it comes to hallucinations. A stark contrast was noted between the GPT-series models and open-source alternatives during the study, with the former demonstrating a hallucination rate of 5.2%, compared to 21.7% for their open-source counterparts. Interestingly, Python code appeared less prone to this issue compared to JavaScript.

Historically, package confusion attacks have been prevalent, often rooted in tactics such as typosquatting or brandjacking. With the introduction of LLMs, the potential for hallucination-based attacks poses yet another layer of threat to the software supply chain.

One notable incident involved a fictional Python package named “huggingface-cli,” which garnered over 30,000 downloads despite its non-existence, highlighting the risks associated with code generated by LLMs. Major companies unknowingly recommended this phantom package, indicating just how pervasive the problem has become.

Are We Prepared for a Hallucination Attack?

So far, the industry has not witnessed any direct attacks stemming from hallucinated packages, but experts agree that it’s only a matter of time. As awareness of this emerging threat grows, developers are urged to take caution and remain vigilant in their coding practices.

Mitigation Strategies

Researchers suggest several approaches to mitigate the hallucination issue. While one proposed solution of cross-referencing generated packages against a master list might help identify fake packages, it doesn’t address the core problem. Instead, focusing on the root causes of LLM hallucinations—such as refining prompt engineering and utilizing Retrieval Augmented Generation techniques—may yield more effective results.

Furthermore, fine-tuning LLMs to generate more accurate outputs for high-risk tasks will require active collaboration from the organizations developing these models. However, there remains a notable silence from these entities regarding the urgency of these findings.

“We have disclosed our research to model providers including OpenAI, Meta, DeepSeek, and Mistral AI, but as of this writing, we have received no response or feedback,” researchers disclosed in their latest update.

Ensuring security in code development is crucial.

Conclusion

As we continue to harness the power of LLMs in software development, awareness of the hallucination phenomenon is critical. The potential for supply chain vulnerabilities linked to non-existent packages poses a formidable challenge for developers. By prioritizing vigilance and adapting our coding practices, we can mitigate the risks of these emerging threats.

Staying informed and engaged with ongoing research in this area will be essential as we navigate the complexities of AI in programming. The future of secure software development relies on our collective adaptability and foresight in tackling this unique challenge.