Common LLM Pitfalls in Data Science

LLM coding agents are confidently wrong about a small set of recurring topics in data science. Many of these failures are now measured and documented. This page lists the ones worth knowing, with citations where they exist, and one concrete fix for each.

The fix is usually a rule in CLAUDE.md plus a habit of asking the agent to show real output rather than describe what it would do.

Security and supply chain

Slopsquatting: hallucinated package supply-chain attacks

The pitfall. The agent recommends a package that does not exist. An attacker registers that name with malicious code. The next person to copy-paste the suggestion installs malware.

Why it is wrong. The canonical study, Spracklen et al. (USENIX Security 2025), tested 16 LLMs on 576,000 code samples and found 19.7% of recommended packages were hallucinations, with 43% of fabricated names recurring across all 10 reruns. The attack is therefore both common and predictable. A real incident: security researcher Bar Lanyado registered an empty huggingface-cli package after ChatGPT recommended it. Within three months it had been downloaded over 30,000 times, including by Alibaba’s own GraphTranslator repo.

The fix. Verify every package name on the registry before installing.

# Python
uv pip index versions <package>
# R / Bioconductor
Rscript -e 'BiocManager::available("<package>")'

If the agent cannot show real output proving the package exists, the package does not exist. Pin everything in a lockfile (uv.lock, renv.lock) and audit with Socket, Snyk, or Aikido.

Insecure code generation

The pitfall. The agent writes code that compiles, passes basic review, and contains a security weakness.

Why it is wrong. Pearce et al. (IEEE S&P 2022, “Asleep at the Keyboard?”) had Copilot complete 89 security-sensitive scenarios drawn from MITRE Top-25 CWEs and found 40% of the 1,689 generated programs were vulnerable. A 2025 follow-up by Fu et al. examined Copilot snippets in real GitHub projects and found 29.5% of Python and 24.2% of JavaScript snippets contained CWEs spanning 43 weakness categories. The most common were CWE-330 (insufficient randomness), CWE-94 (code injection), and CWE-79 (XSS). Models trained on both vulnerable and patched versions of well-known files reproduce the vulnerable version about 33% of the time and the fix only 25%.

The fix. Run static analysis on every generated diff. Semgrep, Bandit, and CodeQL catch most of these. Pearce showed that feeding the warning back to the agent repaired up to 55.5% of issues it had introduced.

Indirect prompt injection

The pitfall. The agent reads a file or URL and treats text inside that document as instructions. A malicious comment in a dependency, a GitHub issue body, or an MCP tool response can hijack the agent.

Why it is wrong. Greshake et al. (arXiv 2302.12173) introduced indirect prompt injection in 2023. Real coding-agent incidents have followed. In May 2025, Invariant Labs showed that a malicious public-repo issue could prompt-inject the official GitHub MCP server so the agent leaked private-repo contents into a public PR (Docker writeup). CVE-2025-6515 covers session-level prompt hijacking across MCP clients. Trend Micro found a SQL injection in Anthropic’s reference SQLite MCP server, already forked over 5,000 times.

The fix. Treat every retrieved string as untrusted data, not instructions. Use an allowlist for the tools the agent can call without confirmation. Never give an agent both private-repo read and public-repo write in the same session. Review MCP server source before installing.

Statistical and ML mistakes

SMOTE for class imbalance

The pitfall. “The classes are imbalanced (90% negative, 10% positive). I’ll apply SMOTE to oversample the minority class before training.”

Why it is wrong. SMOTE creates synthetic minority samples by interpolating between existing ones. For high-dimensional biomedical data the synthetic samples are often biologically implausible. Multiple meta-analyses in the past five years have shown SMOTE rarely improves real-world classifier calibration and sometimes hurts it.

The fix. Add to CLAUDE.md: “Never apply SMOTE without explicit user approval. Raise the tradeoff first.” When you do hit class imbalance, prefer:

• Stratified k-fold cross-validation.
• Class-weighted loss (class_weight='balanced' in scikit-learn, scale_pos_weight in XGBoost).
• Threshold tuning post-training.
• A different metric (PR-AUC, MCC, Brier score) instead of accuracy.

Data leakage in preprocessing

The pitfall. The agent fits a scaler, target encoder, or feature selector on the full dataset, then splits into train and test. The reported test performance is no longer an honest estimate.

Why it is wrong. Information from the test set leaks into training. The leakage patterns themselves are well documented across IBM and most ML reference material. Agents reproduce the pattern because tutorials online do.

The fix. Wrap every transformer in sklearn.pipeline.Pipeline and pass the pipeline to cross_val_score. The test set is touched exactly once at the end.

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("clf", LogisticRegression()),
])
pipeline.fit(x_train, y_train)
test_pred = pipeline.predict(x_test)

For time series, use TimeSeriesSplit and never let any preprocessing step see future timestamps.

Single-seed reporting

The pitfall. The agent runs one cross-validation split, reports a single AUC, and treats it as the answer.

Why it is wrong. A single split has high variance, especially on small biomedical datasets. A “good” model under one seed can be a “bad” model under another.

The fix. Demand at minimum 5 repeated cross-validation splits with different seeds. Report mean and standard deviation.

scores = cross_val_score(
    pipeline, X, y,
    cv=RepeatedStratifiedKFold(n_splits=5, n_repeats=10),
)
print(f"AUC: {scores.mean():.3f} ± {scores.std():.3f}")

Multiple-testing correction left unspecified

The pitfall. The agent reports raw p-values for 20,000 genes and flags everything below 0.05.

Why it is wrong. Without correction, hundreds of genes will be flagged purely by chance.

The fix. State the correction explicitly:

• Differential expression: Benjamini-Hochberg FDR (method = "BH" in R, multipletests(method='fdr_bh') in Python). Threshold padj < 0.05.
• GWAS: Bonferroni or genome-wide significance threshold 5e-8.
• Multiple comparisons across conditions: Tukey HSD or Holm.

If the agent gives you raw p-values, ask explicitly which correction was applied.

Statistical reasoning errors

The pitfall. The agent restates “p < 0.05 means a 5% chance the null is true” or conflates Type I error with significance level, then writes code based on the wrong premise.

Why it is wrong. A 2026 arXiv study, Can LLM Reasoning Be Trusted? A Comparative Study Using Human Benchmarking on Statistical Tasks, documents that LLMs systematically miscompute basic test statistics and fail to flag conceptual errors. Statistics textbooks and Q&A sites repeat the misinterpretations in their training data.

The fix. Force the agent to define each term it uses before computing. Verify any p-value or CI claim by re-running the test in a separate script. Have the agent state assumptions (normality, independence, multiple-testing correction) explicitly.

Train-test-validation discipline

The pitfall. The agent splits 80/20 train-test and uses the test set for hyperparameter selection.

Why it is wrong. Hyperparameter tuning on the test set is a form of multiple comparisons. The reported score is biased optimistic.

The fix. Three-way split or nested cross-validation. The test set is touched once, after all model decisions are final.

Correlation versus causation in differential expression

The pitfall. “Gene X is significantly upregulated in disease, so it drives the disease phenotype.”

Why it is wrong. Differential expression tests are observational. Many DE genes are downstream effects of an upstream cause.

The fix. Phrase results as “Gene X is differentially expressed between groups”, not “Gene X causes the phenotype”. Reserve causal claims for interventional experiments (knockouts, knock-ins, CRISPR screens).

Bioinformatics factuality

Hallucinated gene names, aliases, and coordinates

The pitfall. The agent confidently returns a gene symbol that does not exist, a wrong alias, or coordinates from the wrong assembly.

Why it is wrong. The GeneTuring benchmark (Hou & Ji 2023) tests basic genomics tasks: gene name lookup, alias resolution, SNP-to-gene mapping. GPT-3 and GPT-4 baselines scored poorly and often fabricated symbols. The GeneGPT paper (Jin et al., Bioinformatics 2024) showed grounding the model in NCBI E-utilities calls dramatically reduced hallucinated genes. A 2025 expansion across 10 frontier LLMs concluded all of them remain unreliable as standalone genomic knowledge bases.

The fix. Force the agent to call NCBI or Ensembl APIs instead of recalling facts. Verify every gene symbol against HGNC. Never trust a coordinate without an explicit assembly tag.

Wrong genome assemblies

The pitfall. The agent uses GRCh37 in one script and GRCh38 in another. Coordinate mismatches surface days later when nothing aligns.

Why it is wrong. Agents pick whichever assembly was most common in their training data. They do not always check existing files.

The fix. State reference and annotation versions in the project CLAUDE.md. Add a check at the top of every analysis script:

ASSEMBLY = "GRCh38"
ANNOTATION = "GENCODE v45"
print(f"Using {ASSEMBLY} with {ANNOTATION}")

Outdated workflows

The pitfall. “We can use TopHat and Cufflinks for alignment and quantification.” TopHat has been deprecated since 2016.

Why it is wrong. Agents see a huge volume of pre-2016 tutorials in their training data. The recency of the deprecation does not register.

The fix. State the current standard in CLAUDE.md:

- RNA-seq alignment: STAR or HISAT2.
- RNA-seq quantification: salmon or kallisto.
- Never suggest TopHat, Cufflinks, or Tuxedo suite tools.

Confident wrongness about file formats

The pitfall. “BAM files are tab-separated.” (They are binary.) “VCF format does not support phased genotypes.” (It does.)

Why it is wrong. Bioinformatics file formats are heavily documented but easy to confuse. Agents extrapolate from related formats.

The fix. When in doubt, read the actual specification (SAM/BAM/CRAM, VCF, GFF3) or run the file through a real parser (samtools view -h, bcftools view) and compare to what the agent claims.

Clinical and biomedical text

Hallucination and omission in clinical summarisation

The pitfall. The agent generates a clinical note summary that contains plausible but unsupported findings, or omits a finding that was in the source.

Why it is wrong. A 2025 npj Digital Medicine study on medical summarisation measured a 1.47% hallucination rate and 3.45% omission rate for clinical note generation. Reviews of LLMs in radiology report hallucination rates of 8 to 15% for image-grounded reports, including phantom lesions (Bhayana, Radiology 2024).

The fix. Require source citations to retrieved records. Pair generation with a structured-output schema such as FHIR fields so omissions are detectable. Never deploy without a clinician in the loop.

Code quality and trust

API hallucination: invented function arguments

The pitfall. “Use DESeq2::contrast(dds, name = 'condition_treatment_vs_control').” Maybe the function exists. Maybe the right argument is contrast =, not name =. The hallucinated kwarg passes type checks and fails at runtime.

Why it is wrong. Amazon Science (CloudAPIBench, 2024) shows LLMs frequently hallucinate endpoint URLs and parameter names, especially for low-frequency APIs. Library APIs evolve faster than training cutoffs; rare APIs are under-represented; the model interpolates plausible names from related libraries.

The fix. Always check the help page or package source before running:

?DESeq2::results
?DESeq2::lfcShrink

help(scanpy.pp.normalize_total)

If the agent’s call does not match the signature, do not run it. For frequent API drift, paste the relevant doc page into the prompt.

Overconfidence and miscalibration

The pitfall. The agent says “this should work”, “this is now fixed”, or “tests pass” without actually executing the code.

Why it is wrong. Spiess et al. (ICSE 2025) measured calibration of code models and found verbal confidence and actual correctness diverge sharply. A separate Dunning-Kruger study found LLMs were overconfident in 84.3% of scenarios tested, and GPT models were overconfident in every scenario. RLHF training rewards confident-sounding outputs regardless of quality.

The fix. Treat every “this should work” claim as untested. Require the agent to run the code and paste the output, or to write a failing test that then passes.

Memorisation of buggy code

The pitfall. The agent writes code that copies a known-buggy implementation from Stack Overflow or a popular tutorial, including the bug.

Why it is wrong. Memorisation studies show LLMs reproduce verbatim spans of training data, and this scales superlinearly with how many copies of the snippet appeared in training. The Stanford SAIL group has demonstrated long-form verbatim regurgitation. The practical risk is that the agent’s confident-looking code is a copy of a buggy gist whose issue tracker would have warned you.

The fix. When code looks suspiciously like a known repo, search GitHub for an exact substring of it and check the issue tracker before relying on it.

Fake citations

The pitfall. “This approach is described in Smith et al. 2019, Nature Methods, doi:10.1038/s41592-019-0567-8.”

Why it is wrong. Sometimes the paper exists. Sometimes it does not. The DOI may be made up. The citation format is structured enough that fabrication is easy and looks legitimate.

The fix. Never paste an agent-provided citation into a manuscript without confirming the DOI on doi.org or PubMed. Treat every agent-provided citation as a hypothesis.

Reproducibility and dependency gaps

The pitfall. The agent’s code runs in your working environment but fails in a fresh one. Imports resolve to packages you happened to have installed; seeds are missing; library versions are not pinned.

Why it is wrong. A December 2025 study, AI-Generated Code Is Not Reproducible (Yet), found execution fails in 31.7% of cases when AI-generated code is reproduced in clean environments. Agents declare on average about three packages while the working environment requires about 37.

The fix. Demand a lockfile (uv.lock, renv.lock, requirements.lock) for every project. Set seeds for random, numpy, torch, and cudnn. Verify by running in a fresh container before reporting success.

A meta-rule

The single best rule you can add to CLAUDE.md is:

When you cannot verify a claim from real output of a real command, say so explicitly. Do not guess.

Most pitfalls above are special cases of an agent guessing instead of running a verification step. A blanket rule that demands verification covers cases this page does not list.

Where to go next

• Apply the rules in your own CLAUDE.md.
• Re-read the Why Coding Agents page for the broader framing.
• The Reproducibility section covers the engineering practices that catch most agent mistakes before they propagate.