Pillar One

AI Foundations & Risk Management

Understand what LLMs are, how they work, and why you can't fully trust their answers.

What Is an LLM?

It's different from the AI you use in your lab.

🔬

AI in Your Lab

Trained for a single task — image recognition, data classification. It can't switch tasks; it only recognizes the specific problem it was trained on.

💬

LLM (Large Language Model)

Trained on vast amounts of human-generated text, so you can interact with it through natural language. The underlying architecture is similar, but the training objective is completely different.

🎯

How Does It Generate Text?

Each step predicts "what the most likely next token is," building up output one token at a time. This is its core mechanism — and the source of all its limitations.

ChatGPT, Claude, and Gemini are all LLMs. So are DeepSeek and Qwen — but they carry additional data security risks. See "Smart Choice of AI" for details.

The Hallucination Problem

AI doesn't know when it's wrong — but you're the one signing the paper.

📄

Fabricated Papers

LLMs generate citations that look completely plausible — author names, journal names, years, page numbers, all present. But the paper doesn't exist.

🔢

Wrong Numbers

AI isn't calculating — it's predicting "what numbers statistically should look like here." Computed results and statistics always need independent verification.

😊

Sycophancy

AI is trained to please users. The more you push back, the more likely it is to capitulate and agree with you — even when you're wrong.

Verification Principles

  • All cited papers must be independently confirmed to exist via Google Scholar or PubMed
  • Numbers, statistics, and formulas may not be directly quoted from AI output — verify independently
  • When in doubt about an important conclusion, start a fresh conversation and ask again — if the two answers differ, that's a weak point

Why Does AI "Forget" Things?

This isn't a bug — it's a fundamental design characteristic.

🪟

Context Window

AI has a "working memory" limit called the context window. When a conversation grows too long, the oldest content gets pushed out — which is why AI forgets earlier parts of long conversations.

🔄

It Is Stateless

Every time you press send, AI reads the entire conversation from the beginning and "disappears" after replying. The app creates the illusion of memory by re-packaging your chat history — not the AI itself.

💡

Practical Tips

For long tasks, put critical instructions at the very beginning of the conversation and mark them IMPORTANT. When AI starts going off track, start a fresh conversation.

Further Viewing

Recommended videos for a deeper understanding of LLM mechanics and the hallucination problem.

Next

Once you understand the foundations, the next step is choosing the right tools.

Smart Choice of AI Tools →