AI Safety and Alignment

Separating the distinct problems bundled under one label: misuse, accidents, structural risk and misalignment, and why they need different responses.

Assumes: What an LLM Actually Is

Why writing down what you want is harder than it looks: proxies diverge from intent under optimisation pressure, and Goodhart's law is the general statement of it.

Assumes: What AI Safety Actually Means · Formulating a Learning Problem

Documented cases where agents maximised the stated reward while defeating its purpose, and what distinguishes a bug from a genuine optimisation success.

Assumes: The Specification Problem · The Reinforcement Learning Problem

The four mechanisms — regressional, extremal, causal and adversarial Goodhart — and how each shows up in metrics, benchmarks and reward models.

Assumes: Reward Hacking and Specification Gaming · Classification Metrics

Why models fail when deployment differs from training, the i.i.d. assumption as a safety property, and what robustness guarantees are achievable.

Assumes: What AI Safety Actually Means · Monitoring and Drift Detection

Imperceptible perturbations that flip predictions, why they exist geometrically, and the arms race between attacks and defences.

Assumes: Distributional Shift and Robustness · Adversarial Machine Learning as a Game

Attacks on the training pipeline rather than the input: poisoned corpora, trigger-activated backdoors, and why web-scale pretraining data is hard to defend.

Assumes: Adversarial Examples and Attacks · Data Curation for Fine-Tuning

Outer and inner alignment stated precisely, why a system can pursue the trained objective and still not pursue the intended one, and what mesa-optimisation means.

Assumes: Goodhart's Law in Machine Learning · RLHF with PPO

An honest account of preference-based alignment: what it demonstrably achieves, and the failure modes it cannot address — sycophancy, annotator bias and reward-model overoptimisation.

Assumes: The Alignment Problem · DPO and Direct Preference Optimisation

How to supervise a system on tasks humans cannot evaluate directly: debate, recursive reward modelling, weak-to-strong generalisation and their open problems.

Assumes: What RLHF Does and Does Not Fix · Markov Games and Multi-Agent Learning

Why black-box explanations are insufficient for safety, and the distinction between post-hoc explanation and mechanistic understanding.

Assumes: The Alignment Problem · Model Interpretability

Features, circuits and superposition; induction heads, sparse autoencoders and activation patching as tools for finding what a network computes.

Assumes: Interpretability: Reading the Model · Multi-Head and Masked Attention

Designing evals that measure dangerous capability rather than benchmark skill: capability versus propensity, elicitation, and why contamination makes most numbers unreliable.

Assumes: What RLHF Does and Does Not Fix · LLM Evaluation and Benchmarks

Why instruction-following and instruction-refusing conflict, direct versus indirect injection, and why no purely prompt-level defence has held.

Assumes: Evaluations and Red-Teaming · Safety and Prompt Injection

What changes when a model can act: irreversible actions, permission scoping, sandboxing, human-in-the-loop design and failure containment.

Assumes: Jailbreaks and Prompt Injection · LLM Agents

Designing deployments that remain safe even if the model is misaligned: monitoring, trusted-untrusted decomposition, and control evaluations.

Assumes: Safety of Tool-Using Agents · Scalable Oversight

The capability domains that trigger heightened caution — cyber, bio, persuasion, autonomous replication — and how frontier safety frameworks set thresholds.

Assumes: Evaluations and Red-Teaming

Making a structured, falsifiable argument that a system is safe enough to deploy, borrowing from aviation and nuclear practice.

Assumes: Dangerous Capability Evaluation · AI Control

What is genuinely unsolved, where the field disagrees, which claims are overstated in both directions, and how to read the literature critically.

Assumes: Safety Cases and Assurance · Mechanistic Interpretability