Deep Learning · VibeFormer

0. What you need and what you do not

Nobody will ask you to derive backpropagation. What they might ask is whether a deep model is appropriate for a given enforcement task, and what its use costs in reviewability. So this chapter covers the mechanics well enough that you can reason about them, and spends more time on the two architectures that genuinely matter here: transformers, and graphs.

1. One neuron, with arithmetic

A neuron does three things: multiply each input by a weight, add them up with a bias, then pass the total through a non-linear function.

z = \sum_{j=1}^{p} w_j x_j + b = \mathbf{w} \cdot \mathbf{x} + b \qquad\qquad a = \sigma(z)

The pre-activation z equals the sum of each input times its weight, plus a bias. Equivalently, the dot product of the weight vector and the input vector, plus b. The output a is an activation function applied to z.The bold letters mean *the whole list at once*, which is only notational shorthand. This is the entire neuron. The weights and the bias are learned; the activation function is chosen by a human. A network is this expression nested inside itself a few hundred thousand times.

One neuron, matching the worked example below

  bidder count  -0.8 ----(w = -0.6)----\
                                        \
  winning margin  1.4 ----(w =  1.2)-----+--> [ sum ] --> z = 2.58
                                        /       |
  bid spread      0.9 ----(w =  0.8)----/        |
                                               |
                       bias b = -0.3 ----------/
                                               |
                                               v
                                        [ sigmoid ]
                                               |
                                               v
                                         a = 0.93
                                     (high risk score)

Three inputs, three weights, one bias, one squashing function, one number out. The negative weight on bidder count is the interesting part — the network learned that more bidders lowers the risk score, which is the direction a competition economist would predict.

Activation	What it does	Used where
Sigmoid	Squashes to 0–1	Final layer for binary classification. Saturates at the extremes, which kills gradients
Tanh	Squashes to −1 to 1	Older recurrent networks. Same saturation problem
ReLU	Returns the input if positive, otherwise zero	The default hidden-layer choice. Cheap, and does not saturate on the positive side
GELU	A smoother ReLU	Standard inside transformers
Softmax	Turns a vector of numbers into probabilities summing to 1	Final layer for multiclass, and inside attention

2. From one neuron to a network

Term	Definition
Layer	A group of neurons all reading the same inputs
Multilayer perceptron	Fully connected layers stacked. Every neuron reads every output of the layer below
Hidden layer	Any layer that is not the input or output
Depth and width	Number of layers, and neurons per layer
Forward pass	Pushing input through to a prediction
Backward pass	Computing how each weight should change
Epoch	One complete sweep through the training data
Batch	The handful of examples processed before each weight update

3. How training works

Four steps, repeated millions of times: predict, measure the error, work out which direction each weight should move, move it a little.

Concept	Plain meaning
Gradient	For each weight, which direction and how steeply the loss would change if you nudged it
Backpropagation	An efficient way to compute every gradient in one backward sweep, by applying the chain rule from the output back toward the input. It is not a learning method — it only computes gradients
Gradient descent	The actual learning: move each weight a small step against its gradient
Learning rate	How big that step is. The most important hyperparameter in deep learning
Stochastic gradient descent	Using a small batch rather than the whole dataset per step. Noisier, far faster, and the noise itself helps escape poor solutions
Momentum	Keep some of the previous step's direction, so the path smooths out
Adam	Adapts the step size per weight using running averages. The default optimiser
Learning rate schedule	Lowering the rate over training, so you take big steps early and settle precisely later

w_j \;\leftarrow\; w_j - \eta \, \frac{\partial L}{\partial w_j}

Each weight is replaced by itself, minus the learning rate eta times the partial derivative of the loss with respect to that weight.This one line is all of learning. The arrow means *becomes*. The curly-d term is the gradient: how much the loss would rise if this weight rose. Subtracting it moves the weight the other way, downhill. Eta is the learning rate. Everything else — Adam, momentum, schedules — is a refinement of how far and in what direction to step.

\frac{\partial L}{\partial w^{(1)}} = \frac{\partial L}{\partial a^{(3)}} \cdot \frac{\partial a^{(3)}}{\partial a^{(2)}} \cdot \frac{\partial a^{(2)}}{\partial w^{(1)}}

The gradient of the loss with respect to a first-layer weight is the product of the gradients at each layer between them, multiplied together.This is backpropagation, and it is just the chain rule from calculus. The multiplication explains both failure modes in the table below: if each factor is a bit under 1, a long product shrinks toward zero — vanishing gradients. If each is a bit over 1, it explodes. Residual connections work by giving this product a path where one of the factors is exactly 1.

Problem	What happens	Standard fix
Vanishing gradients	In a deep stack, gradients shrink toward zero on the way back, so early layers barely learn	ReLU instead of sigmoid, residual connections, careful initialisation
Exploding gradients	Gradients grow instead, and weights diverge	Gradient clipping, lower learning rate
Bad initialisation	Start all weights at zero and every neuron in a layer computes the same thing forever	Random initialisation scaled to layer size — Xavier or He
Internal covariate shift	Each layer's input distribution drifts as the layer below learns	Batch normalisation, or layer normalisation in transformers
Dead ReLUs	A neuron stuck outputting zero for every input learns nothing again	Leaky ReLU, lower learning rate

4. Regularisation in deep networks

Method	What it does
Dropout	Randomly switch off a fraction of neurons on each training step, so no single pathway can be relied on. Switched off at prediction time
Weight decay	The L2 penalty from the machine learning chapter, applied to network weights
Early stopping	Halt when held-out performance stops improving
Data augmentation	Train on label-preserving variations of the input
Label smoothing	Train toward 0.9 rather than 1.0, which reduces overconfidence

5. Convolutional networks

Built for data with local spatial structure — images above all, where nearby pixels are related and the same pattern can appear anywhere.

Term	Definition
Convolution	Slide a small window of weights, a filter, across the input, computing a weighted sum at each position
Filter, kernel	The small weight patch. One filter detects one kind of local pattern
Feature map	The output of applying one filter everywhere
Stride and padding	How far the window moves each step, and whether the edges are extended
Pooling	Shrink a feature map by taking the maximum or average of each small region
Receptive field	How much of the original input one deep unit ultimately sees
Parameter sharing	The same filter weights are reused at every position — far fewer parameters than a fully connected layer

H_{\text{out}} = \left\lfloor \frac{H_{\text{in}} + 2p - k}{s} \right\rfloor + 1

The output size equals the input size, plus twice the padding, minus the kernel size, all divided by the stride, rounded down, plus one.The half-brackets mean *round down*. A 200-pixel input with a 3-by-3 kernel, no padding and stride 1 gives 198 — you lose one pixel at each edge because the window cannot hang off the end. Set padding to 1 and you get 200 back, which is why padding exists. Set stride to 2 and you get 99: stride is how a network downsamples. Worth being able to compute, because a mismatch here is the most common reason a vision model refuses to run.

6. Recurrent networks, and why they were replaced

Built for sequences. A recurrent network reads one element at a time, carrying a hidden state that summarises everything so far.

Term	Definition
Hidden state	A fixed-size vector carrying the memory of everything read so far
Backpropagation through time	Unrolling the sequence and backpropagating through every step
LSTM	Long short-term memory. Adds gates — learned switches deciding what to forget, what to store, and what to output — so information can survive many steps
GRU	A simpler gated design with fewer parameters and similar performance
Bidirectional	Two passes, forward and backward, concatenated. Only possible when you have the whole sequence up front

The bottleneck, on a legal sentence

*The agreement, which the parties concluded in Munich in March 2019 following several months of correspondence between their respective sales directors regarding capacity in the southern market, did not restrict competition.* By the time the network reaches did not restrict, the word agreement is forty tokens back — and everything it knows about it has been squeezed through one fixed-size hidden state, overwritten at every step. LSTM gates help but do not solve it. **A transformer's attention looks directly at *agreement* regardless of distance**, which is the entire reason it replaced recurrence for language. Worth knowing because your own CV has LSTM work, and the honest framing is that it is the previous generation for text while remaining perfectly reasonable for short numeric time series.

7. Attention and transformers

Chapter 9 covers this from the language side. Here is the mechanism itself, because it is the architecture behind almost everything current.

Term	Plain meaning
Query, key, value	Each token produces three vectors. The query is what this token is looking for; the key is what each token offers; the value is what it contributes if selected
Scaled dot-product attention	Match every query against every key to get relevance scores, divide by a scaling factor to keep them stable, softmax them into weights, then take the weighted sum of the values
Self-attention	Queries, keys and values all come from the same sequence
Multi-head	Several attention operations in parallel, each free to track a different kind of relationship
Causal mask	Prevents a token attending to anything after it. What makes generation possible
Positional encoding	Injects position, because attention alone is order-blind

8. Generative models, including GANs

Models that produce new data resembling their training distribution, rather than labelling existing data.

Model	How it works
Autoencoder	An encoder compresses input to a small code, a decoder rebuilds it. Trained to reproduce its own input, so the code must capture what matters
Variational autoencoder	The code is a distribution rather than a point, so you can sample it to generate new examples
GAN	Two networks in competition. See below
Diffusion model	Learn to reverse a gradual noising process: start from noise and denoise step by step into a sample. Behind current image generation, and the design of one of your own subject models
Normalising flows	Invertible transformations from a simple distribution to a complex one, with exact likelihoods

GANs, and the forger analogy that is actually accurate

Two networks trained against each other. The generator produces fake examples from random noise. The discriminator is shown a mixture of real and fake and must say which is which. The generator is rewarded for fooling the discriminator; the discriminator is rewarded for catching it. A forger and an inspector, each improving because the other does. Concretely: the generator invents synthetic tender records; the discriminator compares them against 40,000 real ones. Early on the fakes are obvious — negative bid values, impossible dates. As training proceeds the generator learns the real joint structure, including that bidder count and contract value are related. At equilibrium the discriminator cannot tell them apart, which means the generator has learned the distribution.

The GAN training loop

   random
   noise  z
     |
     v
  +-------------+     fake samples
  |  GENERATOR  | ------------------\
  |     G       |                    \
  +-------------+                     v
     ^                           +---------------+
     |                           | DISCRIMINATOR |  real or fake?
     |                           |      D        |
     |   real data  ------------>+---------------+
     |   (40,000 real                  |
     |    tenders)                     v
     |                           verdict + loss
     |                                 |
     +----- gradient: "that fooled -----+
            it / that did not"          |
                                        v
                             D updated to catch G better

   Two objectives, pulling opposite ways:
     G wants D to be WRONG
     D wants D to be RIGHT

The generator never sees the real data. Its only signal is the discriminator's reaction, passed back as a gradient. That is why the equilibrium is informative — the generator can only succeed by matching the real distribution well enough to be indistinguishable — and also why training is unstable, since both networks are chasing a target that moves whenever the other improves.

\min_{G}\max_{D} \; \mathbb{E}_{x \sim p_{\text{data}}}\big[\log D(x)\big] \; + \; \mathbb{E}_{z \sim p_{z}}\big[\log\big(1 - D(G(z))\big)\big]

The discriminator maximises, and the generator minimises, the same quantity: the expected log probability the discriminator assigns to real data, plus the expected log probability it assigns to fakes being fake.A minimax objective — one expression, two players, opposite directions. The squiggle means *drawn from*. Only the second term involves G, and G wants D of G of z to be large, so it wants that term small. There is no single loss being minimised, which is exactly why a GAN has no clean convergence criterion and you cannot tell it is done by watching a loss curve fall.

GAN failure	What happens
Mode collapse	The generator finds a few outputs that reliably fool the discriminator and produces only those. Technically successful, practically useless — it has learned a corner of the distribution, not the distribution
Training instability	The two networks oscillate instead of converging, because each is chasing a moving target
Vanishing discriminator gradient	If the discriminator becomes too good too fast, the generator receives no useful signal. Wasserstein GANs address this with a different objective
No likelihood	A GAN cannot tell you how probable a given sample is, unlike a VAE or a flow. It can only generate

Why GANs matter for your GDPR project specifically

The synthetic-data project in the GDPR chapter rests on this. An agency's screen cannot be audited externally because the data is confidential business information and personal data. Generate synthetic procurement data instead — statistically faithful, containing no real firm — and outsiders can audit the screen without ever touching the real records. The whole question becomes measurable: train the screen on real data and on synthetic, compare performance, and measure re-identification risk at each privacy setting. GANs are one route; differential privacy guarantees are the other half. And mode collapse is the specific danger, because a generator that has quietly learned only the common cases produces synthetic data on which an audit would systematically miss the rare ones — which in this field are the cartels.

9. Graph neural networks — the architecture that actually fits

Most data in competition work is relational. Who owns whom, who bid against whom, who sat on whose board, which decision cites which. Those are graphs, and a graph is the one data shape where the structure carries the signal.

Term	Definition
Node, vertex	An entity. A firm, a person, a decision, a tender
Edge	A relationship. owns, bid against, cites, shares a director with
Node features	Attributes of each node — turnover, sector, country
Degree	How many edges a node has
Neighbourhood	The nodes one step away
Directed, weighted	Edges may have direction (A owns B) and strength (47 shared tenders)
Homophily	Whether connected nodes tend to be similar. The assumption most graph networks rely on

h_v^{(k+1)} = \phi\Big( \underbrace{h_v^{(k)}}_{\text{itself}}, \;\; \underbrace{\textstyle\bigoplus_{u \in \mathcal{N}(v)} \psi\big(h_u^{(k)}\big)}_{\text{its neighbours, aggregated}} \Big)

The new representation of node v is a function of two things: its own current representation, and an order-independent aggregation of a transformation of each of its neighbours' representations.h-v is node v's vector, and the superscript is the round number. The circled plus is any order-independent aggregation — sum, mean, or max — and that constraint is the whole point: a node has no first neighbour, so the operation must give the same answer regardless of order. Swap in mean and you have a GCN; swap in attention weights and you have a GAT. Every graph network is this one line with different choices for the two functions and the aggregator.

Two rounds of message passing on a co-bidding graph

  ROUND 0 - every firm knows only itself
  ------------------------------------------------
        Acme        Borex        Civic        Delta
        (turnover,  (turnover,   (...)        (...)
         sector,     sector,
         region)     region)


  ROUND 1 - each node pulls from its direct neighbours
  ------------------------------------------------
                Acme
              /  |  \
       Borex     |    Civic        Acme now encodes:
                 |                "I co-bid often with
               Delta                3 mid-size regional
                                     construction firms"

  ROUND 2 - each node pulls again, so info travels 2 hops
  ------------------------------------------------
       Borex <--> Civic            Acme now encodes:
          \        /               "my competitors are
           \      /                 each other's only
            > Acme <                 competitors too"
           /      \
          /        \                <-- THAT is a closed
       Delta ---- (no outside         bidding ring, and it
                   edges)             is invisible in any
                                      single tender row

  STOP AT 2 OR 3. Go further and over-smoothing sets in:
  every node has aggregated everyone, so all the vectors
  converge and the structure you were looking for is gone.

Each round extends how far a node can see by one hop. The signal of a rotation scheme is not in any firm's attributes and not in any single tender — it is the closure of the group, which only appears at round two. The depth limit is a real constraint, not a tuning preference: over-smoothing means more rounds actively destroy the pattern.

Variant	How it aggregates
GCN	Averages neighbour representations, normalised by degree. The simplest version
GraphSAGE	Samples a fixed number of neighbours rather than using all, so it scales to large graphs
GAT	Uses attention to weight neighbours, so the model learns which connections matter rather than treating them equally
Message passing networks	The general framework all of the above are instances of

Problem	What it is
Over-smoothing	After too many rounds every node's representation converges toward the same thing, because everyone has aggregated everyone. Usually two or three rounds is the limit
Scalability	Real graphs are huge and neighbourhoods explode. Hence sampling approaches
Homophily assumption	These methods assume neighbours are similar. Sometimes the signal is the opposite — a firm connected to very unlike firms may be the interesting one
Explainability	Harder than for tabular models. The answer depends on a subgraph, so the explanation has to be a subgraph

10. The limits worth stating out loud

Data hunger. Deep models need far more examples than tree ensembles. Confirmed cartels number in the dozens, not the millions.
Opacity. Millions of parameters with no individually meaningful interpretation. The interpretability methods from the machine learning chapter are approximations, and unstable ones.
Adversarial examples. Tiny, deliberately chosen input changes can flip a confident prediction. A firm that understood the screen could in principle structure a bid to avoid it.
Distribution shift. Performance decays as the world moves away from the training data, and legal and market conditions move constantly.
Overconfidence. Reliably miscalibrated without correction, which is the precondition problem from section 4.
Cost and environmental footprint. Training and serving are expensive, which matters for an agency procurement decision and is a fair question to be asked about.

11. How this connects to ATLANTIS

Strand	What deep learning contributes, and costs
Accuracy, the data problem	Transformers are what make unstructured evidence usable at scale — seized documents, decisions, correspondence. That is the capability driving the data appetite the framework was not built for
Fairness	The opacity is the fairness problem's technical core. Article 296 wants retraceable reasons; this is the architecture least able to supply them, and the one most likely to be deployed
Institutional arrangements	Agencies themselves name explainability of deployed tools and human–machine interaction among their three biggest difficulties. Both are consequences of this chapter

12. What the panel brings to this chapter

Panel member	Connection
Thibault Schrepel	Complexity science is about non-linear dynamics, emergence and feedback — and a deep network is a non-linear system whose capabilities appear discontinuously with scale. His emergence defeats prediction argument is partly a claim about this architecture. He also co-authored with Pentland at MIT on foundation models, and his knowledge graph is a graph learning problem waiting to happen
Catalina Goanta	Measurement over unstructured content at scale, which is what these architectures are for
Georgiana Mirza	Digital ecosystems — and an ecosystem is a graph, so section 9 is the formal treatment of her object of study
Tijmen Wisman	SyRI failed on transparency and verifiability. An opaque architecture is the hard case for that holding, and the question of whether a deep model can ever satisfy it is unresolved

13. What is unexplored, and four projects

Project 1. Graph learning on co-bidding structure

The gap: Spain's tool maps relationships between firms, bids and individuals, and Greece analyses communications metadata — both explicitly relational — yet published work treats bid screening as a tabular problem. Nobody has asked whether message passing finds collusion structure that row-wise models cannot. The method: build the co-bidding graph from open procurement data, compute both classical network measures and learned node representations, and test whether graph structure adds predictive power over tabular features alone. Why it works: the graph is constructible from public data, the comparison is clean, and the answer is informative whichever way it falls. Produces: evidence about whether the relational structure matters, plus the vocabulary for arguing about what a subgraph-based explanation would have to look like under Article 296.

Project 2. Synthetic procurement data as the route to external audit

The gap: no outsider can audit an agency screen, because the data is confidential business information and personal data. So the accountability deficit is structural rather than political. The method: generate synthetic procurement data with a generative model plus differential privacy guarantees, then measure the trade-off — train the screen on real and on synthetic, compare performance, and measure re-identification risk at each privacy budget. Watch for mode collapse, because a generator that has learned only the common cases yields synthetic data on which an audit systematically misses the rare ones. Produces: the mechanism that would make independent audit possible at all, which is the precondition for everything else in the fairness strand.

14. Your CV, mapped onto this chapter

What you have	Where it lands	Why it fits
Deep learning foundations — ANNs, CNNs, RNNs, TensorFlow and Keras	All four projects	Enough to build, and more importantly enough to judge somebody else's architecture choice
LSTM and time-series work — ARIMA, LSTM	Project 1	Honest framing: the previous generation for text, still reasonable for short numeric series. Do not oversell it for language
Computer vision — OpenCV, image classification, Pillow	Dark patterns, which is Poland's tool	Interface classification is a vision task with a legal standard attached
Fine-tuning open-weight transformers — TinyLlama, Hugging Face, Gemma, Llama 3	Projects 3 and 4	You have worked inside these models rather than only calling them, which is the difference between discussing and auditing
Model evaluation discipline and calibration awareness	Projects 2 and 4	Overconfidence is the property that breaks the legal argument, and you already report agreement and intervals
Production serving at 28,000 monthly users, including inference latency	Any adoption question	Cost and latency are real procurement constraints for an agency, and you have met them

15. If you remember ten things

A neuron is a weighted sum plus a bias through a non-linearity, and the non-linearity is what makes depth mean anything.
Backpropagation computes gradients; gradient descent does the learning. They are different things.
The learning rate is the hyperparameter that most often decides whether training works at all.
Residual connections are why networks could go deep, by giving gradients a short path back.
CNNs win through parameter sharing — 320 weights where a dense layer needed 40 million — because the inductive bias is built in.
Recurrent networks squeeze history through one fixed state; attention looks directly at any position, which is why transformers replaced them for text.
Attention cost grows with the square of sequence length, which is why context is limited and why RAG exists.
A GAN is a forger and an inspector improving together, and mode collapse is its characteristic failure — which matters because it would make synthetic audit data miss the rare cases.
A graph network passes messages between neighbours, and two rounds reveal cluster structure invisible in any single row. Three of the field's real tools are graph problems.
On tabular data boosted trees usually win. Deep learning earns its place on text, images and graphs — and knowing where not to use it is the stronger signal.