Feed-Forward Network

The feed-forward network is the second sublayer of every transformer block, after attention. It is defined on one row vector $x$ with d components — not the token itself but what earlier blocks and this block's attention have made of it:

$W_1$ and $W_2$ are the weight matrices of the first and second layers; $b_1$ and $b_2$ are their biases, one component per unit in the layer. The first layer expands the row to the hidden width, a multiple of d — the expansion factor, 4× in GPT-2. GELU is applied to each hidden component on its own, and the second layer projects the row back to d.

Why it matters

Attention is the only place in the block where tokens exchange information. The feed-forward network transforms each token's vector separately: one token's $x$ never meets another's, and every token goes through the same $W_1$, $b_1$, $W_2$ and $b_2$. Those four are fixed while the model runs. Training is what changes them.

The expansion is what the feed-forward sublayer is for. $W_1$ maps $x$ into a higher-dimensional space, where each hidden unit computes one feature of it. Because the weights are shared, every token passes through the same space, one per transformer block. Training is what encodes into it what the model learns from the data.

GELU is what makes the expansion count. Without an activation, the two linear layers are one:

The right-hand side is a single layer with weights $W_1 W_2$ and bias $b_1 W_2 + b_2$. So without an activation the expansion buys nothing: one layer computes the same function, with far fewer parameters than the two layers use. In the widget above, set the activation to none and toggle between 2 layers and 1 layer: the output never changes.

GELU

GELU multiplies each input by $\Phi(x)$, the cumulative distribution function of the standard normal distribution:

GPT-2 was trained with an approximation of it that is faster to compute:

The choice of GELU over ReLU is about training. The gradient arriving at a unit is multiplied by the slope of its activation before it reaches the unit's weights, and ReLU's slope is exactly 0 below zero: a unit sitting there passes nothing back, and its gradient contribution to weight updates vanishes. GELU is smooth and its slope below zero is nonzero (except at its minimum, $x \approx -0.75$), so those units keep contributing to the learning process.

Parameters

That is twice the attention sublayer of the same block, whose four [d, d] projections with biases hold 2,362,368 parameters. Across GPT-2's (124M) twelve blocks the feed-forward networks hold 56,669,184 parameters, about 46% of the model, which makes the feed-forward network the largest component of GPT-2 by parameter count. The token embedding is second, at about 31%.

Takeaway

The feed-forward network is the ordinary part of the architecture, and it holds the largest share of the parameters, in GPT-2 and in current LLMs alike. Attention relates tokens to one another; the feed-forward network transforms each token's vector on its own. A transformer block is those two sublayers together.

References

Sebastian Raschka, Build a Large Language Model (From Scratch) (Manning, 2024). Section 4.3 implements the sublayer in PyTorch, which the code exported above follows.

Dan Hendrycks and Kevin Gimpel, Gaussian Error Linear Units (GELUs) (2016). The paper GELU comes from, and the source of the approximation GPT-2 was trained with.

Ashish Vaswani et al., Attention Is All You Need (2017). Section 3.3 describes the sublayer, there with ReLU rather than GELU.

The rows this page runs on are where the dropout chapter leaves the block — the attention sublayer, its shortcut, and the second layer normalization.