What are residual connections?
Abraham Lincoln once famously said, “Don’t believe everything you read on the internet!”
You may be wondering why I’d write a post about something as googleable as residual connections. Well, if you google “What are residual connections?”, you will likely find a few different explanations that residual connections are synonymous with skip connections.
These explanations are not correct.
The point of this post is to explain why residual connections are not skip connections and also provide an explanation of what they they actually are.
Residual connections were first introduced in Deep Residual Learning for Image Recognition by He et al., so I’ll be using this paper as an authoritative source to support the following definition for residual connections:
A residual connection is a learnable mapping that runs in parallel with a skip connection to form a residual block.
This definition introduces a new term “residual block” which is represented in the following figure:
The authors explain this figure by saying that “The formulation of F(x) + x can be realized by feedforward neural networks with ‘shortcut connections’.”
They go on to say that “Shortcut connections are those skipping one or more layers” and “F(x) represents the residual mapping to be learned”.
In other words, we can understand He et al. as saying that: A residual block can be realized by a residual connection with a skip connection. This definition for residual blocks is in agreement with our earlier definition for residual connections.
Thus, restating our previous definition, residual connections are not skip connections but work together with skip connections to form residual blocks.
Since we’ve now defined residual connections and residual blocks, we might as well define residual networks too: A residual network is a network which contains residual blocks and is trained to optimize residual connections.
Anyways, that’s all for this post. If you have any feedback or even if you totally disagree with me, please feel free to reach out.