\n", "\n", "The [GoogleNet](https://arxiv.org/abs/1409.4842), proposed in 2014, won the ImageNet Challenge because of its usage of the Inception modules.\n", "In general, we will mainly focus on the concept of Inception in this tutorial instead of the specifics of the GoogleNet, as based on Inception, there have been many follow-up works ([Inception-v2](https://arxiv.org/abs/1512.00567), [Inception-v3](https://arxiv.org/abs/1512.00567), [Inception-v4](https://arxiv.org/abs/1602.07261), [Inception-ResNet](https://arxiv.org/abs/1602.07261),...).\n", "The follow-up works mainly focus on increasing efficiency and enabling very deep Inception networks.\n", "However, for a fundamental understanding, it is sufficient to look at the original Inception block.\n", "\n", "An Inception block applies four convolution blocks separately on the same feature map: a 1x1, 3x3, and 5x5 convolution, and a max pool operation.\n", "This allows the network to look at the same data with different receptive fields.\n", "Of course, learning only 5x5 convolution would be theoretically more powerful.\n", "However, this is not only more computation and memory heavy but also tends to overfit much easier.\n", "The overall inception block looks like below (figure credit - [Szegedy et al. ](https://arxiv.org/abs/1409.4842)):\n", "\n", "
![](\"https://github.com/Lightning-AI/lightning-tutorials/raw/main/course_UvA-DL/04-inception-resnet-densenet/inception_block.svg\")
![](\"https://github.com/Lightning-AI/lightning-tutorials/raw/main/course_UvA-DL/04-inception-resnet-densenet/tensorboard_screenshot_GoogleNet.png\")
![](\"https://github.com/Lightning-AI/lightning-tutorials/raw/main/course_UvA-DL/04-inception-resnet-densenet/resnet_block.svg\")
![](\"https://github.com/Lightning-AI/lightning-tutorials/raw/main/course_UvA-DL/04-inception-resnet-densenet/resnet_notation.svg\")
![](\"https://github.com/Lightning-AI/lightning-tutorials/raw/main/course_UvA-DL/04-inception-resnet-densenet/resnet_loss_surface.png\")
![](\"https://github.com/Lightning-AI/lightning-tutorials/raw/main/course_UvA-DL/04-inception-resnet-densenet/tensorboard_screenshot_ResNet.png\")
\n", "\n", "[DenseNet](https://arxiv.org/abs/1608.06993) is another architecture for enabling very deep neural networks and takes a slightly different perspective on residual connections.\n", "Instead of modeling the difference between layers, DenseNet considers residual connections as a possible way to reuse features across layers, removing any necessity to learn redundant feature maps.\n", "If we go deeper into the network, the model learns abstract features to recognize patterns.\n", "However, some complex patterns consist of a combination of abstract features (e.g. hand, face, etc.\n", "), and low-level features (e.g. edges, basic color, etc.).\n", "To find these low-level features in the deep layers, standard CNNs have to learn copy such feature maps, which wastes a lot of parameter complexity.\n", "DenseNet provides an efficient way of reusing features by having each convolution depends on all previous input features, but add only a small amount of filters to it.\n", "See the figure below for an illustration (figure credit - [Hu et al. ](https://arxiv.org/abs/1608.06993)):\n", "\n", "
![](\"https://github.com/Lightning-AI/lightning-tutorials/raw/main/course_UvA-DL/04-inception-resnet-densenet/densenet_block.svg\")
![](\"https://github.com/Lightning-AI/lightning-tutorials/raw/main/course_UvA-DL/04-inception-resnet-densenet/tensorboard_screenshot_DenseNet.png\")
| Model | Val Accuracy | Test Accuracy | Num Parameters |
|---|---|---|---|
| GoogleNet | 90.40% | 89.71% | 260,650 |
| ResNet | 91.84% | 91.06% | 272,378 |
| ResNetPreAct | 91.80% | 91.07% | 272,250 |
| DenseNet | 90.72% | 90.23% | 239,146 |