{"cells": [{"cell_type": "markdown", "id": "18c2a355", "metadata": {"papermill": {"duration": 0.010187, "end_time": "2025-04-08T10:42:16.485695", "exception": false, "start_time": "2025-04-08T10:42:16.475508", "status": "completed"}, "tags": []}, "source": ["\n", "# Tutorial 2: Activation Functions\n", "\n", "* **Author:** Phillip Lippe\n", "* **License:** CC BY-SA\n", "* **Generated:** 2025-04-08T10:42:09.983114\n", "\n", "In this tutorial, we will take a closer look at (popular) activation functions and investigate their effect on optimization properties in neural networks.\n", "Activation functions are a crucial part of deep learning models as they add the non-linearity to neural networks.\n", "There is a great variety of activation functions in the literature, and some are more beneficial than others.\n", "The goal of this tutorial is to show the importance of choosing a good activation function (and how to do so), and what problems might occur if we don't.\n", "This notebook is part of a lecture series on Deep Learning at the University of Amsterdam.\n", "The full list of tutorials can be found at https://uvadlc-notebooks.rtfd.io.\n", "\n", "\n", "---\n", "Open in [{height=\"20px\" width=\"117px\"}](https://colab.research.google.com/github/PytorchLightning/lightning-tutorials/blob/publication/.notebooks/course_UvA-DL/02-activation-functions.ipynb)\n", "\n", "Give us a \u2b50 [on Github](https://www.github.com/Lightning-AI/lightning/)\n", "| Check out [the documentation](https://lightning.ai/docs/)\n", "| Join us [on Discord](https://discord.com/invite/tfXFetEZxv)"]}, {"cell_type": "markdown", "id": "c7d6b372", "metadata": {"papermill": {"duration": 0.008961, "end_time": "2025-04-08T10:42:16.503104", "exception": false, "start_time": "2025-04-08T10:42:16.494143", "status": "completed"}, "tags": []}, "source": ["## Setup\n", "This notebook requires some packages besides pytorch-lightning."]}, {"cell_type": "code", "execution_count": 1, "id": "fea60810", "metadata": {"colab": {}, "colab_type": "code", "execution": {"iopub.execute_input": "2025-04-08T10:42:16.520682Z", "iopub.status.busy": "2025-04-08T10:42:16.520506Z", "iopub.status.idle": "2025-04-08T10:42:17.708557Z", "shell.execute_reply": "2025-04-08T10:42:17.707299Z"}, "id": "LfrJLKPFyhsK", "lines_to_next_cell": 0, "papermill": {"duration": 1.199807, "end_time": "2025-04-08T10:42:17.711142", "exception": false, "start_time": "2025-04-08T10:42:16.511335", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.\u001b[0m\u001b[33m\r\n", "\u001b[0m"]}, {"name": "stdout", "output_type": "stream", "text": ["\r\n", "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.0.1\u001b[0m\r\n", "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython -m pip install --upgrade pip\u001b[0m\r\n"]}], "source": ["! pip install --quiet \"seaborn\" \"numpy <3.0\" \"torchmetrics >=1.0,<1.8\" \"torch >=1.8.1,<2.7\" \"matplotlib\" \"torchvision\" \"pytorch-lightning >=2.0,<2.6\""]}, {"cell_type": "markdown", "id": "945a130d", "metadata": {"papermill": {"duration": 0.013311, "end_time": "2025-04-08T10:42:17.738540", "exception": false, "start_time": "2025-04-08T10:42:17.725229", "status": "completed"}, "tags": []}, "source": ["