Activation function

How it is used

Common examples are ReLU, sigmoid, tanh and GELU. Without activation functions, even a deep network would behave like a single linear layer. Engineers choose or swap the activation function when they design an architecture.

Example sentences

1. Try ReLU as the activation function in the hidden layers and sigmoid at the output.

2. Which activation function does this transformer use?

Common collocations

• ReLU activation function
• choose an activation function
• non-linear activation function
• output activation

Related terms

• neural networkA model made of many connected units arranged in layers, loosely inspired by the brain. Each connection has a weight that is adjusted during training.
• layerOne level of a neural network: a group of units that takes the output of the previous level, transforms it and passes the result on to the next one.
• weightsThe learned numbers on the connections of a neural network, which decide how strongly each input affects the next layer. Together they hold what the model has learned.
• backpropagationThe method that calculates how much each weight in a neural network contributed to the error, by passing the error backward from the output layer toward the input.

This word in other fields

The same English word can have a different meaning in another field:

• activation function · Software Engineering

More terms: Neural Networks and Deep Learning

• deep learning
• parameter
• transformer
• attention