Build the computation graph
Consider one scalar input x, a weight w, a bias b and a binary label y. The logit is z = w x + b. Convert it to a probability with p = sigmoid(z) = 1 / (1 + exp(−z)). Binary cross-entropy is L = −y log(p) − (1 − y) log(1 − p), using natural logarithms. Choose x = 2, w = 0.5, b = −1 and y = 1. The logit is zero, the predicted probability is 0.5, and the loss is log(2), approximately 0.6931471806. This is one logistic output unit. It is useful for understanding a network's local derivatives, but it is not an example of training an entire language model. Draw the graph as x,w → multiplication → addition with b → sigmoid → loss. Every arrow is a dependency. During the backward pass, each node combines its local derivative with the gradient arriving from its consumer. If a value feeds several branches, its gradient receives the sum of those branches' contributions.