Transformers in Deep Learning 

01. Introduction

In 2017, a Google research team published "Attention Is All You Need" — a paper that became the most influential work in modern AI. The Transformer architecture abandoned recurrence entirely, relying on self-attention to process sequences in parallel.

"The dominant sequence transduction models are based on complex recurrent or convolutional neural networks. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms." — Vaswani et al., 2017

02. The Architecture

The Transformer follows an encoder-decoder structure. Both halves are composed of stacked identical layers containing multi-head self-attention and position-wise feed-forward networks.

def scaled_dot_product_attention(Q, K, V, mask=None):
    # Calculate scores
    scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)
    
    if mask is not None:
        scores = scores.masked_fill(mask == 0, -1e9)
        
    # Apply softmax to get attention weights
    weights = torch.softmax(scores, dim=-1)
    
    # Return weighted values
    return torch.matmul(weights, V)

Evolution of Transformers