A primer: from CPUs to Triton

1.1 Why GPUs exist

A CPU (central processing unit) has a few smart cores optimized to run one instruction stream very fast: branch prediction, out-of-order execution, big caches. A GPU (graphics processing unit) makes the opposite bet: thousands of simple cores, grouped into a few dozen SMs (streaming multiprocessors, the GPU's version of a core cluster). It wins whenever you do the same simple operation on millions of numbers at once, which is exactly what neural networks do. The machine this project was built on has roughly 75x more raw arithmetic in its GPU than in its CPU.

1.2 Kernels, threads, warps, blocks

A kernel is a function launched on the GPU across many parallel workers. You write the body once; a grid of workers runs it. One worker is a thread; 32 threads execute in lockstep as a warp; up to ~1024 threads form a thread block, which runs on one SM and can share fast memory and synchronize. Write c[i] = a[i] + b[i] as a kernel and you write no loop: worker i handles element i, thousands at a time.

1.3 The memory pyramid (the actual game)

Two consequences drive everything:

• Coalescing. When a warp's 32 threads read 32 adjacent numbers, the hardware serves them in one wide transaction. Scattered reads can cost 32. Arranging data so warps touch adjacent memory is the difference between using 100% and 10% of the memory bandwidth.
• Latency hiding. The GPU keeps far more threads resident than executing: while one warp waits on memory, another computes. A kernel that cannot keep enough work in flight leaves the arithmetic units idle.

1.4 Why ML needs custom kernels: fusion

A neural network layer might compute softmax(x): normalize a row so it sums to 1. Done with library calls, that is three passes over the data, each reading the whole array from DRAM and writing it back. A fused kernel does all three steps in one pass: load once, compute on-chip, write once.

1.5 CUDA, Triton, Helion

CUDA (NVIDIA's GPU programming platform) gives full control in a C++ dialect, and demands full control back: manual thread indexing, shared-memory management, synchronization barriers, and tensor cores (dedicated matrix-multiply hardware, ~16x faster than ordinary arithmetic for that job) driven through special instructions. A production CUDA matmul is hundreds of lines, and one misplaced barrier means silently wrong answers.

Triton raised the level: you program a block of data, not threads. Here is a complete Triton-style softmax kernel:

@jit
def softmax_kernel(x_ptr, out_ptr, N, stride, BLOCK_N: constexpr):
    row = program_id(0)              # which row this instance handles
    cols = arange(0, BLOCK_N)         # a whole block of column indices
    mask = cols < N                   # guard the ragged edge
    x = load(x_ptr + row * stride + cols, mask=mask, other=-inf)
    x = x - max(x, axis=0)            # the whole row, in registers
    num = exp(x)
    out = num / sum(num, axis=0)
    store(out_ptr + row * stride + cols, out, mask=mask)

No threads, no warps, no barriers. The compiler decides how the block's elements spread across threads, how the reduction communicates, where barriers go, and how to hit the tensor cores. Internally, Triton is a serious industrial compiler: Python is lowered through an IR (intermediate representation) and the MLIR and LLVM compiler frameworks down to PTX (NVIDIA's portable GPU assembly), then SASS (the real machine code).

Helion sits one level higher still: PyTorch-like tile code goes in, and the compiler generates the Triton kernel and autotunes it, searching block sizes and other knobs automatically, verifying each candidate, and caching the fastest. Everything in this world compiles JIT (just-in-time): a kernel is compiled on first call for each combination of data types and compile-time constants, then cached.