GPU (Graphics Processing Unit)

Why GPU Instead of CPU

CPUs are optimized for complex sequential processing and typically have only a few dozen cores. GPUs, on the other hand, can execute simple operations simultaneously across thousands to tens of thousands of cores. Neural network training is fundamentally a repetition of matrix operations, and this processing pattern aligns well with the parallel architecture of GPUs.

For example, when training a 70B parameter Dense Model, gradient calculations for each parameter must be performed in parallel. Computations that would take months on a CPU with sequential processing can be completed in days to weeks on a GPU cluster.

The Constraint of VRAM

When discussing GPUs in the context of AI, VRAM (Video RAM) is just as important as computational performance. All model weights and activations must be loaded into VRAM, and VRAM capacity effectively determines the upper limit on model size.

A single NVIDIA A100 (80GB) can accommodate roughly 40B parameters (in FP16). Running a 70B Dense Model requires at least 2 cards, and training one requires 8 or more. The reason LoRA and QLoRA attract so much attention is that they can dramatically reduce VRAM consumption.

Cloud vs. On-Premises

GPUs are expensive, with a single NVIDIA H100 costing several million yen. For this reason, many companies use cloud GPUs (AWS, GCP, Azure) on demand. On the other hand, when running large volumes of inference continuously, on-premises setups can be more cost-efficient, making this a critical decision in the operation of local LLMs.