About the NVIDIA A100 GPU

The NVIDIA A100 is based on NVIDIA's Ampere GPU architecture. It's a powerhouse in the realm of GPUs, catering to demanding applications in AI, data analytics, and high-performance computing.

The A100 is fast. Its 432 third-generation Tensor Cores and 6912 CUDA Cores enable it to providing significant improvements in performance for AI and machine learning applications compared to previous generation GPUs

Apart from pure performance, its massive 80 GB of VRAM and 2 TB/s of memory bandwidth make it ideal for large-scale LLM, data analytics, and scientific computing tasks that require large amounts of fast memory.

The A100 also boasts advanced features like Multi-Instance GPU (MIG), enabling it to efficiently serve multiple workloads simultaneously — up to 7 10GiB VRAM instances per physical GPU. Overall, the NVIDIA A100 represents a significant leap forward in GPU technology, offering strong performance and efficiency for the most demanding computing tasks.

See full data sheet.

5,586 tok/sec LLaMa 7B

A token is a word or subword, so the A100 can generate 19,694 words per second on the LLaMa 7B model. Batch size 384.

6,912 CUDA Cores

CUDA cores are the basic processing units of NVIDIA GPUs. The more CUDA cores, the better.

432 Tensor Cores

Tensor cores are specialized processing units that are designed to efficiently execute matrix operations, used in deep learning.

624 Teraflops FP16

With sparsity.

80 GB VRAM @ 2 TB/s

The more VRAM, the more data a GPU can store at once. The A100 80GB PCIE has 1,935 GB/s of memory bandwidth, while the A100 80GB SXM has 2,039 GB/s.