How the modern GPU market works
An interactive 3D explainer: who designs AI chips, how they are made and where production gets stuck (TSMC, CoWoS packaging, HBM memory), a shelf of 12 NVIDIA and AMD accelerators, who buys the GPUs and what an hour of rental costs. Every number links to a vendor spec, an SEC filing or a cloud price page.
Above is an eight-step scene. Press "▶ Play" to let it run, or use "Next" and the arrow keys. Drag to rotate; zoom with the wheel, a pinch or the "+" and "−" buttons. Underlined words open the glossary. On step three, click any card to open its passport. L hides labels, F hides every panel.
In short
One set of companies designs, another builds. NVIDIA and AMD sell accelerators to anyone; Google and Amazon make chips only for their own clouds. None of them owns a fab — TSMC prints almost all of it.
Assembly is the bottleneck. A chip is printed on a silicon wafer, cut into dies, placed next to stacks of HBM memory on a shared silicon interposer and assembled into a CoWoS package. Only three companies make HBM, CoWoS packaging is TSMC's, and TSMC itself says demand for it grew even greater after generative AI took off.
Memory is what separates the cards. A100 has 80 GB at 2 TB/s; B300 and MI355X have 288 GB at 8 TB/s. Capacity decides whether a model fits on one card; bandwidth decides how fast it answers.
The giants buy. In calendar 2025 Amazon spent $131.8B on property and equipment, Alphabet $91.4B, Meta $69.7B. Microsoft's fiscal year ends on 30 June: $64.6B for the year to 30 Jun 2025. AI-first clouds are smaller: CoreWeave $10.3B, Nebius $4.1B in 2025.
An H100 hour costs $3.99 to $6.88 at public list prices as of 2026-10-09: Lambda $3.99, Nebius $4.50, CoreWeave $6.16, AWS $6.88. B200 and B300 cost 1.4–2.6× more: from $8.60 / $6.16 ≈ 1.4 at CoreWeave to $17.80 / $6.88 ≈ 2.6 at AWS.
Rent or buy
One example: an 8 × H200 server. Buying costs $392,703 — the configured price in reseller Thinkmate's price list. Renting (prices as of 2026-10-09): Nebius 8 × $5.40 = $43.20 an hour, CoreWeave $50.44 an hour (Lambda lists no H200). An owned server also uses power: 10.2 kW (NVIDIA's maximum for such a server) × $0.1453 per kWh (US average commercial rate, July 2026) ≈ $1.48 an hour.
Payback = price ÷ (rent − power). Against Nebius: $392,703 ÷ $41.72 ≈ 9,413 hours of use; against CoreWeave: ÷ $48.96 ≈ 8,021 hours. A month has 730 hours, so utilisation decides:
| 50% busy | 90% busy | |
|---|---|---|
| Nebius | 9,413 ÷ (730 × 0.5) ≈ 25.8 mo | 9,413 ÷ (730 × 0.9) ≈ 14.3 mo |
| CoreWeave | 8,021 ÷ 365 ≈ 22.0 mo | 8,021 ÷ 657 ≈ 12.2 mo |
A new generation arrives about every two years: H100 was announced on 2022-03-22, B200 on 2024-03-18. At 50% utilisation the server pays back roughly when it stops being the fastest; at 90% it does in about a year.
The threshold is about 50% utilisation. If the server serves two years, until the next generation, buying beats renting once it is busy at least half the time: from 46% against CoreWeave (8,021 ÷ (730 × 24)), from 54% against Nebius (9,413 ÷ (730 × 24)). Below that the server sits idle more than it works, and it still costs money: the GPUs are already paid for and lose value every month; power, rack space and repairs are on the owner too. A rented server costs nothing when idle: switch it off and the bill stops. Rack space, network and salaries are not included — with them the threshold is higher still.
What else buying includes:
- Thinkmate's price includes a 3-year return-for-repair warranty. Advance parts replacement with next-business-day onsite service: +$450 for 3 years, +$32,166 for 5 years.
- In colocation the data centre gives space, power, cooling and "remote hands" on request. It does not diagnose GPUs, handle vendor returns, firmware or drivers — that takes your own engineers or a service contract (Supermicro and Dell quote on request).
- While a broken GPU is out for warranty replacement, the server is down. When renting, that is the provider's problem.
- There is a middle ground — managed hosting: own hardware, serviced by the provider. Financially that is already close to renting.
- Rack space, network and salaries are not in the numbers: there are no public prices, and they only push payback further out.
Bottom line. Buying pays if the server will be busy more than half the time for the two years before it is outdated, or if data may not leave the premises. If utilisation is below 50% or hard to predict, renting is cheaper — it also fits peaks and experiments, when a new GPU is needed now and there is no own site or staff. Buying also means people or a service contract: if the racks are not yours, the data centre will not do this work.
What this means for a small team
Buying GPUs is rarely needed: one H100 at Lambda is $3.99/h × 24 h ≈ $96 a day, eight are $96 × 8 ≈ $766 a day. The choice starts with how much memory the model needs. If 80 GB is enough, the cheapest H100 will do. If not — H200 (141 GB), MI300X (192 GB) or B200 (180 GB). A small model is fine on an L40S (48 GB) or RTX PRO 6000 (96 GB). AI-first clouds are cheaper than the giants, but free capacity is not guaranteed.
All sources and how we calculated (checked 2026-10-09)
| What | Value | Source |
|---|---|---|
| A100 SXM | 80 GB HBM2e, 2,039 GB/s, FP16 312 TFLOPS (624 sparse ÷ 2), 400 W, NVLink 600 GB/s; announced 2020-05-14 | nvidia.com/a100, NVIDIA, 2020 |
| H100 SXM | 80 GB, 3.35 TB/s, 700 W; FP8 1,979 / FP16 989.5 TFLOPS (3,958 / 1,979 sparse ÷ 2) | nvidia.com/h100 |
| H200 SXM | 141 GB, 4.8 TB/s, 700 W | nvidia.com/h200 |
| GH200 | 144 GB HBM3e, 4.9 TB/s, FP8 1,979 TFLOPS, 450–1,000 W for the whole board, NVLink-C2C 900 GB/s; in production 2023-05-28 | NVIDIA, PDF, NVIDIA, 2023 |
| B200 per GPU | 1,440 GB ÷ 8 = 180 GB; 64 TB/s ÷ 8 = 8 TB/s (DGX B200, 8 GPUs) | DGX B200 |
| B200/B300 FP8 per GPU | 72 PFLOPS sparse per HGX board ÷ 2 ÷ 8 = 4.5 PFLOPS | HGX |
| B300 per GPU | 288 GB, 8 TB/s, up to 1,400 W | NVIDIA Technical Blog |
| GB200 Superchip | 372 GB HBM3E, 16 TB/s, FP8 20 PFLOPS sparse ÷ 2 = 10 PFLOPS, NVLink 3.6 TB/s | nvidia.com/gb200-nvl72 |
| GB300 Superchip | 2 × 288 GB = 576 GB; 2 × 8 TB/s = 16 TB/s; FP8 720 PFLOPS per rack ÷ 36 ÷ 2 = 10 PFLOPS; FP16 360 ÷ 36 ÷ 2 = 5 PFLOPS | NVIDIA Technical Blog, nvidia.com/gb300-nvl72 |
| L40S | 48 GB GDDR6, 864 GB/s, FP8 733 TFLOPS, 350 W, 4.4" × 10.5" (≈ 112 × 267 mm); announced 2023-08-08 | nvidia.com/l40s, NVIDIA, 2023 |
| RTX PRO 6000 Blackwell Server Edition | 96 GB GDDR7, 1,597 GB/s, FP8 2 PFLOPS (mode not stated), up to 600 W; announced 2025-03-18 | NVIDIA, NVIDIA, 2025 |
| MI300X | 192 GB, 5.3 TB/s, FP8 2,614.9 TFLOPS, 750 W | AMD, PDF |
| MI355X | 288 GB, 8 TB/s, FP8 5,033 TFLOPS, 1,400 W, TSMC 3/6 nm | AMD, PDF |
| MI300X / MI355X module size | 102 × 165 mm — OCP OAM standard, not from AMD | OCP OAM v1.5 |
| GB200 NVL72 | 72 GPUs, 36 CPUs, 13.4 TB memory, liquid-cooled | nvidia.com/gb200-nvl72 |
| Wafer | 300 mm (12-inch) | TSMC GIGAFAB |
| H100 / Blackwell process; transistors | TSMC 4N / 4NP; 80 and 208 billion | NVIDIA, 2022, NVIDIA, 2024 |
| CoWoS packaging | dies + HBM stacks on one interposer up to ~2,700 mm² (√2,700 ≈ 52 mm); demand rose after late 2022 | TSMC |
| Property and equipment spend | Amazon $131.8B, Alphabet $91.4B, Meta $69.7B, CoreWeave $10.3B, Nebius $4.1B (all calendar 2025); Microsoft $64.6B (fiscal year 1 Jul 2024–30 Jun 2025, not calendar) | 10-K / 20-F filings via SEC: Amazon, Alphabet, Meta, Microsoft, CoreWeave, Nebius |
| AWS, US East, Linux, on-demand | p5 (8× H100) $55.04/h ÷ 8 = $6.88; p5en (8× H200) $63.30 ÷ 8 = $7.91; p6-b200 $113.93 ÷ 8 = $14.24; p6-b300 $142.42 ÷ 8 = $17.80 | AWS |
| CoreWeave, on-demand | HGX H100 $49.24 ÷ 8 = $6.16; H200 $50.44 ÷ 8 = $6.31; B200 $68.80 ÷ 8 = $8.60 | CoreWeave |
| Nebius, on-demand, rates from 2026-10-01 | H100 $4.50, H200 $5.40, B200 $8.50, B300 $9.50 per GPU-hour | Nebius |
| Lambda | H100 $3.99, B200 $6.69 per GPU-hour; no H200 listed | Lambda |
| 8 × H200 server | $392,703 — configured price of GPX GXS5-2508-8HGX-H2 in the reseller's price list; 3-year return-for-repair warranty included, onsite +$450 (3 yr) / +$32,166 (5 yr) | Thinkmate |
| 8 × H200 server power | 10.2 kW — NVIDIA's maximum for DGX H100/H200 | NVIDIA |
| Electricity | 14.53 ¢/kWh — US average commercial rate, July 2026 | EIA |
NVIDIA does not publish SXM module or Superchip outlines and we found no verifiable measurement in reviews, so their size on the scene is a placeholder.