It looks like a contradiction. A large cloud company announces its own custom AI chips, then places a huge order for a rival’s graphics chips. If the in-house chip is so good, why keep buying? The answer says a lot about how the cloud business works, and about what you pay when you rent a GPU by the hour.
Customers ask for specific hardware
A cloud provider sells whatever its customers want to rent. Over the past decade, much of the machine learning software ecosystem grew up around one vendor’s GPUs and the programming tools that go with them. Researchers learned on them, libraries were optimized for them, and code written for them runs there with the fewest surprises. A customer training a large model usually wants the environment that already works, and will pay for it.
If a provider offers only its own custom chip, it excludes customers whose software is tied to the dominant platform. So providers stock both: the popular GPU to meet demand today, and their own custom AI chips for workloads where it is cheaper to run.
Why design custom AI chips anyway
Three motives show up repeatedly.
- Cost. A vendor selling general-purpose accelerators builds in a profit margin. Buying thousands of units is a huge expense. A chip designed in-house, for the provider’s own workloads, can deliver more work per dollar and per watt, even after the design cost is accounted for.
- Supply. When demand surges, the best chips are in short supply and orders back up. Having an alternative reduces dependence on a single supplier’s schedule.
- Fit. Custom hardware can be tuned to particular tasks, like serving a recommendation model or running inference for an assistant, rather than the broadest possible set of uses.
Where custom AI chips fit: training versus inference
Training a model is a giant, expensive effort that happens occasionally. Inference, which means using the trained model to answer queries, happens constantly and at scale. Inference is where a modest saving per request adds up to a lot, so it is the natural first target for in-house chips. Training at the frontier still tends to rely on the established GPUs because of software maturity and raw performance.
This split explains why a provider can buy more of the standard chips and expand its own at the same time. Total demand for compute is growing, so both can increase together.
The software moat
Hardware is only half the story. The vendor that dominates has a mature stack of compilers, libraries and developer tools. Competing chips need software that makes it easy to port existing code, and building that takes years. Some projects aim to make models run on many kinds of hardware without rewriting, which would loosen the lock-in. Progress is real, but it is gradual.
A rough picture of the economics: designing a chip takes years of engineering, tools, test runs and manufacturing commitments, so custom AI chips only pay off at very large scale. A cloud provider running enormous numbers of identical requests can spread that fixed cost, while a typical company cannot. That is why the move is limited to the biggest operators, and why the rest of the market relies on off-the-shelf GPUs.
The arms race has side effects. Providers can end up competing with their own suppliers, a tension similar to when your biggest customers become your competitor. Huge infrastructure spending also sits beside belt-tightening elsewhere, as record revenue and job cuts at the same time explains. For background on the hardware, see Wikipedia’s article on the graphics processing unit.
What it means if you rent compute
- Compare prices per unit of work, not per hour. A cheaper chip that takes longer may cost the same or more.
- Test your actual workload on each option. Benchmarks on paper may not match your model.
- Check for software support: does your framework run well on the provider’s custom chip, and are there code changes to make?
- Consider availability. Popular instances can have waiting lists or require reservations.
- Avoid deep lock-in where you can. Keeping your code portable gives you bargaining power later.
Where custom AI chips are likely headed
The trend is toward a mixed fleet. Providers will keep offering the leading GPUs because customers demand them, and will steer suitable workloads to their own chips where the savings are largest. For the rest of us, competition between hardware sources should put downward pressure on the cost of compute over time, though demand growth can offset that for a while. The sensible stance is to treat hardware as a tool you choose per job, and to measure before committing.
A caution about headlines
Reports of big chip orders are easy to misread. An order can be spread over several years, may be conditional on delivery, and says little about how the hardware will be split between internal projects and customer rentals. Treat such figures as a signal of intent and not a precise forecast, and wait for actual availability and pricing on the provider’s own catalog before changing your plans.
The simplest takeaway is to measure on your own workload. Custom AI chips may cut your bill, or they may need code changes that cost more than they save. Ask your provider for a trial of both options and compare results, since custom AI chips and standard GPUs will sit side by side for some time.
Image: Original: Alexander Klepnev
Derivative work: IamMM / Wikimedia Commons, CC BY-SA 4.0.
