Introduction:
This month, we have been thinking a lot about hardware, including Nvidia earnings, AMD acquisitions, Cerebras IPO filing, Llama perf numbers, and more.
So, this month’s updates are hardware-centric!
UPDATES – ML at the Edge:
Looking at our work over the last six months, we have worked with several leading start-ups, bringing the power of machine learning to the edge and on device. The market is valued at over $15 billion and has colossal growth potential, driven by the increasing need for near real-time data processing. It throws up many challenges, like connectivity, smaller models, latency, power budgets, security, and privacy. This market will need innovation at the hardware, device, and foundation model levels, so we are working on these.
Here are a few companies we are working with:
1. We hired Kunal Shah for Index Ventures-backed Cartesia, a pioneer in developing State Space Models (SSM). The Founders are the creators of Mamba. SSMs offer a new primitive that promises higher quality and efficiency training large-scale foundation modes. You can check out their flagship model, Sonic. Kunal joins a talented early team to focus on data engineering. Cartesia wrote a blog post last week on implementing SSMs on device.
2. We hired Chris Sine for Quadric, building an AI processor optimized for on-device AI to run CNN-based models (including vision transformers) at the edge (cars, robots, game consoles, smartphones, and more). Quadric puts programmability and flexibility front and center using DevStudio. Chris is helping Quadric architect and scale their design verification team.
3. We concluded a six-month project with the SiMa engineering team. SiMa now supports multi-modal networks on its edge-based ML chips (which you can read about here). Traditionally, edge chips focus on accelerating computer vision, but you will increasingly see them incorporate LLMs and transformers to offer multi-modal Gen AI.
4. We are supporting Useful Sensors founders Pete Warden and Manjunath Kudlur. Useful Sensors leverage Tiny ML to make devices smarter without the internet. This is a mammoth market bringing ML to everyday objects. OEMs can integrate sensors to make almost every product you can think of intelligent.
5. We worked with a Madrona-backed start-up called Lassen Peak on their FPGA-based sensor module technology. Lassen Peak is changing the game for Police agencies worldwide by creating a handheld concealment weapon detection system using Terahertz frequency imaging radar. This is one of the most inspiring technologies I have encountered in 2024, making Police work much safer.
MARKET INSIGHT – AI Hardware
Last month, Cerebras Systems announced a confidential IPO filing. This is an anticipated moment for the AI hardware industry. It is the first public offering amongst a generation of hardware start-ups trying to compete with Nvidia. This is also significant to the general start-up market, which is hurting from a lack of liquidity. Are things improving and what does this mean for the specialized compute market?
The evolution of AI hardware has come in three major waves. Each wave of innovation builds on the last and must continue evolving to keep pace with the next. While start-ups like Groq and Cerebras Systems have gained more traction since the release of GPT, others, like Graphcore, have fallen behind.
Wave 1 – Nvidia
1. Nvidia released AlexNet in 2012, which the company says led to a breakthrough in the modern era of AI.
2. Google introduced its first tensor processing unit (TPU) 2015 as an AI accelerator AISC for machine learning workloads.
3. Intel acquired Nervana Systems in 2016 to enter the deep learning training chips market.
Wave 2 – Training
1. Cerebras Systems was started in 2015 by the former group of founders at SeaMicro (acquired by AMD). This is a story of “re-invention.” Companies like Trilogy and Texas Instruments tried and failed to build waferscale chips in the 1970s and 1980s.
2. Groq was started in 2016 by a Google TPU designer. It has benefited from the popularity of open-source models like Llama.
3. Nigel Toon, a well-known semiconductor entrepreneur in the UK, started Graphcore in 2016. When Softbank acquired Graphcore, the price was less than the money raised. This is a missed opportunity for the European semiconductor industry unless the technology within Softbank can become more significant.
4. Habana Labs was founded in 2016 by Avigdor Willenz, David Dahan and Ran Halutz. Habana was acquired by Intel in 2019 for $2bn. Through its product, Gaudi, Habana formed Intel’s accelerator strategy following the termination of Nervana Systems’ Crest family of processors. Intel Capital backed Habana.
5. Two Stanford Professors and a former Oracle executive launched Samba Nova Systems in 2017, backed by Intel Capital.
6. Hyperscalers started making their own chips. Amazon acquired Annapurna, Tesla started building a supercomputer called Dojo, Apple broke a 15-year partnership with Intel to develop ARM-based chips, and Meta created a silicon team in Reality Labs for VR.
Wave 3 – Inference and hyper-specialized chips
1. Etched AI made a bet on transformer models by burning the transformer model architecture onto the chip. We worked with Etched from day one to hire some of their founding team, and it has been a phenomenal story of fast execution as they raised $120m in June with funders, including Peter Thiel.
2. d-Matrix had the distinct advantage of coming in on the third wave. The founders came from Inphi and saw AI adoption accelerate, specifically the AI inference opportunity that could dwarf AI training.
3. There has been an explosion of specialized hardware start-ups. A few that I think are notable:
a. MatX
i. Designed only for huge models, it was founded by well-known founders from Google. Reiner Pope was a tech lead of PaLM, while Mike Gunter was a top architect in the TPU team. They built an excellent early team, including others from the Google TPU team.
b. Extropic
i. Backed by HOF Capital, which has also invested in OpenAI, XAI, and others, Extropic is focused on increasing chip density at dramatically lower cost. I don’t know much about the technology, but it seems like a company focused on paradigm-shifting ideas rather than incremental improvements. I’m excited to see them emerge from stealth.
c. Celestial
i. They have a serial entrepreneur at the helm, Dave Lazovsky, and are the most likely start-up to bring the promise of photonic fabrics to market. They can integrate their technology into existing chip designs, so they won’t have to displace existing chips. We will likely see many chips using Celestial fabrics in the future!
d. Touch
i. Who would be against Avigdor Willenz. No more to say… https://en.globes.co.il/en/article-serial-entrepreneur-avigdor-willenz-founds-new-chip-startup-1001487294
What next?
· First mover advantage?
o Wave 2 companies have more mature software stacks, but Wave 3 is building chips native to inference. They got to see how the market was shaping up. You will see Wave 2 companies developing inference products and Wave 3 hiring software engineers aggressively! Hardware takes time, so 2016 vintage start-ups have a significant advantage in building a company to go public.
· Nvidia dominance softening?
o Yes. I do see this.
(1) Rise of AMD. Lisa Su is proving to be one of the best CEOs of our generation. AMD has filled the void left by Intel. AMD are making smart talent bets like acquiring Silo AI for $665m, which brings 300 top AI scientists. Last week, AMD announced the acquisition of ZT Systems to match the breadth of data center products offered by Nvidia. In contrast, Intel announced they will cut more than 15,000 jobs, and morale is low, with many long-time engineers now looking outside the company.
(2) Nvidia slipped up. After years of flawless execution, Nvidia announced delays in their much-anticipated Blackwell product. JP Fricker, Chief Architect at Cerebras Systems, explains here.
(3) Start-ups. Cerebras, Groq, and Samba Nova are each winning large customers. AMD has arrived. Nvidia will continue to grow, but market share will shift over time.
· Is this a bubble?
I don’t believe there is an AI bubble. Demand has surged so high that it has created an imbalance between hardware investment and return on this investment. I believe in Brad Gerstenr’s thesis-driven approach to investing (https://blog.eladgil.com/p/altimeters-brad-gerstner-on-macro). There are market fluctuations, but trends play out over the long term. This only becomes a bubble if the value of AI is less than anticipated. I don’t see this playing out.
CONCLUSION
I am bullish on the long-term horizon of AI hardware companies outside Nvidia. The geopolitical environment is going to be a significant factor. It affects the IPO market, which means less liquidity for later-stage start-ups. A powerful combination of maturing software stacks, open-source models, and tremendous customer demand exists. The upcoming US election puts a hold on 2024 IPOs, but 2025 could be a landmark year for AI hardware start-ups if the election spurs an economic uplift (good article from WSJ here).