Megaport Connect Cloud Networking Summit
A first-of-its-kind virtual summit built to help network engineers and IT leaders upskill, share ideas, and connect around the future of cloud and hybrid networking
The Next Cloud Is AI Infrastructure: GPUs, Storage, and High-Performance Networking
Robert Benoit
Vice President, WW Solutions Engineering, VAST Data
Rob is Vice President of WW Solutions Engineering at VAST Data with 28+ years in enterprise technology. He helps organizations adopt scalable AI, data, and storage solutions, combining deep engineering expertise with a customer-first approach to drive innovation.
David Boland
Vice President Cloud Strategy, Wasabi Technologies
As Wasabi's Vice President of Cloud Strategy, David Boland plays an integral role in identifying innovative opportunities to advance Wasabi's long-term strategic direction. His team serves as a central driver of market and competitive intelligence, and strategic innovation options across compute, storage, networking, security, and AI services.
The Next Cloud Is AI Infrastructure: GPUs, Storage, and High-Performance Networking – Key Takeaways
Overview
Robert (Rob) Benoit, Vice President, Worldwide Solutions Engineering at VAST Data, and David Boland, Vice President Cloud Strategy at Wasabi Technologies, present a joint session on what it takes to build infrastructure that can support AI workloads. Rob Benoit brings more than 28 years of experience in enterprise technology and engineering, while David Boland focuses on cloud strategy and innovation across compute, storage, networking, security, and AI. The two previously worked together at NetApp.
The session follows the AI pipeline from raw data to inference. David Boland grounds it in a real customer example, with names changed, built around an application that identifies food and tells the user whether an image is a hot dog, and uses it to show where object storage and high-performance file and block storage each fit. Rob Benoit covers the infrastructure shifts behind large-scale AI, the new data types AI creates, the requirements these place on the data plane, and examples including retrieval-augmented generation and edge-to-core AI factories.
The central theme is that the next cloud is built around data movement. Compute, networking, and storage, with data flowing between them, have to operate as one distributed computer. Buying GPUs is only one part of the equation: if storage cannot keep up or the network cannot move data fast enough, very expensive GPUs sit idle.
1. AI Infrastructure Works as One Distributed Computer
Rob Benoit asks the audience, many of them network engineers, to think past connectivity. AI infrastructure is a single distributed system of compute, networking, and storage, with data as the layer that runs across all of them. For that computer to work, every piece needs to scale at the same pace while supporting every workload placed on it. One of the largest environments VAST Data works with connects GPUs over hundreds of switches and thousands of connections to about an exabyte of capacity, all operating as one computer.
He explains why traditional storage struggles in this model:
- Jobs move. Workloads move across GPUs, pools, and even sites, so access between sites has to be seamless.
- Accelerators must stay busy. Whether they are GPUs, TPUs, DPUs, or IPUs, which are emerging with disaggregated inference, keeping them working requires access to the data for all jobs, not just one.
- Data is shared. The same corpus is accessed by many workloads at once for different use cases and simulations.
- Data has a lifecycle. Data moves through model training and eventually inference in a constant cycle that Rob Benoit describes as a flywheel.
Storage built around constantly copying and moving data does not scale under these conditions, so fully distributed storage and data become the baseline.
AI pipelines also handle multimodal data that must coexist: text documents with their own volumes and I/O profiles, images, audio, video, and scientific data such as genomic research or quantitative trading datasets. Rob Benoit notes that the industry now calls this kind of environment an AI factory.
2. A Customer Pipeline From Raw Images to Training
David Boland walks through the pipeline from left to right, from raw data collection to inference and prompts. His customer built an application that identifies food: when a user scans an item, it works out what it is and responds with a prompt, such as whether or not it is a hot dog.
The pipeline begins with ingest into object storage. The customer scraped the internet for every picture of food it could find, collecting billions of images, and David Boland explains that object storage is the right fit for massive amounts of raw data because it scales into the exabytes and has strong data management and metadata properties.
Next comes curation and preparation. For a photo of a couple holding a hot dog, the people are cropped out and the image is aligned and adjusted to capture as much of the food as possible, so the training model only sees the part that matters. Each image is then turned into a tensor or an embedding, depending on the type of model, with embeddings now more common, and used for training.
3. Checkpointing and Storage Tiers Carry the Model Through the Pipeline
Training can run for days, weeks, or months. Rob Benoit uses ChatGPT 3.0 as an example, describing it as the first commercialized LLM and the point in 2022 when AI took off with consumers. It took 35 days to train across about 2,000 GPUs. If something went wrong on day 34, such as bad data, the only option was to start the 35-day run again.
Checkpointing creates reference points that allow a fast rollback, but every checkpoint creates more data, more I/O, and more stress on the network. In the large environment Rob Benoit describes, up to 1,000 models are being trained at any point in time, with checkpointing happening almost every second.
David Boland adds that each stage is connected via a network and has its own compute requirements, CPUs in some cases and GPUs in others. For processing and training, data moves from object storage to a more performant file or block system. The later stages each use storage differently:
- Checkpoints. Recent checkpoints sit on more performant storage, while older ones move to object storage so teams can go back and troubleshoot later.
- Validation. Models are validated before rollout, possibly with a human in the loop to confirm the model is identifying a hot dog and not something else.
- Inference. The model moves across the network to a different platform, where agents or apps query it, with KV cache and vector databases on more performant storage.
- Retention. Old models should be kept for compliance and regulatory reasons, such as investigations into training content or outputs, in what David Boland calls cheap and deep archive on object storage, alongside prompt outputs.
4. Every Stage of the Pipeline Generates More Data
Rob Benoit stresses that data is not transformed in place: each step creates additional data elements connected to the original source, which, with KV cache and vectors included, adds up to a significant amount of storage to move and manage.
David Boland agrees and adds that fine-tuning creates even more data. The customer also retrains from scratch roughly every 18 months: rather than fine-tuning the previous model, it copies all of the raw training data again to create a new model that is more efficient and less expensive to run. It keeps earlier training content for years. A customer since 2019, it has grown from two petabytes to six petabytes of data, plus its output.
5. Object Storage Costs Go Beyond Capacity
The customer now holds 30 billion images in object storage buckets, totaling almost six petabytes. As a startup without much money, it needed an inexpensive place to store that data, and object storage was the right fit.
David Boland cautions that not all object storage is created equal. People sometimes assume one S3 service is the same as another, but costs go beyond capacity. Hyperscalers may not charge for ingress, but they charge API PUT fees for each object written into a bucket. The price looks small on paper, but AI training involves billions of objects. Using AWS S3 as an example, writing one billion objects costs $5,000 in PUT charges. For the customer’s 30 billion objects, that adds up to $150,000 in PUT charges alone, not counting storage, an amount the customer has saved over the past few years.
Other charges add up as well:
- List and get operations. Charges apply every time a customer lists a bucket’s contents or retrieves an object.
- Data retrieval. Retrieval fees are a further cost.
- Data movement. Objects may sit on the East Coast while available GPUs are on the West Coast. Moving data between hyperscaler regions costs about $20 per terabyte, or $20,000 per petabyte, and with five petabytes moving for retraining every 18 months, that cost recurs.
David Boland says that with Megaport there are no egress charges, no data retrieval charges, and no API PUT, GET, or operations charges, so customers do not have to worry about hidden costs for moving data across the network.
6. Object Storage Has Grown From Archive to High Performance
Beyond scalability and cost, David Boland highlights object storage capabilities that matter at the beginning and end of the pipeline, and potentially for some checkpoint data:
- Customizable metadata. A picture can be given 768 different labels, such as whether a hot dog is red, on a bun, or topped with mustard or ketchup, and the customer uses 768 descriptions for each picture. That metadata takes up space, but it lets the team ask for every hot dog, pizza, or golden retriever and retrieve only what matches.
- Lifecycle management. Data that has not been touched for a long time can move to a different location or storage tier based on lifecycle policies.
- Data protection. Versioning, immutability, and object locking are built into the S3 interface.
Rob Benoit adds that object storage began as archive, the cheap and deep tier AWS created with the first S3, and has since evolved into a platform for high-performance use cases. Model training began on high-performance file storage and is now moving to high-performance object storage, and VAST Data supports both. The environment with almost an exabyte of storage described earlier is an object pool, and Rob Benoit says it generates about 12 terabytes per second of throughput.
7. Vectors, Structured Data, and Event Streams Expand the Data Estate
Rob Benoit then turns to the other data AI creates, starting with vectors and embeddings. One way to bring data into a model is to train the model on it, using tensors so the information is built in. Many organizations instead want to use the model as the engine and the brain while keeping their private data to themselves.
To make that efficient, the data is vectorized. A mathematical process places content into a grid and assigns each piece a value, enabling semantic search, also called vector search, so a search for a hot dog finds the content whose values sit closest together. Vector databases have existed for more than 20 years, but Rob Benoit says they are being reimagined for AI. At VAST Data, the vector database is a natural extension of the platform that works like metadata for any data brought into it.
Structured data gets less attention, but Rob Benoit sees a big use case for it. It is used to capture transactions and categorize metadata, and VAST Data uses a structured element to store vector embedding values. Structured data can also hold the policies and governance elements that shape how models answer questions, as well as outputs kept for auditing.
Where traditional machine learning and analytics were batch-based, inference models that take in real-time events depend on event streaming through tools such as Kafka, a structured environment VAST Data and others have deployed to change how these platforms ingest data.
8. Inference Moves KV Cache Out of GPU Memory
Most GPUs today are doing model training, Rob Benoit notes, largely for foundation model builders such as Anthropic, but the world is shifting to inference. Agents, chatbots, and application-level AI are all inference, and agents, which he calls the next generation of applications, need state.
That is where KV cache comes in:
- Prefill. A prompt arrives and the GPU does the work of creating tokens, where a token is a word or part of a word. Commercial models charge by token usage, which has created a whole field of tokenomics.
- KV storage. The tokens generated from the input are organized into KVs, which are stored in GPU memory.
- Decode. The model generates more tokens to create the response, which goes back to the agent or chat interface that made the request.
KV cache is the model’s attention state. For a short session, GPU memory is probably fine, but GPU memory is very expensive. For a chatbot used over a couple of days or an agent running for a month, KV cache is its memory. Rob Benoit compares an agent without KV cache to a person who works on something for hours, goes to bed, and has to relearn everything the next morning.
Because newer applications and agents require more and more KV cache, it is moving out of GPU memory and down the tiers into shared storage that still delivers high performance and can hold longer-term state. Losing state means recalculating all of the tokens, leaving expensive GPUs doing rework. The trade-off is that the network needs more throughput, and storage needs more space and a distributed design that can serve cache state to thousands of GPUs or TPUs during decode. Rob Benoit describes persistent shared cache as an important part of how inference evolves.
9. The Data Plane Has to Span Sites
Rob Benoit frames the data plane as a lifecycle: data is ingested, prepared through cleaning, labeling, and metadata association, used for training with checkpoints and validation samples, retrieved as vectors, metadata, or answers from LLMs, and ultimately served for inference. That lifecycle requires three capabilities:
- A large namespace. One large data space that brings all of the data sources together.
- Real-time metadata. The ability to query data and understand what it is, what state it is in, and what context it belongs to.
- Parallel access. Many applications on different infrastructure components, across thousands of GPUs and CPUs, need access to the same datasets across multiple environments.
Most large environments are also distributed, Rob Benoit points out, because there is not enough GPU density in one site for the work companies want to do. Data has to move between sites across the network, using techniques that reference it, move it, or cache it. That connectivity turns separate sites into one virtual, distributed systems architecture. The speakers add that GPUs will not necessarily sit next to the storage, so data needs to go wherever GPU capacity is available.
10. RAG and Edge-to-Core Pipelines in Practice
Rob Benoit uses retrieval-augmented generation (RAG) as an example of connecting a model to data without putting that data inside the model, such as a network operations engineer asking how to replace a failed leaf switch. Source material such as runbooks, network diagrams (which are images), incident reports, and knowledge base articles feeds what VAST Data calls document RAG or document intelligence. An extraction framework, usually an LLM, extracts and embeds the passages to create vectors and a searchable index. At query time, the question is embedded, results are ranked and retrieved through similarity search, and the response is generated along with its sources, the way tools such as Gemini show where an answer came from.
His second example is an edge-to-core AI factory, which shows inference at scale in a distributed environment where the network is central. It is a sanitized autonomous driving example, but it applies to physical AI generally, where a vehicle or robot is the endpoint. Data from sensors and cameras moves through authentication frameworks using MQTT, a standard way to move sensor data, and is written into an object store. It is then enriched and decoded in a pipeline before being handed to a GPU server for inference. Models are retrained, run through simulations, and returned to the vehicle.
Rob Benoit contrasts a LIDAR-based, rules-driven sensor approach, such as Waymo, with a camera-based approach where the model does the driving, such as Tesla. One is very expensive and controlled, while the other is already available and relatively cheap, which matters for adoption. GPUs are everywhere in these models, including in the car itself or, for the humanoid robot builders VAST Data works with, in the robot’s backpack. He sees this as a smaller, denser version of the same AI infrastructure, tied together by network connectivity.
11. The Next Cloud Is Built Around Data Movement
Rob Benoit closes by arguing that the next cloud is built around data movement. Data is the fuel for AI: it rides across the network, lives in storage, and is processed in GPUs. That makes the network a foundational part of the data architecture, and engineers at VAST Data spend a lot of time designing it, because a wrong design means the systems do not operate as intended.
The architecture has to support every multimodal data type, since no single type can be dictated, along with the vectors, structured content, and inference state such as KV cache that AI adds. Shared access to data keeps very expensive GPUs, DPUs, and TPUs productive. The worst outcome is making large investments and then finding a bottleneck in storage or the network. The principle is to build, scale, and design everything as one system, in which compute, networking, and storage, with data flowing through that stack, form the computer. Rob Benoit refers to Wasabi as a partner in keeping that pipeline moving through the whole lifecycle of data.
The session closes on the point that the AI cloud comes down to three components, network, compute, and storage, and that very few providers offer all three. Megaport is named as one that does and is credited with leading the industry on vision, because pipes, compute, and storage cannot be obtained from just anybody. In the next generation of cloud service providers, AI infrastructure will be delivered by those who have all three.
Key Takeaways
- AI infrastructure is one distributed computer. Compute, networking, and storage need to scale together, because a bottleneck anywhere leaves expensive GPUs idle.
- Copy-and-move storage does not scale for AI. Jobs move across GPUs, pools, and sites, and the same corpus serves many workloads at once, so fully distributed storage becomes the baseline.
- Each pipeline stage needs the right storage. Object storage suits raw data, older checkpoints, archived models, and prompt outputs, while training, recent checkpoints, KV cache, and vector databases need more performant storage.
- Checkpointing makes long training runs recoverable. It avoids restarting multi-week training from scratch, but every checkpoint adds data, I/O, and network load.
- AI data only grows. Each pipeline step creates new data linked to the source, and periodic retraining from scratch means raw data is kept for years.
- Object storage costs go well beyond capacity. PUT, list, get, retrieval, and inter-region transfer charges add up, with PUT charges alone reaching $150,000 for the customer’s 30 billion objects.
- Inference shifts the pressure to persistent state. KV cache is moving out of GPU memory into high-performance shared storage, raising network throughput and storage capacity requirements.
- The data plane must span sites. A large namespace, real-time metadata, and parallel access matter because GPU capacity is distributed and data has to go where GPUs are available.
Conclusion
The session argues that AI infrastructure cannot be designed as separate compute, storage, and network projects. GPUs are only productive when data reaches them from storage that can keep up, over a network that can move it. A single hot dog image passes through ingest, preparation, training, checkpointing, validation, inference, and archive, and each step creates new data with its own storage needs.
Object storage anchors the beginning and end of that pipeline and increasingly supports high-performance use cases, but its economics depend on more than capacity once per-object operations, retrieval, and data movement are counted across billions of objects. As inference grows, vectors, structured data, and KV cache add further demands on shared, high-performance storage and the network that connects it.
For practitioners, the lesson is to design for data movement from the start. Large AI environments are distributed across sites, GPU capacity is not always next to the data, and agents need persistent state to avoid expensive recomputation. Treating compute, networking, and storage as one distributed computer is what keeps expensive accelerators working instead of waiting.
The Next Cloud Is AI Infrastructure: GPUs, Storage, and High-Performance Networking – Full Transcript
Host: We can’t have an infrastructure conversation in 2026 without talking about AI. And if you’re building infrastructure for AI, you know that buying a bunch of GPUs is only one part of the equation. You need storage that can keep up, networks that move massive amounts of data, and all of these things need to work together. Otherwise, you’ve got some very expensive GPUs that are just sitting around waiting. So our next session is going to look at what it takes to build infrastructure that can actually support those workloads. And joining us are Rob Benoit, VP of Worldwide Solutions Engineering at VAST Data, and David Boland, VP of Cloud Strategy at Wasabi.
Rob brings more than 28 years of experience in enterprise technology and engineering, while David focuses on cloud strategy and innovation across compute, storage, networking, security, and AI. So together, they’re going to explore how GPUs, high-performance storage, and networking are changing the way infrastructure is designed. So please welcome Rob and David.
David Boland: Thank you. All right. Good afternoon. Thank you, everybody.
Robert Benoit: Good afternoon.
David Boland: I’m David.
Robert Benoit: Rob.
David Boland: So, quick overview of what we’re going to talk about. We’ve got about, what, 30, 45 minutes, somewhere in there.
It’s exciting to be here. Now, when I first learned we were going to be here and I was going to be on stage with Rob, who I’ve been following for years, because unbeknownst to him, he and I worked together years ago at NetApp, and I was always impressed with the work that he did. Former colleagues. And so I was really excited to be here. And when I talked to Alexis, I said, yeah, absolutely, I want to be up on stage. Didn’t even think twice about it. I get to talk about AI.
And I thought to myself, this is very cool. We have all kinds of fun things to talk about. AI is in the news. We’ve got tokens and AI agents and KV cache and all kinds of buzzwords, and I figured I would just go down the list of buzzwords and make it super exciting and drop the mic, and the two of us would walk off stage.
And I would do that, and then you guys would be like, whoa. Yay. Exciting. It’s fantastic. And then I realized, oh, wait a minute. It’s storage.
Robert Benoit: And network engineers.
David Boland: Network engineers, and it’s a half hour after lunch, so the digestion system is kicking in. So folks are probably nodding off and might not really fully understand what’s going on. So in my mind, you’d be doing this as we wrapped up. So hopefully, that won’t be the case.
But then I said, okay. Again, it’s storage. It’s not really GPUs. Although we talk to GPUs. We can send data to GPUs. It’s not as exciting as the network and the compute side of the world. And so I said, well, all right. How are we going to make this a little bit entertaining? Keep you guys involved in the conversation and maybe just keep you a little more excited about storage. And then I remembered back to my days in business school, and the professor said, well, if you’re going to give a presentation, you should tell a story.
And the story, of course, has a protagonist, and there’s a problem to overcome. They’re challenged. They figure it out. They’re successful and they win. And I said, great. I will tell a story. But then I said, it’s storage. And so really finding a good story about storage is difficult.
Robert Benoit: But everybody loves a good story.
David Boland: Everybody loves a good story. So I said, all right, maybe not a story so much as I’ll use a customer example. We have a real-life customer example of how they use storage, both object storage and high-performance file and block storage, in order to get their AI product to market. And so I said, yeah, that’s what I’ll do. I’ll try to incorporate some examples from this customer into the story so that you can see exactly where the pipeline we’ll talk about goes.
Now, the company that I’m talking about is a true-life customer. Jian-Yang is not really the user in the company, and his company is not SeeFood. The names and the likenesses of people have been changed to protect the innocent. But basically, it’s a true story. And I’ll go deeper into exactly what they’re doing in a moment.
So just to set it up, then I’m going to turn it over to Rob for a couple of slides, and I’ll go back to the SeeFood story. In this case, we have a customer who has built an application that identifies objects. And for purposes of the presentation, in this case, the objects are food. So as you scan a food, not knowing what it is, it goes through the AI application and figures out exactly the type of food it is or isn’t and gives you a prompt. Yes, this is a hot dog, or no, this is not a hot dog. And there’s a lot of steps involved in figuring out everything from the beginning of that scan to the outcome of the prompt. And hopefully, if I do a good job, and I know he will, we’ll explain to you exactly the hot dog’s journey through the AI pipeline.
Robert Benoit: Excellent. Thanks, David. I’m excited to see how that story plays out. Before we get there, I just want to talk about the infrastructure changes that are going on to support these large-scale AI initiatives.
And I know there’s a lot of network folks in the room, but think past connectivity. When we think about the AI infrastructure frameworks, think of all these components working together as one distributed system. Compute, networking, storage, and data is the layer that runs across these, all working together. Think of it as one computer, a fully distributed computer, but it all works together. And in order for this computer to work, all those pieces need to work and scale at the same pace and in the same way, and support all the workloads that you’re going to put into this system.
So I work at VAST Data. And to give you an example of this, one of the largest AI-based infrastructure computers that we work with, just to give you a sense of the scale at one of our customers, is about 300 GPUs connected over hundreds of switches, thousands of connections, connected to about an exabyte of capacity. That’s one distributed computer. One of the largest ones in the world, but that is what we mean by a computer.
So when we think about it and we translate this down to storage, the challenges that we have in traditional storage are, number one, if you think about the way that data and workloads flow across that large system, and really any system you’re designing, jobs move across these GPUs, through pools and even sites. We talk a lot about site-to-site connectivity, but in a distributed environment, that has to be seamless.
Because the goal is to keep those GPUs busy. Those accelerators, whether it’s GPUs or now TPUs, DPUs, or even IPUs, which is a new one when you start thinking about disaggregated inference and some of the other workloads that are evolving very quickly. You need to have access not to the specific data for a job, but you need to have access for all the jobs.
So what you’ll find is the corpus of data that you put in these systems needs to be accessed by many different workloads all at the same time for different use cases and simulations. And the last one is the lifecycle. When we talk about data, there’s a lifecycle to data. It starts one way and then it moves as these processes evolve, whether it’s through model training or eventually into inference, and it’s a constant cycle.
It’s a flywheel that data lives on as you go through these systems. So traditional storage environments, where you’re constantly copying and moving data around, just don’t scale. It just doesn’t work.
You have to reinvent the way that this works. And this is where completely distributed storage environments and data become almost a baseline of how you evolve these systems. So we talked a little about data. And when David talks about the pipeline, which is the way that data is going to flow through these systems to find that picture of the hot dog, there’s multimodal data types that we work with.
It’s text documents, which have certain volumes and I/O profiles that you have to work with. We have image data, audio data, videos, and then scientific data. That’s things like genomic research, or quant datasets for analysis of financial patterns and trading profiling. All this data in these environments needs to coexist, and it needs to be able to be integrated into these different sets of pipelines.
So this is what we look at as an AI infrastructure, or some people call it an AI factory. That’s the industry term now as it continues to evolve. So just setting the context: different data types, multimodal data types, scalable infrastructure, all designed around this idea of a systems architecture that’s designed across a high distribution at scale. So let’s get back to the journey of that hot dog.
David Boland: Hot dog journey. Excellent. Thanks, Rob. So this is, let’s say, an illustration of the AI pipeline, and I’ll walk you through it from left to right. The right being where we want to get with the inferencing and the prompts. The left is the collection of the raw data. And in this case, using our friend Jian-Yang and SeeFood’s example, what he’s done is he’s scraped the internet for all the pictures of food that he could find. And he’s got billions of them.
I’ll talk more about that in a moment. But in the process of scraping that data, he has to put it someplace. And of course, he’s going to put it in storage. And in this case, object storage is the right fit for just massive amounts of raw data. Object storage tends to scale into the exabytes.
It has fantastic properties as far as data management and metadata goes. So in this case, all of the video and all of the images of food go into the ingest piece of the pipeline, the very first stop. Once it’s there and they’re ready to start curating that data for training purposes, they’ll take the picture of the lovely couple with the hot dog. They’ll crop the people out.
You want to align and adjust everything so that you get as much of the food as possible. So there’s some preparation involved here. The data is transformed so that you only really expose to the training model the piece that you want it to see.
Then the image, in this case, is either turned into a tensor or into an embedding, depending on which kind of model you’re going to use. Tensors or embeddings. And most likely these days, it’s more embeddings than tensors. But those tensors are then used in the model for training purposes. And as the training is going on with all of the images, what will happen is you could be running for days, weeks, months.
Robert Benoit: And ChatGPT was a good example. ChatGPT 3.0 was the first, what I would call, commercialized version of an LLM. I would say that’s the acceleration of when AI really took off, when it came down to a consumerization of that platform in 2022. Anyone know how long ChatGPT took to train, the first model of 3.0? It took 35 days.
So 35 days to run that model, done across about 2,000 GPUs. Let’s say on day 34, that model broke. Something happened on the training. The training wasn’t right. There was bad data or whatever it was, and you wanted to roll back.
Well, the only option you had was to roll that back and start all over again with a 35-day run. This is where checkpointing came in, which allows you to create these reference points where you can fast roll back. Every checkpoint creates more data, more I/O, more stress on the network to be able to do that.
Now, they’ve optimized over time since ChatGPT 3.0. But checkpointing is a key process in how you do that. That large environment I talked about, they run anywhere from 5 to 1,000 models being trained at any point in time, and they’re checkpointing almost every second when you think about that.
David Boland: Yeah. And so checkpointing is critically important, so you don’t have to roll back and start from scratch. And as we move from left to right through this architecture, each one of these boxes is connected via a network.
And each one of these boxes has a different compute requirement. It could be CPUs in some cases, GPUs in others. But clearly, as you move from ingest data on the object storage side, you’re going to want something that’s a little bit more performant on the processing and the model training. So it will go from object storage to a file system or a block system.
And then as that’s running, checkpointing is critically important, so you don’t have to start from scratch every time. And then you most likely want to keep some of the older checkpoints so you can go back and troubleshoot things in the future if you need to. So checkpointing, and storing those checkpoints either on a more performant solution for the most recent checkpoints, but then object storage again for the older ones, is critically important. Then you go through the process of validating the models, make sure it’s good, maybe throw a human in the loop in there to make sure what you’re looking at is a hot dog and not something else.
And then when it’s ready to roll out, you’re ready for your inferencing solution. So then you move it across the network to a different platform where inferencing can take place. And then you have agents or apps that query the content and determine what it is that you’re looking at.
And I know that Rob will talk more about KV cache and vector databases in a moment, but that’s where these will come in. Your KV cache and your vector database on the more performant storage solution. Then ultimately, at the very end of the process, what will happen is there’ll be new models. What do you do with the old model?
Well, you should put that in storage someplace for compliance and regulatory reasons. If there is ever an issue where something needs to be investigated, maybe there was some content that shouldn’t have been used on the training model, maybe the output was wrong, you need to have an investigation. Or the powers that be say, give me your compliance information on these things. You want to be able to find where those older models are.
And that should most likely be stored in an archive someplace. Cheap and deep. Object storage is a great fit for it. And then the outputs of the prompts. Is this a golden retriever or is it a piece of pizza? All those things could also be stored for the long run in object storage.
Now, this next picture is about the same. Not as pretty, because it doesn’t have all the pictures of the food and the dogs and all that. But again, this is another example of the pipeline. Just a little more block-and-file-ish, so for the engineers out there, this is more pleasing to the engineer’s eyes.
This is basically what the pipeline looks like from left to right. Object storage being at the very beginning, where you can have hundreds of petabytes or exabytes, as Rob said, of data. Moving that to processing for training, CPUs and GPUs, higher-performance storage, the training models, the GPUs again. And then the inferencing, and compliance and regulatory issues.
Robert Benoit: Yeah. One key thing on this is each step along the way generates more data. It’s not that we transform it in place. It just creates additional elements of data that are connected to the original source data. So if you look at the data explosion of this, and I’ll go through some specifics around things like KVs and vectors, this is a significant amount of storage that needs to be moved and managed across this ecosystem.
David Boland: Exactly right. And as you’re fine-tuning the models, you’re adding even more data. So your data will grow from beginning to end as it goes across the pipeline. There’s no two ways around it.
What often will happen is the data that the models have been trained on, the raw data I showed earlier with the pictures of the dogs and the foods and the people, well, every 18 months or so, Jian-Yang and the folks at SeeFood want to run a new, more efficient model. So they will take all that data again, copy it, and move it through training. Instead of fine-tuning the last model, they’ll create a new one from scratch that’s more efficient and less expensive for them to run.
So they do not get rid of the content that they were training on before. They sit on that for years. In our case, with this customer, they’ve been a customer since 2019, years and years, as they’re collecting data. So they’ve grown from 2 petabytes to 6 petabytes of data today. And so it continues to grow, plus all the output.
Now, one more slide and then I’ll turn it back over to Rob. Object storage, and the importance of object storage here, especially for the beginning and the end of the pipeline. So, as I mentioned, Jian-Yang and team have 30 billion images in object storage buckets. Those 30 billion images comprise about 5 petabytes, almost 6 petabytes of data.
They came to us as a startup without a lot of money in their pockets. They really were looking for an inexpensive solution to store their data. Object storage is the right fit.
But not all object storage is created equal. Sometimes people say S3 is S3, object storage is object storage. But how it’s delivered to the end user is critically important.
There are costs associated with storage that go beyond just the capacity. Especially in the hyperscalers, if you’re putting data into their cloud, it may not have an ingress charge, but they’ll have an API PUT cost associated with each object that’s going into their storage buckets. And on paper, they’ll tell you, well, $0.04 per thousand objects is not a lot. But now we’re talking billions of objects for training models.
And so putting a billion objects into, let’s take AWS S3 as an example. A billion objects, that’s $5,000 in PUT charges. Not an ingest charge, but a PUT charge. And in the case of SeeFood, looking at 30 billion objects, as they grew from 2020 to today, that’s $150,000 of PUT charges alone.
Not storage charges, PUT charges. So they’ve been able to save $150,000 over the past few years on those charges alone.
Additionally, there’s charges every time you want to list what’s in your bucket. Every time you want to get something from your bucket. Every time you want to move an object across their network, there’s a transport cost associated with it. Objects may be living in a bucket on the East Coast, but you may be looking for GPUs on the West Coast.
Moving those objects from East Coast to West Coast is not free in the case of the hyperscalers. They’re going to charge you network charges of about $20 a terabyte to move that data from one of their regions to another region just to have it go through the compute piece. And in this case, that’s $20,000 per petabyte. Every 18 months, moving 5 petabytes, that’s another $20,000 times whatever it is.
So it can be expensive if you’re not using the right object storage solution from the right vendor. The beautiful thing about Megaport is that you don’t have the egress charges. You don’t have the data retrieval charges, which is another charge I didn’t mention. You don’t have API PUTs, GETs, or operations charges.
You don’t have to worry about all the hidden costs associated with moving your data across the network. It comes as part of the overall solution. So object storage at the beginning and the end, and maybe for some of the checkpoint stuff, is critically important because of the scalability, because of the cost. There’s great things you can do with customizable metadata.
If you want to take a picture of a hot dog and give it 768 different labels to say, this is a red hot dog, this is on a bun, this has got mustard, it’s got ketchup, you have a choice to put 768. In the case of these guys at SeeFood, they do use 768 descriptions for each picture. So that metadata fills up a lot of space.
That metadata tells them exactly what they’re looking for. And if they go into their data and they want to say, give me everything that’s a hot dog, give me everything that’s a pizza, or give me everything that’s a golden retriever, it’ll scan the metadata and then retrieve only what you ask for. So it’s easier to use from a customizable metadata perspective. It helps with lifecycle management too.
If something hasn’t been touched in a long time, the customer says, I haven’t used this file, I’m not looking at it for a while. After it sits there, it could be moved to a different location or different storage based on lifecycle management properties. And then data protection always has the versioning. And in the case of object storage, you’ve got immutability and object locking capabilities that are all built into the S3 interface.
Robert Benoit: Yep. No, it’s great. I’ve been doing this for a long time, and one of the things that we’ve been tracking, not just at the company I’m at now but at my former company, is the use cases for object storage. It really started as an archive.
AWS created the first S3 element. It was all their cheap and deep storage. It’s really evolved. We use objects for high-performance use cases. Model training started on high-performance file-based storage, and it’s now high-performance object-based storage. We can do both.
And this is really the evolution of where we’re seeing object storage go, based on a lot of things that David talked about, a lot of scalability elements.
That customer I talked about that’s got almost an exabyte storage pool, that’s an object pool. So we use object storage even for that. And you can imagine the I/O. I think we generate 12 terabytes per second through that system. Massive.
So we talked a little bit about the pipeline itself. There are other data elements, when we walk through that pipeline slide, that are relevant when we think about the AI infrastructure use cases, or these AI factories. And they’re not necessarily just the data you put in. It’s all of the other data elements that are created. One of the key things is vector data, or embeddings.
So there’s a couple of ways that you can put data into a model. Number one, we can train a model with all the data. And that’s where you do things like tensors, and you train the model around that data, and it’s got the information built into itself.
But a lot of people don’t want to put their data inside of a model. They want to use the model. They want to use it as the engine and the brain, but they want to provide their own data and still keep their data to themselves, their private data. So to make that efficient in how to use that data, we vectorize it.
So it goes through a process where it’s basically a linear mathematical equation that puts it into a grid that’ll take the different content you put in there and give it a mathematical value. And then you can do a protocol called semantic search against it. So you can basically do a vector search.
So when you look for that hot dog, it’ll find all the values of that content that are closest together, that are semantically next to each other. So vector databases have been around for 20-plus years. But in the world of AI, when you’re thinking about all this content and what you’re doing, vector databases are now being reimagined.
So at VAST, what we do is the vector database is just a part of the architecture. We make it a natural extension to what you do. Think of it as metadata for any of the data that you bring into the platform. The other side is, we don’t talk a lot about structured data. Everything we talk about in model training and inference is usually against unstructured data, but there is a big use case for structured data. Things like capturing transactions as they go on, categorizing a lot of the metadata.
We use a structured element to store our vector embedding values. It could also be a point where you keep your policies or the governance elements of how those models are going to answer questions. You can store your outputs for some of the auditing things that you do. So all those things can be done.
The other thing on the structured side is that most work done in what was machine learning or traditional analytics was always batch-based. You put a big dataset together, you run it through a batch process, and off it came. When you think about bringing events into these inference models, where they’re starting to take this real-time data, things like event streaming become really important, especially when you start looking at some of the use cases. So real-time event streaming through things like Kafka is really important.
And that’s more of a structured environment that we’ve deployed, and others have deployed, to really start transitioning how you look at data ingestion in these platforms.
So I already touched on this earlier, and I’ll just put a little bit more context to this from the data side. Most of the world that’s using GPUs today is doing model training. At least when you think about the millions of GPUs that are out there, the majority of them are used by these big foundational model builders, the Anthropics of the world, the big, massive model builders. And they are consuming as many GPUs as they can wrap their arms around.
The world is shifting to inference at some point. So when you think about agents and chatbots, or you start bringing more application-level AI capabilities into it, that’s all inference. They’re going to use models, use the data that comes in through those models, to provide the outcomes and the outputs. The agents are the next-generation applications. So when you start thinking about agents and how they operate, you have to create state for them.
So we talk about KV cache. You have a prompt that comes in. That’s a question that’s asked of the model. It goes through a prefill stage, and to give you the two-second version of it, prefill is basically the GPU doing the work to create tokens. And a token is either a word or a portion of a word. We all talk about tokenomics.
And if you ever use any of the commercial models out there, you’re going to get charged on token usage. There’s a whole tokenomics of how you optimize that. And I’m sure everybody’s run out of their monthly tokens on their Claude or ChatGPT bills.
But that’s done in the prefill stage. So you generate tokens from the inputs. And then the way that’s organized creates KVs. And those KVs are then stored. So prefill happens in compute, and then the results of that are stored in KVs, and that’s in GPU memory. And then when you decode that, which is actually creating the response, you generate more tokens.
And then that response is done, and that’s handed off to whoever prompted, whether it was an agent or through a chat interface.
KV cache, think of it as the attention state. So if you’re doing a session and you’re only on there for a couple of minutes, GPU memory is probably fine to do it. But GPU memory, remember, as expensive as the GPU is, is very, very expensive.
So as you start getting larger context state, let’s say a chatbot that you’re working with for a couple of days, or an agent that may be running for a month, KV cache is its attention state. It’s its memory. So think about it: you worked on something for a few hours, you went to bed at night, and then when you woke up, you had to go and relearn everything you did yesterday. That’s what happens with an agent without a KV cache.
And based on the way agents are being deployed and the way these new applications are being developed, they require more and more KV cache. So KV cache is moving out of GPU memory. That’s the fastest spot it can be. It’s moving down tiers.
So eventually it’s going to get into a shared storage environment that still allows high-performance capabilities, to keep longer-term state without having to recalculate. Because the reality is, if you lose state, you have to go back and recalculate all those tokens again. And what that costs is those really expensive GPUs doing a whole bunch of rework.
And it’s not optimized to do that. It’s just expensive to do it. So there’s a much better way to do it.
And the cost, though, on the network is you need more throughput. The cost in storage is you need more space. You need a distributed environment like we talked about earlier in order to provide this cache state to thousands of GPUs, or even TPUs, or some of the other ones that will pull from that state as they look in the decode phase.
So persistent shared cache is really an important piece as we look at the evolution of inference.
So we talk a lot about inference. If you look at it from a data plane perspective, and we touched on this a little bit earlier, this is a lifecycle. So it’s about, you bring the data in. David talked a lot about data ingestion of raw data, and we talked about data types. You prep the data. So you clean it, you label it, you create metadata associations to it. You may vectorize it, depending on what it is.
And then you train a model. You have checkpoints, which create more data. You have samples of that data, which you can validate, to see where you’re at, and then retrain or fine-tune.
And then you have the retrieval side of that, which is now pulling either the vectors, pulling additional metadata, or just getting the direct answers back from the LLMs. And at the end, this is all done for the inference side of it. It’s all based on the amount of models you have and the state and memory of those models.
So some of the things we look at from a data plane perspective: you really need a large namespace, one large data space that allows you to bring all these data sources together. You need real-time metadata. The ability to go query that data and understand what it is and what state it’s in and what context it’s in is really important. And then across thousands of GPUs and CPUs, because CPUs do play a role in all this, you need parallel access. You need to have many different applications, running in different infrastructure components, have access to the same datasets across multiple environments.
Now, we’ve talked a lot about this all being in one large data center. The reality is most of these large environments are being distributed amongst environments. There’s just not enough GPU density in one site to do a lot of the work that companies are looking to do. So they’re scaling it out and distributing it.
So you have to move data between sites, across the network. You can reference it and then move it, cache it. There’s many different intelligent techniques on how to leverage that connectivity. So we’re not going to talk connectivity, but it is about connectivity, to be able to connect those different sites together and create that one virtual system, the distributed systems architecture, which represents that computer.
David Boland: Right. It’s about availability of GPUs, when you can get them, and where they are.
Robert Benoit: That’s correct.
David Boland: They may not necessarily be next to your storage. Your data needs to go to where the GPU is available.
Robert Benoit: 100%. So here’s a quick example. We talked about retrieval-augmented generation. This is where you’re not putting data in the model. You want to be able to connect to that data. That’s what it means. You’re basically retrieving the data from an external site so you can generate it on demand, as you need it or as you find those different datasets. This is just an example.
Let’s say this is done for a network operations person. Say, hey, how do I replace this failed leaf switch? Well, you may have a whole set of source documents. And we call this in my company document RAG, or document intelligence.
You put a whole bunch of documents together, source documents. Runbooks. It could be your diagrams, which are pictures, by the way. So we’re talking images and text at this point. It could be your incident reports. It could be your knowledge base articles. Whatever it is, you send that through an extraction framework. It’s usually an LLM that’ll do the extraction and embedding for you. You embed those passages, which creates your vectors.
And then what you get is a searchable index that an LLM can use to find the right content it needs based on the prompt that it’s responding to. So on the retrieval side, you have the embedded question, you rank it and retrieve it, so it works off of the similarity search. You augment and then generate the response to that question, and then you actually get the source with all of the context of the source.
So if you’ve ever used Gemini or a lot of these tools, it gives you the answer, but it also gives you where it got the information from. And that’s all part of a standard. These are all standard, what we call, RAG pipelines.
Another example, because I think it illustrates inference at scale, but more in a distributed environment where the network is a key piece to this, is edge to core, what I call edge-core AI factories.
I think Roy mentioned it this morning, but this is actually a sanitized version of autonomous driving. This could be really any physical AI. It works the same. I mean, it’s all physical AI at the end of the day. An autonomous vehicle is just the robot itself, or whatever. That’s the endpoint for physical AI.
But if you think about it, you have sensors, cameras, whatever that endpoint is. They go through authentication frameworks, using MQTT, which is a standard way to move data from those sensors out to the different elements you want, and that writes it into an object store, where now you get telemetry of it, you get the type. You can then put it into a pipeline where it gets enriched, it gets decoded. And then that hands off to a GPU server where it gets inferred, or in the sense of autonomous driving, depending on which type you’re doing.
There’s really two types. There’s a LIDAR environment, which is really a rules-based sensor environment, think Waymo, versus the camera-based LLM, where the model is doing the driving, think Tesla. So the two different models. One’s very expensive and controlled. The other one is out there and ready to go and pretty cheap, to think about where the adoption is.
But this is a pipeline they’ll do. And then they’ll retrain those models over and over again, put them through simulations, and then they’re back in the car to go drive again. But this is an example of how you connect, at high scale, these large inference models, how they’re going to start taking edge content, ingest it, put it in object environments, do all the work based on what you’re looking to do, and then, of course, feed the GPUs to either retrain the model or actually drive the car, depending on what the scenario is.
GPUs are all over the place. And in a lot of these models here, you actually have GPUs that are in the car or in the backpack of the robot. We work with a couple of companies that are building humanoid robots, and they basically have GPUs in their backpack. So it’s an evolving space, it’s a really interesting space. But it really ties, I think, nicely all the network connectivity elements together, wrapping them into a much smaller, denser version of what we talk about from an AI infrastructure perspective.
So bringing this all home, if we think about the next cloud built around data movement, it really is. Data is the fuel for AI. You have all the infrastructure components, but data is what rides across the network, lives in the storage, and then gets processed in the GPUs.
The network becomes part of the data architecture. It’s a foundational piece. My engineers in my company, we spend a lot of time designing the network, a lot of time. If we do it wrong, systems don’t operate the way we want them to. So a lot of time building out that network. Data includes all multimodal data types. I mean, David walked through it really well, which is, this could be videos, it could be files, it could be all different types of objects.
You can’t dictate what one is going to be. You have to support them all. So multimodal data is a key thing you have to look at.
And of course, AI adds all these new types of data sources into it: vectors, structured content, and of course all the inference state elements you need, KVs and other things, to keep the application side of this running. And then shared access to data keeps really expensive GPUs and DPUs and TPUs productive.
So the worst thing you want to do is make all these investments and find that I’ve got a bottleneck somewhere, whether it’s storage, whether it’s network, that doesn’t allow these systems to operate the way they’re supposed to. So it’s building it as one system, and building, scaling, and designing all based on this idea that compute, networking, and storage, with the data flow between that stack, is the computer.
And that’s one thing that we certainly have in our partners at Wasabi. We look at it to say, how do we continue to drive that pipeline through the whole lifecycle of data that lives in these architectures?
David Boland: The AI cloud really is three components. It’s your network, it’s your compute, and it’s storage. And there are not a lot of providers out there that have all three of them.
Robert Benoit: Yeah. I mean, very, very few.
David Boland: Megaport, of course, does. Megaport, I think, is leading the industry on vision, because you can’t get pipes and you can’t get compute and you can’t get storage from just anybody. And so as we move into this next generation of cloud service providers, the AI infrastructure really is going to be delivered by folks who have all three of them.
Robert Benoit: Absolutely. So thank you.
Need help mapping your cloud infrastructure path?
With a Megaport Solutions Architect
Find out how to connect your business in 60 seconds.
We’ve helped hundreds of customers, just like you, to simplify their connectivity.