Cloud Networking for AI Architectures – Key Takeaways
Overview
Riggs Goodman, Principal Partner Solutions Architect focused on AI security and AI networking at AWS, presents a technical view of how AI changes network architecture. The session explains why training and inference workloads behave differently from traditional applications, how data volume and latency affect design, and why the network stack changes when GPUs communicate at high scale.
The central theme is that AI workloads place new pressure on cloud and hybrid networking. Traditional enterprise workloads often rely on TCP-based request-response flows and data center architectures that can serve many workload types in similar ways. AI, especially model training, introduces massive data movement, GPU-to-GPU communication, loss-sensitive transfer patterns, and very low tolerance for added latency.
The discussion is presented primarily through an AWS lens, covering Amazon VPC, AWS Direct Connect, AWS PrivateLink, high-performance storage, GPU infrastructure, Elastic Fabric Adapter, Scalable Reliable Datagram, and AWS UltraCluster. Megaport appears as a Direct Connect partner for private connectivity into AWS, particularly when large datasets must move from customer or colocation environments into cloud-based AI infrastructure.
1. Generative AI Changed the Scale of Infrastructure Demand
Artificial intelligence as a term dates back to the 1950s, but broad public interest accelerated around 2022 as generative AI systems became easy to use through natural-language interfaces. That shift matters for networking because the underlying models depend on large-scale training data, large model architectures, and infrastructure that can support both training and inference at high throughput.
Machine learning uses data to make predictions and classifications. A model might estimate a house price from square footage, bedroom count, bathroom count, and location data, or classify an image as a dog, a pig, or another object. Generative AI extends that pattern into natural language and unstructured data, using large models trained on massive datasets to accept prompts and produce responses.
Under the covers, these systems rely on advanced algorithms and high-scale matrix multiplication. Transformer architectures are one example. As models grow, so do memory, data, and network requirements. The session contrasts an early 1957 model with a single parameter or weight against GPT-4, described as having 1.8 trillion parameters. A large model may need to span eight GPUs, 16 GPUs, or more.
2. Training Makes the Network Part of the Compute System
Model training repeatedly feeds batches of data into a model, compares the model’s output with the expected output, calculates the difference, and uses that difference to update the model. The transcript refers to this difference as batch loss or compute loss. Over many iterations, the model is updated until it reaches an acceptable level of accuracy or output quality.
For a small model on a single GPU, the network role is comparatively simple: data reaches the GPU, training runs, and the next batch follows. At AI training scale, models and training data are distributed. Multiple GPUs must process training work and then exchange updates so that the model remains consistent across all participating GPUs.
That GPU-to-GPU exchange is known as collective communication. After a training batch is processed and loss is calculated, GPUs share updates so each copy or portion of the model has a common view. The system cannot allow different GPUs to proceed with different versions of the model. The network therefore becomes part of the training system itself, not just a transport path to an application.
3. AI Networking Requirements Center on Data, Loss, Latency, and Stack Design
AI changes network architecture for four main reasons:
- Massive data volumes: Training data can be extremely large, and AI infrastructure must move that data among storage systems, servers, and GPUs.
- Loss-sensitive communication: Traditional workloads commonly rely on TCP for reliability. AI training often uses communication patterns designed to reduce latency and move data directly from one memory location to another.
- Latency sensitivity: Small amounts of added latency can accumulate when communication happens repeatedly across very large GPU clusters.
- Network stack changes: GPU-to-GPU communication may use different encapsulations, drivers, and hardware capabilities than ordinary TCP/IP application flows.
Loss is especially important for model training. The goal is to move data from memory to memory in a pattern that avoids packet loss. If packets are lost, the system may need to return to the most recent checkpoint where data was saved, slowing the training process.
Latency matters for both training and inference. In training, even microseconds can accumulate because GPUs exchange data repeatedly across large datasets and many iterations. In inference, latency affects the user experience directly.
The IDC requirements referenced in the session align with this pressure: low latency and high bandwidth, high-speed connectivity such as 100 Gbps and 400 Gbps interfaces, and infrastructure and information security. Security is not explored in depth, but the rationale is clear: training and inference workloads may involve sensitive data, so the network architecture must support both performance and appropriate controls.
4. Large Model Training Requires Data Center-Scale Throughput
The Meta Llama 3.1 training example illustrates the scale involved in modern AI infrastructure. The session cites figures from a Meta paper describing the training environment:
- 16,000 GPUs used to train the model.
- 240 petabytes of storage across thousands of servers.
- 2 terabytes per second of sustained throughput for training data.
- 7 terabytes per second of peak throughput.
- 400 Gbps interconnects across storage and GPU infrastructure.
The distinction between terabytes and terabits is important: the throughput discussed was in bytes, not bits. Building infrastructure capable of sustaining that volume of data movement requires a large, carefully designed data center architecture.
The training environment referenced protocols and fabrics including RDMA over Converged Ethernet (RoCE) and InfiniBand clusters. It also used enhancements such as improved ECMP behavior to spread traffic across large GPU environments. The lesson is that ordinary network defaults are not always enough when training jobs depend on thousands of GPUs and storage systems moving data continuously.
5. AWS AI Infrastructure Combines Compute, Storage, and Connectivity
From the AWS perspective, AI infrastructure is built from related compute, storage, and network layers. Compute options include AWS-designed AI chips such as Trainium and Inferentia, as well as NVIDIA GPUs. The right choice depends on whether the workload is training a model, running inference, or doing some combination of both.
Storage options include Amazon S3, Amazon EBS, and Amazon FSx for Lustre. Large training clusters need high-performance access to training data, while generative AI applications may also use databases that provide additional context. That context can live in vector databases or more structured databases depending on the application design.
Networking services connect those systems. Amazon VPC is the underlying construct for building workloads on AWS. AWS PrivateLink can provide private connectivity between VPCs or services. AWS Direct Connect provides private connectivity from colocation facilities or customer environments into AWS.
Direct Connect is the primary point where Megaport fits into the architecture described. If a customer has training data in a facility or colocation environment and needs to move it into AWS, private connectivity can provide the path. The session describes options ranging from 50 Mbps up to 400 Gbps, depending on the connection model and requirements.
6. Direct Connect Provides Dedicated and Hosted Connection Models
AWS Direct Connect provides private connectivity between a colocation facility and AWS. A customer data center typically reaches the colocation facility first, and AWS owns the connectivity from the AWS router into the AWS cloud. From there, traffic can reach destinations such as S3 buckets or other AWS services.
In a dedicated connection, a physical fiber cross-connect links an AWS router to the other party’s router. If Megaport is providing the managed service, the connection can terminate on a Megaport router, with Megaport then providing onward connectivity to the customer environment, another rack, another colocation facility, or other parts of the customer’s network.
In a hosted connection, the connection is virtual rather than a dedicated physical customer port. Hosted connections allow customers to consume speeds from 50 Mbps up to 25 Gbps instead of being limited only to physical port speeds. AWS still owns the connectivity into the AWS cloud, while the partner provides the path from the AWS side through its network toward the customer environment.
Both models support the same broad AI objective: moving large amounts of data into cloud environments where training or inference infrastructure can use it. The right model depends on bandwidth requirements, location, partner architecture, and how the customer wants to consume private cloud connectivity.
7. GPU Placement and Server Architecture Affect Latency
Latency sensitivity begins inside the server. The session uses an example of a server containing eight NVIDIA H200 GPUs connected to a top-of-rack switch. In a simple mental model, one GPU communicating with another GPU might send traffic out to the top-of-rack switch and then back down to the same server.
That extra path adds latency. Even if the delay is measured in microseconds, it can matter because of how often GPUs communicate during training. A small amount of latency repeated across a large training job can become significant.
To avoid unnecessary off-server traffic, AWS and GPU providers such as NVIDIA provide connectivity inside the server itself. If a training job requires only the GPUs within that server, communication can remain local rather than leaving and returning through the top-of-rack switch. As GPU counts grow, architecture must continue minimizing distance and hops between components that exchange data frequently.
8. UltraCluster Supports Low-Latency, Non-Blocking GPU Communication
When a training job requires more GPUs than a single server can provide, the network architecture between servers becomes critical. AWS UltraCluster is described as an AWS-owned and AWS-managed network architecture inside AWS Regions that provides low-latency communication among GPUs and the bandwidth needed to move data from storage systems such as FSx for Lustre into the GPUs.
The session states that UltraCluster can support up to 30,000 Trainium chips with Trainium1. With Trainium2, the bandwidth and capability increased further. The architecture is described as providing petabyte non-blocking networking for GPU communication while minimizing latency among GPUs.
Non-blocking bandwidth matters because large AI training environments cannot afford communication bottlenecks between GPUs or between storage and GPUs. GPUs have finite memory, so data must be loaded in batches and training progress must be checkpointed back to storage. That makes the storage-to-compute path just as important as the GPU-to-GPU path.
9. Storage Locality Can Reduce Training Latency
Object storage access patterns can introduce additional latency if data must be retrieved across multiple availability zones. Amazon S3 is described as a regional service, meaning access to an object can involve infrastructure across availability zones within a region.
The session calls out Amazon S3 Express One Zone as a way to reduce latency for certain AI workloads. Rather than accessing data across multiple availability zones, S3 Express One Zone serves data from a single availability zone. If that availability zone is the same one where the GPUs are located, the architecture can reduce the latency involved in loading data into GPUs and saving data back during checkpointing.
This is both a storage decision and a network design decision. For training architectures that repeatedly load batches and write checkpoints, reducing storage access latency can improve the workflow.
10. EFA and SRD Change the Network Stack for AI Workloads
AI workloads can require a different network stack from traditional TCP applications. AWS developed Elastic Fabric Adapter (EFA) to support high-performance communication for workloads such as GPU-to-GPU training.
EFA allows an AI application to interact with the network without needing to traverse the CPU in the same way a traditional application path might. The application communicates through lower-level libraries and frameworks such as NCCL or MPI, using components such as Libfabric and EFA drivers to support memory-to-memory communication. The purpose is to reduce the time required to move data from a memory location inside the server to the network interface.
AWS also uses Scalable Reliable Datagram (SRD) as part of its networking stack for AI and other high-performance workloads. The session contrasts SRD with traditional ECMP behavior. With conventional ECMP, a flow is assigned to a path based on a hash, often using the five-tuple. Once that path is selected, the same flow continues to use the same link, even when other equal-cost paths are available.
SRD allows intelligence in the Nitro system to take a single flow and spray it across multiple paths in the network. This can increase effective bandwidth, reduce latency, and reduce the likelihood of hot links where most traffic lands on only a subset of available links because of hashing behavior. For AI training clusters, using more of the available network fabric helps support the bandwidth and consistency required for distributed training.
11. Inference Still Depends on Low-Latency Backend Networking
Much of the session focuses on model training because training creates the most obvious changes in network architecture. Inference also has important networking requirements.
From the front end, inference can look like a traditional application. A user or application sends a TCP request, the request reaches an application tier, the model is invoked, and a response returns. But the backend infrastructure may still depend on multiple GPUs. Larger models, including examples such as Llama 3.1 or Anthropic’s Claude Sonnet, may be distributed across GPUs because of memory requirements.
That means inference can still require GPU-to-GPU communication behind the application interface. The client-facing protocol may look familiar, but the backend AI infrastructure still needs the right compute, memory, and network architecture to provide acceptable response latency.
Key Takeaways
- AI changes network architecture because the workload changes the traffic pattern. Training and inference introduce massive data movement, GPU-to-GPU communication, and tighter latency requirements than many traditional applications.
- Training is a distributed synchronization problem. When models span multiple GPUs, collective communication keeps each GPU’s view of the model consistent after training batches are processed.
- Data scale drives infrastructure scale. The Llama 3.1 example included 16,000 GPUs, 240 PB of storage, 2 TB/s sustained throughput, and 7 TB/s peak throughput.
- Loss and latency are not minor details. Packet loss can force recovery from checkpoints, and small latency increases can accumulate across repeated GPU communication.
- Private connectivity helps move AI data into cloud environments. AWS Direct Connect, including dedicated and hosted connection models delivered with partners such as Megaport, can support large-scale data movement into AWS.
- GPU and storage locality matter. Keeping communication inside a multi-GPU server and placing storage close to GPUs can reduce latency in training workflows.
- The network stack extends into the server. EFA, NCCL, MPI, Libfabric, SRD, and Nitro-based path selection support communication patterns designed for high-performance AI workloads.
- Inference still requires careful backend design. User-facing traffic may look like a standard TCP application, but large models may still depend on distributed GPU communication.
Conclusion
AI workloads require architects to think about the network as part of the compute system rather than as a generic transport layer. Training jobs depend on large volumes of data moving from storage to GPUs, GPUs synchronizing with each other, and progress being checkpointed back to storage. In that environment, bandwidth, latency, loss behavior, and path utilization directly affect how efficiently models can be trained.
Cloud-based AI architecture combines several layers: private connectivity for moving data into the cloud, storage services that can supply data at the required speed, GPU infrastructure that can run training or inference, and specialized networking capabilities that reduce latency and use the fabric efficiently. AWS services such as Direct Connect, UltraCluster, Elastic Fabric Adapter, and SRD are presented as parts of that broader architecture.
For practitioners, the main lesson is that AI should not be treated as just another workload placed on an existing network design. Some organizations may never train a large model themselves, but teams supporting AI initiatives still need to understand why the architecture changes. The data path, storage location, GPU placement, connectivity model, and network stack all contribute to whether AI workloads can meet their performance and user-experience requirements.
