Megaport Connect Cloud Networking Summit

The Full Stack Network: From Multi-Cloud to Bare Metal (And Everything We Broke Along the Way) – Full Transcript

Host: It’s one thing to talk about architecture in theory, but it’s another thing entirely to build and operate these environments in the real world, which brings us to our next speaker. Richard Olson is the tech lead for networking and compute at Canva, and he’s helped grow their network from a couple of VPCs into a global multi-cloud environment spanning hyperscalers, neoclouds, and bare metal. Richard has spent about 20 years working in networking and describes himself as a bit of a network automation hipster. Today, he’s going to share some of the architecture decisions, trade-offs, and the things that broke along the way at Canva as they scaled their architecture. So please welcome Richard.

Richard Olson: Thank you so much, Alexis, and thank you, everyone, for having me. Good morning, good afternoon, good evening, depending on where you’re coming from. And I just want to say a special thank you to Chicago for the weather. It is exactly what the weather is back in my hometown.

And speaking of my hometown, I’m from a little town that some of you may not have heard of called Canberra. In Australia, a little over 100 years ago, Sydney and Melbourne were arguing about who should be the capital city, and they couldn’t agree. So what they did is they drew a straight line from Sydney down to Melbourne, they found the spot with the most kangaroos, and they called that Canberra, which means meeting place. And so it’s pretty much your Washington, DC, but populated by marsupials and venomous snakes. And that’s where I live.

All right. Who am I? I started life in technology a little bit differently to most folks. I was throwing rocks into trees. I was literally getting a bit of field cable, finding a good-sized rock, taping it up with some electrical tape, and then hurling that into the bough of a nearby tree, because I was in the military. And when you’re in the bush in the middle of nowhere and your boss wants to talk to his boss, you don’t have a phone network. So I was the guy out in the middle of nowhere who would build that phone network in a day, rip it down the next day, move, and there we are.

From there, I left the military and followed, I would say, a fairly standard career trajectory in networking: a bit of level 2 NOC, level 3 design. And I taught myself to code along the way, which was cool. And then I started to get into cloud. I actually worked for a big cloud company. But when I was at that cloud company, surrounded by all of this cutting-edge technology, super cool, I just felt that I wanted to go back to my roots. There was this little voice inside of me saying, you love networking.

I was a foolish teenager. I actually bought an ATM switch on eBay. Had no idea what it did. It was $200. It was the worst $200 I ever spent when I was a teenager, but clearly something said networking. And so, about five years ago, I saw a job pop up at Canva, and it was cloud networking. So here were two of the things: the one I was doing, which I like, and the one that I love and have always done. And so that’s how I wound up at Canva.

For those that don’t know Canva, Canva is a global online visual communication platform designed to empower the world to design. It’s a simple way to create beautiful designs. And what that means for folks in the room, network engineers and architects, is you have infinite whiteboards, and you can draw awesome network diagrams. And I’m going to show you a few of those today. So Canva supports the end-to-end visual communication process, which is really cool.

All right. We’ve got a fair bit of content to cover today, and I’m going to dive in in a second. We’re going to start from the beginning, with a single-cloud foundation, one region, one cloud, and we’re going to grow that into multi-region. And then we’re going to head over to multi-cloud and neocloud. And I’m going to talk about some of the integration patterns that we’ve used, some of the lessons we’ve learned, and, hopefully of interest, some of the ways we’ve learned to apply some cost efficiency.

Now, just to add a small disclaimer to some of the things that I do and the way that I talk. Unlike a lot of the networks that some of you folks might be managing, Canva is 99.9% cloud-based, and that 0.1% is office internet. So some of the solutions which you’ve got available to you, your routers from your favorite vendor, your switches, I didn’t have those. We are operating zero physical infrastructure from that perspective.

All right. So our story begins. Back then, everything important lived in prod, and there wasn’t really a network so much as a loose collection of VPCs and subnets which didn’t need to talk to each other. But then, as the company grew, the VPC count exploded and nothing could communicate. When I joined Canva around June 2022, I was one of the first three cloud networking engineers, and our mandate was simple: build a network.

I mentioned the rate of growth was huge back then, and we’re talking network growth here. We had 20 VPCs in 2019, and that exploded to 200 in 2022. It tripled in 2021 alone. And at the time, some of these VPCs had started to communicate between back-end services, and we were using a mix of ad hoc VPC peering where teams needed that inter-VPC connectivity.

But one of the first things that we wanted to do as the networking team: we had an egress capability going from a back-end service out to the internet, and we wanted to centralize that capability. And we also had hundreds and hundreds of NAT gateways. Most of these VPCs weren’t even using these NAT gateways, and as some of you folks may know, NAT gateways have an hourly availability cost, and that’s something that we wanted to reduce.

So what we did, we rolled out transit gateways, pretty standard architecture in Amazon. We grabbed all of our dev and staging VPCs and plopped them into what we called the non-production transit gateway, and all of our prod VPCs into the prod transit gateway. And what that gave us is a way to do code progression or promotions. It meant that when I was making a code change, I would test it on my non-prod transit gateway. This is the infrastructure as code that some folks have been talking about. And let’s say dev and staging went well, that would then be promoted up into prod and the change would be rolled out. Sensational.

And so we rolled out a centralized NAT gateway capability. We shut down hundreds and hundreds of NAT gateways. Big cost saving. Fantastic. And this then evolved into our centralized firewall capability. But just as we built this network, we actually needed to replace it, because after being in a single region for a decade, we needed to expand our footprint globally.

So there’s a bit of a lesson learned here. You might have got it from the last slide. Please start with a unique addressing plan. You don’t need to use 10.0.0.0/16 everywhere just because the docs say that. By folding our dev and staging together on one network, and our prod network also on 10.0.0.0, we spent a lot of time shuffling databases, adding extra prefixes, and that sort of stuff.

The next lesson that we learned: transit gateways are great for connectivity within a region, and they definitely have the capability to go between regions. But with transit gateway peering, transit gateways are a region-based resource. So if you’re peering between regions, you’re using static routes. And at the time, we had a really messy IP schema, so the idea of summarized static routes everywhere was quite a painful thought.

So we turned to Cloud WAN. This is circa 2023, and it was about 12 months earlier, I think, that AWS had announced the general availability of AWS Cloud WAN. And it brought several enhancements that we really needed. One of them, which is nice to have, is any-to-any mesh. We could easily achieve this with transit gateways plus infrastructure as code. But what Cloud WAN gave us was this nice declarative policy in JSON, and I could simply append a region to a list and magic would happen. Job done.

Dynamic routing, though, was the big one. I just touched on managing that list of static routes. Now, we could have done that with infrastructure as code, coming up with a recursive list and populating all of our VPCs everywhere. But as a lot of you are network engineers, I’m sure you know dynamic routing typically beats static routing nine out of 10 times when you’ve got that sort of an environment.

The last thing that Cloud WAN gave us that we really liked was segment sharing. For those that aren’t aware of segment sharing, in Cloud WAN you define segments which you attach VPCs to. You can see on the diagram I’ve got a fabric and a platform segment, for example. And then you can selectively say, hey, fabric can talk to platform, but it can’t talk to the isolated segment. And we really liked that, because it gave us a way to control this selective connectivity.

So, TL;DR, it was a no-brainer over a number of regional transit gateways using static routes. And it’s actually been really boring, in a good way. We’ve only made 20 configuration changes since we deployed that in 2023, and most of them are adding or removing segments or regions.

Inside AWS, just like the diagram shows, we actually run two Cloud WAN core networks, similar to the transit gateways I mentioned, one for prod and one for non-prod. We run this across somewhere between six and eight global regions, depending on where we are at the time. And having these two Cloud WANs, like I said before, still gives us that code promotion path, not that we’ve needed it, with 20 configuration changes over a couple of years.

But what we discovered during the Transit Gateway era, in the later days, was that we had a subset of services which needed to communicate with both production and non-production. We had this lovely separation, prod will never talk to non-prod, but we had services such as secrets management, infrastructure as code, and build CI runners, which would need to touch and mutate infrastructure across both environments and needed to talk to prod and non-prod. And that’s where we coined the term platform. To provide platform services connectivity between the Cloud WANs, we actually use Transit Gateway. Still to this day, you can’t natively connect Cloud WANs.

AWS has recently added routing policies, but at the time, as you can see on the diagram, we had a special filtering segment we called platform to non-prod, with no VPCs attached, which let us manipulate the routing policy before there was a routing policy. And so that meant that we’re not sharing all of prod to non-prod, just those select VPCs.

A fun lesson came from this connectivity using transit gateways. If you do connect multiple Cloud WANs, make sure you’ve got transit gateways in all of the regions where you connect these Cloud WANs. Otherwise, what happens is your packet that’s leaving your non-production VPC, say in ap-southeast-2, Sydney, good region there, if you don’t have this local transit gateway, is going to deterministically randomly, and that’s their documentation, select a path, probably through a suboptimal region. So that means if Australia needs to go to the US, unless you’ve got that local transit gateway which makes it cross to prod immediately, it may actually go via Europe to get to the US. So having a local transit gateway ensures that traffic crosses between prod and non-prod in the local region, and from there you have optimal routing to the remote region.

All right. I just mentioned segments, one of the nice things about Cloud WAN. You can tell from the title where I’m going here: the cost of complexity. So we created those two concepts. We called them routing domains, the prod and non-prod routing domains. And over the top of that, we overlaid what we called functions. There were functions like the one I’ve just mentioned, platform, but we had lots of functions: supply chain, source control, general purpose, build, isolated, CI. You can see where this is going.

These are all separate segments on that Cloud WAN. And what ended up actually happening is no one knew where to deploy anything. We use Kubernetes as our compute platform, which I’ll talk to a little bit more later. But a customer would come to us and say, where do I put my thing? I need to talk to this stuff. And in the end, no one knew where to deploy anything, and nothing could communicate anyway. We had about 15 different combinations at one stage.

So the lesson learned: every new segment felt like good security hygiene at the time, but each one added routing policy, attachment rules, and one more thing to think about at 2 AM during an incident. So we flattened things. We parked everything into the fabric segment, regular workloads and that sort of thing. The platform segment stayed, because that’s where those cross-domain prod to non-prod services stayed.

And this didn’t actually result in a loss of security. Some of you probably use VRFs, and you know that they are quite effective at separating things. It didn’t result in a loss of security, but that was quite a coarse control, and it had a lot of operational overhead and workarounds needed. And so this pushed the complexity to the application layer. That’s probably the lesson learned here. For us, network complexity has an ongoing cost, even when it looks free from day one.

All right. Now we’re entering the multi-cloud era. We’ve just gone multi-region. At the time, I think we were in three. This was about 2023. And a second cloud turned up. GCP arrived. And everyone at the time was talking about multi-cloud. I remember seeing all this media on LinkedIn, but then it was real. We were actually doing it. It wasn’t just talk.

And at the time, we had a requirement for high-speed internal connectivity. We’re talking more than your classic 1 to 1.5 gigabit that you can get on a VPN tunnel. I didn’t want to have dozens of VPN tunnels. And we were also going to be transferring a lot of data, and data transfer out via the internet is one of the most expensive ways as well. So a VPN service wouldn’t cut it. We ended up connecting using a service called GCP Cross-Cloud Interconnect.

There’s going to be a talk later by Will, who’s going to go in depth on things like Direct Connect and Cloud Interconnect. But essentially, the service is AWS Direct Connect on one side, GCP Interconnect on the other, and the Cross-Cloud Interconnect part comes in because GCP manages the cross connects. For anyone that’s done a cross connect in a data center, typically you get issued the LOA, letter of authority or authorization, and you need to arrange that yourself. But Google handled that for us.

And it worked. This was mid-2023. I think the product had just come out, and it was the obvious choice. This is approximately two years, I’d say, before the multi-cloud interconnect products started to come out. It was the obvious choice. We only needed to communicate between two regions, two clouds, in Northern Virginia. We needed that high throughput, like I said, and we wanted to maintain a reasonable price, and Direct Connect was substantially cheaper data transfer. But by building on a point-to-point connectivity product, we’d ultimately have to replace that.

More on that in a little bit, though, because remember that complexity that I mentioned before? Now we had a segmentation headache between clouds. You can see here we’ve got a few fewer segments. We started flattening these different things, but it got a bit tricky extending this between two clouds. Walking through the diagram, on the left we’ve got AWS, showing the Cloud WANs, their segments, the VPCs, and the VPCs connected to Direct Connect gateways. And on the right, we’ve got Google, with a classic hub-and-spoke VPC peering arrangement and some Cloud Routers.

The problem came because our first ever multi-cloud connection didn’t have fancy routers. We were constrained by the features that the hyperscalers gave us. This is a direct connection from a Direct Connect gateway to that Cloud Router. So we didn’t have communities, we didn’t have route maps, and we only had very light controls around summary routes.

To provide this connectivity between segments in a fashion similar to what we had on the AWS side, we ended up using a mesh of VIFs or VLANs, depending on your hyperscaler terminology. As the diagram shows, prod goes to prod. Simple. Where’s the problem? Well, we had those platform services, and everything on the GCP side needed to talk to those platform services. So what we did is we had a special platform Direct Connect gateway on one side, and it would have a pair of redundant VIFs to each of the Cloud Routers on the other side. And that meant that we provided that any-to-any connectivity for platform.

It was also tricky, what happened here. This provided the connectivity, but the next challenge we had was on the Google side. AWS Route 53 was our authoritative private DNS at the time, and we wanted GCP resources to be able to consume that private DNS. And GCP has forwarding zones. That’s pretty straightforward. The problem was that these forwarding zones are actually sourced from a public address range.

Don’t ask me to quote it right now. It’s 35-dot-something. But what that meant was, similar to the issues with the 10.0 private addressing back in the early VPC days, now we had effectively a duplicate overlapping range which had to be advertised across multiple segments. And in the end, what we had to do was just be super careful where we propagated that from. It would only go from build to build, non-prod to non-prod, prod to prod, and that way we ensured that the DNS response made it back to where the request came from. So this worked okay. Fortunately, we never had to consider both clouds having multiple regions and achieving that any-to-any connectivity until we had a little bit of time to think about it.

But as I mentioned just earlier, we ultimately had to change this. While GCP was in a single metro adjacent to an AWS region, having a point-to-point connection was workable. Latency and routing were optimal. But the question was always asked: we’ve just gone multi-region and multi-cloud in a single year, so what if we go to another cloud, another region? If I throw on another Google region, or another Amazon region, or another cloud, how are we going to make sure that we have that end-to-end connectivity?

So this basically created the classic networking problem of a full mesh versus a hub and spoke. The more regions you get, the more links. Three is a nice trade-off, three regions, three links, but then you get more links and more links. And so this diagram shows eight clouds and eight regions, and if I’m doing my mathematics correctly, this is about 56 redundant links for just eight regions, hub and spoke versus a full mesh.

So in 2024, we pivoted to a NaaS provider, or network-as-a-service provider. Back in 2024, as I mentioned just a moment ago, the more recent multi-cloud interconnect initiatives hadn’t been announced. So with the exception of a handful of services, like that Google Cross-Cloud Interconnect product I mentioned earlier, there weren’t really any other private connectivity options. So now, instead of AWS in our primary region being the hub with other clouds hanging off of it, or a full mesh, NaaS providers are the hub, and every cloud is just a spoke. We can roll out an arbitrary number of clouds and regions and provide that any-to-any connectivity that the network needs to provide.

So fast forward to today. This is a real diagram. Multiple interconnected NaaS providers give us complementary capabilities. We’ve got global coverage, fantastic, physical ports, virtual routers, internet services, multiple jurisdictions, more than about half a dozen clouds, and somewhere between 15 and 20 regions, depending on the current requirements.

All right. Then the next chapter. Up until 2024, this had all been fairly orderly. We went single cloud, single region, then multi-cloud and multi-region at the same time. And then GPUs and bare metal entered the picture. We needed to be able to connect to wherever the GPU capacity was, and that increasingly meant neoclouds. And most neocloud services offer bare metal. They don’t have a decade or two of experience operating high-scale virtualization platforms. So the problem every neocloud integration comes down to is that they’re all completely different.

We run Kubernetes, which provides us a universal compute platform. And early on in our Kubernetes journey, we opted to have routable pods. The hyperscalers that we use both defaulted to this pattern, and we’re 100% cloud, like I mentioned. So it just made sense.

And just to level set, I’m not going to deep dive into Kubernetes, but in Kubernetes, you have nodes. This could be a virtual machine or a bare metal machine running the kubelet, which phones home, and just like all servers, they’ve got an IP address. But within each of these worker nodes, there are dozens if not hundreds of pods or containers running workloads for us, and each of these pods has a unique IP address assigned to it by the CNI, or container network interface. So when I say routable pods, what that means is that the network is aware of every pod and which node it’s on.

In AWS, this is solved using something like the AWS CNI or Cilium ENI mode, where the CNI will actually talk to the AWS API and say, hey, can I have an IP address, or a /28 or a /80 prefix? And the hypervisor knows that that prefix or IP address lives on that NIC and knows how to route to it. But when we’re going back to bare metal, all of those primitives don’t exist. So while connecting to hyperscalers is a fairly well-known and documented process, what I’m going to walk through now is how a number of different neocloud providers integrate with your network, both from a node and a Kubernetes perspective.

All right. Provider A: bring a router. This provider gave us a layer 2 domain and the ability to run virtual machines with multiple NICs, but no routing. So we went BYO. We brought a router. It’s a virtual machine, and it provides a first-hop redundancy gateway. You can see VRRP down there. It’s been a long time since I’ve configured VRRP. And we do dynamic BGP down to our nodes.

Just a quick note on the dynamic BGP. Folks that have run a Kubernetes environment will be familiar with this, but typically a lot of the CNIs will assign a prefix to a node. And how does your network know what node has what prefix? That’s where BGP comes in. Why dynamic BGP? Because these nodes are coming and going constantly. Folks that have configured BGP, you typically punch in a static IP address for your neighbor: here’s the set of capabilities and configuration that we’re going to negotiate. That doesn’t work so well when you’re getting 100 alerts because a couple of nodes rebooted. So we use dynamic BGP.

Now, for this particular provider, fiber interconnects are hard to get in the regions that we use. So it phones home over the internet, and that works well, but that virtual machine is now in the critical path. We’re no longer outsourcing that operational complexity to the provider, so we have to consider that very carefully. And on top of that, it’s got throughput limits, so we’re at the mercy of the internet underlay.

Lessons learned. Just like during our first steps building out that proper WAN with a NaaS provider, we learned that hyperscalers have some network rate limits. Some of you would be familiar with this. Typically, the more CPU you throw at a virtual machine in a cloud, the more network throughput you can get. Maybe there are network-optimized instances. This is very well documented, known, and productized. But in some of these neoclouds, it turns out that they also have rather aggressive rate limits, and we had some instances where VMs weren’t able to achieve more than 1 or 2 gigabit. We needed a lot more than that. Sometimes there are workarounds to these constraints, but I guess the lesson learned is that performance testing is absolutely critical for uncovering these issues.

Bring everything. So we’ve just brought a router, and now we’re bringing everything. In this case, Provider B was happy to interface with fiber, but had no virtual machines. A lot of neoclouds don’t have those virtual machine capabilities, like I mentioned. And in this case, the internet was also heavily capped, so I can’t run a virtual router, and I can’t use the internet anyway to phone home using an SD-WAN-style capability.

There’s no routing either, so literally everything is ours. We have to run VRRP like the previous provider, but now we’re also providing things like DHCP, and we’re still doing that dynamic BGP. Now we’re doing literally everything. You can see down the bottom, all platform connectivity concerns are now managed by us. So we’re essentially running a small data center network on somebody else’s layer 2, and after years and years of cloud abstractions, we’re back to the absolute fundamentals.

It might seem trivial to folks that do on-prem: just chuck a pair of routers in the rack next door, sweet as. But we operate in an environment which doesn’t have any on-prem infrastructure. This would be completely foreign to us. And it also needed to be done in days, not months. New data center provider, racks, cabling, cross connects. If you can do all that in days, I would be impressed. Something else in the diagram, down the bottom there, which I just mentioned: the vendor manages the cabling and layer 2. So you could almost say at this point that we’re essentially just outsourcing that rack-and-stack and cabling capability.

All right. Bring a Kubernetes controller. So we’ve had providers A and B, bring a router and bring everything. This one’s bring a Kubernetes controller. This one actually looks the easiest on paper. This is a proper cloud. It’s got real VPCs. They own the route tables.

The problem that this created is in the Kubernetes layer now, because there was no integration with our CNI. I briefly touched on how, in AWS, you make an API call, and it knows that that IP address or prefix is on a particular NIC. We didn’t have any of that automation built in. Where the hyperscalers solve this for you and tie the CNI into their native constructs, this provider only solves it with their own managed Kubernetes. But we also couldn’t drop in our own router, because they own the route table in this case. There is no layer 2, like Holly mentioned in her talk.

So we had two choices. We could either adopt their Kubernetes and diverge our platform, and we had a lot of tooling built around our Kubernetes platform, a whole bunch of release and deployment processes and all that sort of stuff. Or, and this is the option that we picked, we actually wrote a Kubernetes controller. What that controller does: the cluster is aware of which node has which prefix, and this controller actually reaches out, touches the API, and updates the route table. So now that route table is aware of which prefix, in our case a /26 in this cloud, lives on which node.

All right. Just bring yourself. From a networking perspective, by far the easiest. This one is pretty much our benchmark. We get a pair of high-speed fiber interconnects and redundant BGP sessions to their routers, and they handle everything. So if there’s node and pod route aggregation, and I mentioned /26 before, they all get rolled up into whatever our prefix is, say a /18, or a /64 for v6.

But unlike a hyperscaler, with a lot of strict guardrails and quotas and that sort of thing, on this side there were actually humans assisting in the process, so we could do pretty much anything from a network integration perspective. And when a provider gets this right, the network team, so my team, barely notices them, so kudos to them. It’s really just a pair of high-speed links and redundant BGP connections, and all of the aggregation is handled. Platform concerns, DHCP, VRRP, gone, a distant memory.

So putting all of these together, we actually realized there’s an interesting U-curve of complexity, with ease of integration on the x-axis and how much of the network the provider owns on the y-axis, or have I got those axes back to front? A more capable cloud can actually be harder to integrate than a simpler one. On the left-hand side there, you can see bring a router. Pretty easy, layer 2 only. Bring everything, somewhat easy, also layer 2. But then down the bottom here is this real dip, where we had to bring a Kubernetes controller and cut actual software. We’re not just talking network integration here. We’re talking a full software package in that controller. And then straight up the other end, just bring yourself. They own everything, and the hyperscalers are all up that end as well.

All right. We’ve done a few neoclouds over time, and we’ve run into some obscure issues. Distributed computing fallacies apply to unprepared workloads. What does that mean? Well, the computing fallacies: latency is zero, bandwidth is infinite, the network is reliable. We grew up in one cloud and one region, and there’s a lot of data in that region. Data has gravity. It’s hard to move. It’s expensive to move, and it costs a lot to store when you’ve got lots of it.

But now we’re in a world with neoclouds, where you’re dropping workloads wherever you can get GPUs or compute. So what workloads, and particularly application teams, have to be aware of is that those fallacies do not apply. Latency is definitely not zero, and bandwidth is definitely not infinite, and we just need to tune for that as appropriate.

This next one, default ARP setting, is basically a warning to say that clouds, neoclouds, and hyperscalers may do interesting things with the default builds and images that you deploy in them. In this case, a default ARP setting suppressed replies for the pods. This was from a hyperscaler, basically disabling internal ARP on that virtual machine. And there’s a bunch of other stories similar to that I could share, about how we actually hit the ARP table limitations in one cloud, which I think was 2,000 by default in Ubuntu.

Full IPv6 support is still rare in the wild. Some of you who are more familiar with Kubernetes are probably familiar with the fact that it eats IP addresses for breakfast. How are we doing all of these routable pods with IPv4? Well, the short answer is IPv6. And still to this day, in 2026, it’s difficult to get full IPv6 support. It’s rare in the wild.

And the last one down here ties into those distributed computing fallacies a little bit. Storing 1 gig for a month often costs the same as downloading it once, and some of the other talks are going to go into a bit more detail on that. But what if you have to download it twice? Does it then make more sense to just replicate it and then download it for free infinitely?

Speaking of free, cost engineering for fun and profit. All right. Cost saving tip number one. By pivoting to NaaS providers with virtual routers and internet access, we’d actually already built a way to achieve ubiquitous inter-cloud connectivity. So why not just pop a firewall on those routers?

Very obviously, this removes managed NAT gateway and managed cloud-native firewall processing fees and all that sort of stuff. And you can see that we’ve got a number of different clouds plugged in here. This is now a standard egress deployment for every cloud. We’re not using a proprietary cloud number one solution and cloud number two solution. And in the case of the neoclouds, which have no native firewalling, this ensures our egress controls are in place for the entire network, not just the hyperscalers.

So throwing out a hypothetical: 1 petabyte of transfer each way, a download and an upload. On the download, ingress to the cloud is typically free, so you’re paying processing fees. Let’s just say it’s 6.5 cents a gig. So for 1 petabyte, that’s $65,000. On the upload, you’ve got data transfer out as well, plus your firewall processing fees, so about $115,000, or $180,000 in total.

The flip here is, what if you had some sort of fixed-rate transfer through a NaaS provider? If you have consumption-based billing versus fixed-rate billing, you are always going to have a break-even point. And so you’ll have to do your own math on what your download versus upload patterns are, but essentially, beyond that break-even point, it’s definitely going to be more worthwhile just to outsource it. So on the upload, perhaps now you’re only paying Direct Connect fees out, and the download is still free on Direct Connect ingress into the cloud.

But everything in tech has trade-offs. I think it was Russ White who said if you can’t find the trade-offs, you’re not looking hard enough. So moving your egress to a NaaS provider, just like regular on-prem networking, means you now own the capacity management. Cloud is all about elasticity. If you’ve got a really elastic workload that goes from zero to 100 gigabit and does that for maybe an hour a day, well, bad news, you’re probably going to have to size for peak capacity, which means you’ve got unused capacity for those 23 hours. Alternatively, you’re going to have to manage expectations around performance. Your job no longer takes an hour, it now takes four hours, or something along those lines.

There’s also a reliability piece to this. Cloud-native firewalls have been engineered to ensure resiliency to failure scenarios, and now you own that. So it’s no longer a case of, we’ll pop an endpoint in AZ one, two, and three, and job done, everything just works. You’re now on the hook for that firewall going bad.

Just a slight side quest on this one, on the reliability side. There are ways to do this from a NaaS perspective. One of the things that we played with was actually injecting a pair of, and we’re talking v4 here, a pair of /1 routes into the VPC. What that means is we’ve got these two routes, which cover the entire internet, dynamically coming in through BGP on the Direct Connect product, and then we kept the static 0.0.0.0/0 route. So that meant that in the event, which should not happen, that both of our firewalls and routers went down in that particular POP for that metro, both of those dynamic /1s would be withdrawn, and the static 0.0.0.0/0 would take over and start hitting the cloud-native stuff, just as a bit of a fail-open circuit.

Cost saving tip number two. This one is pragmatism over perfect design, otherwise titled: cut your inter-AZ costs by one third with this one simple trick. For VPCs that talk to each other, we use VPC peering instead of Cloud WAN. We’ve got universal connectivity across every region, in this case via the Cloud WAN, but you can see there we’ve got three hypothetical regions, each with two compute VPCs and a resource VPC. We actually scale out our compute horizontally in some regions, just based on breaking apart that blast radius. They all talk back to a resource subnet, and there’s a lot of traffic going between, in this case, three VPCs in that region.

So we have universal connectivity, everything can talk, but for those cheaper regions, I beg your pardon, for those high-affinity links, you don’t pay for intra-AZ connectivity anymore. VPC peering is free within an availability zone and 1 cent in either direction, slightly confusing terminology, across AZs. Cloud WAN or Transit Gateway, or pick your other product, NCC from Google, are typically about 2 cents per availability zone.

So in a typical region with three availability zones, setting aside application-layer optimizations, which honestly are probably the better option, or topology-aware routing and those sorts of things, if you’re doing 1 petabyte from one AZ to itself and two other AZs, we’re now comparing $60,000 with Transit Gateway or Cloud WAN versus $40,000 using that peering link. So straight away, that’s a one-third saving, because that intra-AZ stuff is free.

And the pragmatism comes in here because, as networkers often operating at high scale, we try to simplify everything, and it is a beautiful design to have a hub and spoke with every VPC nice and standard, nice and simple. But in this case, the cost-effectiveness overrode that simplicity.

All right, cost saving tip number three: private VIFs, not transit VIFs. Similar to the last one, where I was talking about that processing fee, with our hybrid networking we also bypass the Cloud WAN or transit gateway or cloud router if possible. On the AWS side, we use private VIFs into Direct Connect gateways rather than transit VIFs. That means the inter-cloud routing doesn’t pay that 2 cents per gigabit processing fee, and we run a DX gateway per POP. That gives us fine-grained routing control. And you can also see there that there’s one per segment, but that’s nice and simple these days, as I’ve talked about before.

But trade-offs: this is now creating an operational cost. How many DX gateways are we going to have? What issues are we going to run into? And for us, being a software shop, everything is infrastructure as code. From a scale perspective, this is actually trivial. I don’t need to worry so much. You can have up to, I believe it’s 50 private VIFs, and each of those VIFs can have up to 20 VPCs. So that’s 1,000 VPCs per region, as we do per-region DX gateways. Plenty for most. And saving 2 cents a gig on every petabyte of data that we transfer is basically a top-of-rack switch for your data center.

So, lessons learned. There have been some recent changes to how these quotas are managed, but typically VIFs had a quota limit of 100 routes per BGP session. If you hit 101, your BGP goes down, alerts go off, and you’ve got to figure out what the heck just happened.

We actually got that quota lifted, but then we hit the next issue. We were no longer dropping BGP at 101 routes, but it turns out there was a secondary quota: because of this private VIF architecture, the maximum number of propagated routes going into a VPC was still 100. And so route 101 didn’t drop BGP, but it was just put in the bin by the VPC route table. The VPCs all ignored it. And in one region, that actually got stuck, and even when we withdrew routes, it needed a support ticket to get fixed.

So the solution for us in this case was aggressive region-based summarization into the hyperscalers. They’ve still got that cap of 100 in the case of Amazon, and I believe it’s 300 in the case of Google. But then it was juggling: do we advertise a 10.0.0.0/8 from the DX and then have more specific region routes in the VPC? In the end, what we did was have per-region summarization and just leave that simple static route in our Terraform code that points back to the Cloud WAN for inter-AZ traffic.

All right. Everything we learned along the way. I’ve been hitting these points quite a few times. The wrong complexity doesn’t scale. Sometimes infrastructure as code is a good trade-off, and that does help you scale a little bit more.

With great control comes great operational overhead. With a lot of the solutions that I’ve mentioned, you need to take on an extra load. Some of those you can make go away with code. Some of those you’ll need to pick up in your observability. Some of them might just simply need more engineers, and there’s always a trade-off. If you can shave $1 million off of your cloud hyperscaler bill, how many engineers does that buy you?

And the last one, this one’s really tricky. You could almost do a PhD on this. Know your cost per gigabit or gigabyte for every path.

All right, that’s a wrap from me. Thank you very much, everyone, for listening. And feel free to hit me up on LinkedIn for any fun tips on how many copper cores and steel cores were in field cable back when I was in the army, or to talk about cloud networking. Thank you.