Megaport Connect Cloud Networking Summit

The On-Ramp Illusion: Connecting to AWS, Azure, GCP – Full Transcript

Host: We are going to be kicking off the afternoon with a topic that I think just about every cloud and network engineer has dealt with at some point: connecting to major cloud providers. AWS has Direct Connect. Azure has ExpressRoute. Google Cloud has Cloud Interconnect. And of course, they all have their own terminology, architectures, and ways of doing things.

But getting connected is only part of the challenge. You still have to figure out routing, security, provisioning, and how to make all of those environments work together.

Our next speaker is Will Collins. He’s from Humana, and I’m sure you all know who Humana is. Will has spent nearly two decades working across enterprise networking, cloud, and automation. He’s also the co-host of The Cloud Gambit podcast and a LinkedIn Learning instructor.

Will is going to walk us through the real-world trade-offs of cloud on-ramps, including what works, what doesn’t, and some of the things that don’t always make it into the datasheet. Please welcome Will.

William Collins: Good afternoon. Last year at Megaport Connect I presented on one cloud provider, AWS. And it was a lot of fun. Watching Alexis and some of these creators doing very awesome things to tell the story gave me some ideas. So I brought Snoop Dogg along to tell the story of the VPC. So me and the Snoop D-O-G-G looked at his neighborhood in Long Beach.

That was the VPC. And then each house was a subnet. And we had signs in the neighborhood that were route tables. It was a lot of fun.

This year we’re going to widen that aperture, and we’re going to look at multicloud.

And if there’s one thing that I think is really worth taking away here, it’s this. We’re going to talk about a lot of details. We’re going to look at components. We’re going to have a lot of scars from war stories in networking over the years, like many of you. So there’s going to be a lot there. But I want you to think bigger picture, about an operating model that network engineers can actually operate, that we can scale. The operating model is super important here.

A little bit about myself. I hate talking about myself. First of all, if you want to connect with me on all the platforms, you can scan that QR code. Especially if you like this talk, give me a like, give me a follow. Even if we get to the end and you don’t like this talk, just make sure and connect with me and give me a like and a follow on all my stuff. I’d really appreciate it.

I’ve done everything from racking and stacking. I came into the industry right around the time that CatOS was exiting the industry. So a lot of war stories with CSS load balancers, data center stuff. I’ve always been pretty heavily focused on automation.

And one interesting thing here is, if you take a machete and slice down the middle of my career, I started out really working on the enterprise side, as a consumer of cloud technologies, and figuring out the hard way, over a long time, how to operationalize stuff. And then I actually made a pivot to the startup space and got to see this from an alternate angle. Having both angles was really useful. I worked for a company called Alkira, and we built a multicloud network platform. So getting both vantage points was really interesting.

So enough about me, let’s get into the real stuff. Sometimes I have a little too much fun up here, because if I’m not going to have fun, then I might go to sleep, and that wouldn’t be fun. What we’re going to look at right here, these are very affectionate caricatures. So don’t take these too seriously, but I like to use personas to tell the story.

And Megaport was so kind as to provide this slide deck with all these Greek statues in it. And I love Greek mythology, so it’s right up my alley. First up is Aphrodite.

Aphrodite is our developer. She’s beautiful. Everything she does is beautiful. Her VS Code setup is beautiful. She has all sorts of colors in her theme. It’s awesome. And when Aphrodite builds software, it always works on her machine.

Next, we have our resident network engineer. And this is really close to my chest right here. Poseidon gets a bad rap, and everybody thinks he’s salty because he lives in the ocean or something. But really, the reason he is salty is that pager is literally an extension of his arm. He’s getting blown up all the time because stuff keeps happening. He doesn’t really care about VS Code. He’s locked into SecureCRT. The black backdrop, fluorescent green text coming down, looks like the Matrix. And he probably has a list of devices organized by location or something like that.

Now we have Hermes, platform engineering. Hermes is the connection between worlds. He’s in VS Code a lot too, and he prefers to have everything through an API. Loves APIs. Loves cloud, and you’ll often see him updating very long and confusing-looking YAML files that power pipelines in cloud.

And last, I had a lot of fun with this one. I love our friends in security, by the way. I love them. But it’s just easy to poke fun sometimes. So we have Athena, the goddess of war. If you think about Athena, and you want to bring new software into the company, something that we don’t already have, Athena might make you go through 10 security reviews.

Very time-consuming, but if you go and you try and fight that, if you go into a room full of security engineers, you lose. They always win. So tread lightly. But Athena is the goddess of war, very strategic, very wise, and on her shoulders is the responsibility of keeping everything in the company safe.

And last but not least, we have the commander in chief. I bet you can guess who this is, hopefully. We’ve got Zeus. And Zeus is the CEO of Mount Olympus. Zeus does not care about VS Code. He doesn’t care about SecureCRT. None of that matters.

Zeus is in boardroom meetings, and he operates by throwing lightning bolts. And rumor has it that one time Zeus may have gone to a CEO summit. And he came back from that CEO summit and said, “Hey, we’re multicloud now.” Why? I don’t know. But we’re multicloud, and then guess who has to react to this? All of our lovely friends over here.

So act one, the Oracle. Delphi was popular for prophecies that oftentimes came true. Now, these prophecies were interesting because they were really useful after battle. So think of something like this. The Oracle of Delphi says, look at that river. If you cross that river, an empire will fall. I don’t know which one. What empire? Six, seven, six, seven. I have no idea. But an empire will fall.

This is a really good perspective, though. Because when I think about my past, early in my career, I got so fixated on datasheets and components on datasheets, on speeds and feeds. And Holly put it very elegantly earlier: the devil’s in the details, zooming out. It’s so important in our industry to really take a systems approach to how you think. Because you can line up a whole bunch of stuff from a datasheet, and it doesn’t mean your app’s going to run nicely over all these things.

Another thing about prophecies, past, present, future, is we have this thing called brownfield that sits around for a while. And brownfield is a great lesson to learn from. I had a leader a long time ago that would make me take notes when I was early in my career, like, hey, this is why we are where we are, and this is why we’re doing what we’re doing. It was such a good exercise.

I would call this person’s name out, but he’d probably be embarrassed. But he was a great leader and taught me a lot of good lessons here. So let’s look at how we got to where we are and what we’re going to do about it. Site-to-site VPNs.

This is literally the first thing I ever set up in cloud, a really long time ago. And it’s so funny. I do side quests all the time in my talks, so I’m going to go on a different tangent here. There’s a connection between smell and the part of the brain for memory. In neuroscience, it’s really interesting.

When I set up my first site-to-site VPN to AWS, ages ago, I was sipping a caramel macchiato from Starbucks. And it’s bizarre, because sometimes I’ll be having a caramel macchiato and site-to-site VPNs pop in my head. It’s quite bizarre, and honestly it’s not a good thing, but here we are. But basically, the point of this is that in AWS, under the site-to-site VPN, you have a connection.

And a connection is basically two tunnels. You terminate these tunnels to two public endpoints in the cloud. It’s easy. It’s what we’ve been doing forever.

And I remember thinking at this time, why does everybody think cloud’s so hard? This is not bad. I was younger and I was ambitious. And I thought, you know what? I really know what’s up. I got it. And then I hit the Hydra.

VPN tunnels started coming out of my ears. We had a big company. We were trying to modernize. And all these VPCs started popping up. So I’m provisioning VPN tunnels all the time from our data center to AWS. And this is just four right here. But in some of the bigger environments I worked in, I think we had over 100 at one. So think how many tunnels. And I’m troubleshooting.

And many times, something goes down, and it’s like, hey, where’s William? I was the person on the pager 24/7. And I went to my boss at one point and I was like, “I’m a VPN janitor now. I don’t want to do this anymore. I would rather go into plumbing or the trades, because this is killing me.”

So why so many VPCs? Well, in the early days of AWS, you would basically use accounts as barriers for cost and other things, because some of these more advanced features and designs weren’t here yet. So the more accounts that got spun up across your organization, the more VPCs went along with them. And this was just tedious.

Now, one thing to keep in mind, and I think it was Holly that hit on this as well, she covered it a little bit, is that this limitation still exists in the cloud today: transitive routing. There’s something called VPC peering. In Azure, it’s VNet peering. And it’s basically a point-to-point link between two networks, your cloud networks.

The challenge with this is that it can’t actually do intelligent packet forwarding at the end of the day. So if I wanted to use VPC B here to aggregate and connect a whole bunch of other VPCs natively, so VPC A could talk to one of these folks, it’s not going to work. Transitive routing does not exist.

So the industry reacted with a very interesting design. I understand how we got here, but this was a little bit tedious. There was a lot of work involved. Basically, to limit this tunnel sprawl, you would set up a transit VPC and you’d put vendor appliances in it. For us, it was Cisco CSRs.

And you would only have a few tunnels from the data center to these CSRs, which were in different availability zones. And then all the other spoke VPCs you had would be BGP over IPsec.

On the surface, maybe this doesn’t sound tedious to operationalize. But at the end of the day, think about throughput. If you’re doing migrations from on premises to the cloud, you have throughput issues.

You’re patching everything. You’re getting billed twice, from spoke and from hub. So there’s a lot of things here that were very challenging. And I forget who touched on this, but the experience that you have in the data center managing an ASR doesn’t actually translate directly. Even thinking of failover, we don’t have multicast here. And if you don’t have multicast, then that means you don’t have HSRP. So if you come into this as a network engineer thinking, okay, we can just automatically put these here and it’s just going to work, it’s just not that simple.

So AWS reacted with Transit Gateway. And Transit Gateway solved a ton of problems. This was a very good solution for the time, and it just made life much easier. Basically, you have a service. It’s a regional service.

And then you have these things called attachments. You have an attachment per VPC, and also attachments for on-premises networks. And then there’s Direct Connect gateway. If you had a single Direct Connect somewhere, you could facilitate connectivity back to other transit gateways and VPCs, even in other regions or across accounts. It was very good. And this solved quite a few issues.

Instead of dealing with the public internet for everything, you’re bringing that onto a private network. You’re killing the jitter. And there’s so many other things. You don’t have the sprawl problem with all the tunnels.

So I’m out of this tunnel purgatory that I was in. And you’re not dealing with these patching things all the time. The way I like to think about that previous slide is that it’s like lift and shift. All the developers learned the hard way, and all the engineers learned about lift and shift the hard way as well.

It’s something a lot of folks in the industry went through at this time. Now, the things it really didn’t solve, and this might be me nitpicking a little bit, but you still have that lead time. You’ve got to order the circuit. You have to do the LOA, the letter of authorization. You have to wait for the cross-connect. You have to have the smart hands. You have to go through all these steps just to get everything in place. And then you actually have to build a design around it.

Which is, hey, it is what it is. We’re engineers.

So this is really great. And this hub really became a service for one region. But it’s just for one cloud. And we’re talking multicloud here. So what happens next?

Three clouds, three doors. If you look all the way to the left, this circuit thing that you order, you have Direct Connect, ExpressRoute, Interconnect. And then all the things on the right are the things that you have to think about when you want to go and operationalize things. That’s your responsibility.

I was talking to someone before I came up, an hour ago, and I thought, oh, I should tell this story. We were talking about an outage. So back in, it would have been 2014, 2015.

When you set up an Azure ExpressRoute circuit, you basically have a pair of connections that are terminated on a pair of edge routers in Microsoft’s backbone. And it’s active-active by default. Okay, that’s not bad.

But then Athena walks in the door. And again, she’s the goddess of war. She says, “Hey, I need you to put firewalls everywhere that you can possibly put them. We need firewalls.” So of course we’re putting in firewalls. Well, later on down the road, after this design’s been in production a little while, we have an outage. And failover takes forever. I remember it was a bad day. I was a very salty Poseidon on this day in history.

And come to find out, I went back and looked at our configuration, our original design, and we had BFD, Bidirectional Forwarding Detection, enabled. Well, it turns out at that point in time, if you created an ExpressRoute between 2014 and 2016, Microsoft didn’t have BFD enabled by default on their edge routers. So what do we know about BFD? It’s got to be enabled on both sides to work. Because it was not enabled on both sides, we went back to all the default timers and all this other stuff. And when you factored in the firewalls and everything else, that outage lasted a pretty long time.

So that was a lesson learned. That was a detail. And at that time, the way that I found that out, I went into Azure’s portal and I looked and looked. I’m like, there’s no way to toggle BFD. I thought it had been enabled by default in the design. And that was something I missed. So all these little details, like Holly talked about earlier, matter. The devil is in the details.

And then you look at all these services, and each service has some sort of story here. No matter how much you know about network engineering, and you all know a ton, you’re network engineers, you have this knowledge, you have to look into the cloud and see how a lot of these things are applied there.

So act two. Homer, ancient Greek poet. I would have to guess, if he worked in IT today, he’d probably be an enterprise architect.

And he was famously quoted as saying, “You don’t choose multicloud. Multicloud chooses you.” And I think that’s absolutely correct. It can be hard when you’re the person in the trenches, when you’re that network engineer and it’s like, okay, another cloud, another cloud. It’s difficult.

And then you start thinking, why do we got to be in so many clouds? Let’s just stick with one. But the thing is, we’re here and this is reality. A lot of it happened through acquisition. There are many reasons why businesses are multicloud. A lot of reasons. And a lot of them make a lot of sense when you really dig down into the details.

At the end of the day, though, all of you are probably builders. We’re all problem solvers. And we have a lot of network engineers here. What are we going to do about it? How are we going to make the best and most consistent experience we can across these clouds that we have?

So in this frame of thinking, as I started on the solution and tried to think about this at a broader scale, I started to have two buckets. This bucket right here is things that are within my control, things I can influence, a lot of hygiene stuff, IP addressing, which we’re going to get into in a second. There’s a lot of things that are within my control. I control my own destiny with these things.

But then there are cloud-native constructs, where you can dial things and you can change some settings. But good luck trying to get a cloud provider to change the underlying behavior. Part of the reason they’ve reached the scale that they’ve reached is by keeping these consistent, and not doing every little thing where maybe every customer might want to change some part of the design. So we’re going to talk about some of the things on the left here, the things that are within our control. And my whole goal when I started looking at things this way was, okay, I’ve got these things I can control. So really focus on the things you control. And like it says right there, keep the provider-aware things where they fit, where you must, and figure them out.

Okay, so, Achilles heel: addressing. What was it? Achilles’ mother basically dipped him in the River Styx. And if you’ve seen Troy, the movie, Achilles, I think Brad Pitt plays him, absolutely jacked. He’s invincible. He’s taking on full armies on his own. But what did his mother miss?

What she should have done is, when she dipped him in by his heel, she should have just flipped him over and then used his arms and dipped the rest of him in there. Because he had a vulnerability, and that was his heel. That’s a problem. So he died. And the big vulnerability in network engineering a lot of times, this Achilles heel, and it just pains me that we’re in 2026 and this is still a thing, is IPv4 addressing in big, hybrid, multicloud networks. I’ve probably spent more time working and solutioning around problems with IP addressing than maybe anything else. It’s a pain.

I remember, back earlier in my career, and this was almost 20 years ago, I was told you need to buy a book about IPv6, because in a few years you’re not going to have IPv4 anymore. And here I am 20 years later, spending so much time dealing with IP addresses. So there’s a lot of creative ways that you can think about things.

Richard earlier hit on this pretty heavily, and I’m going to hit on it too. Again, this is different per environment, but the thing is organization. And the thing is understanding the workloads, and there’s so much that goes into it. But at the top there, that /12, think of that.

That’s your supernet. And say, okay, I’m giving that whole supernet to all the cloud things in my company. Okay. And then you go down a level. For non-prod, the second octet is even. And for prod, the second octet is odd. Okay.

A little bit of organization. And then when you start getting into the actual cloud networks themselves, VPCs and VNets, the third octet always matches between prod and non-prod. I feel like my father or my grandfather talking about this. What is it? What am I even talking about? But it is a real problem. And the whole point is, hey, we’re in this situation.

For availability and uptime, for our business and for the applications and the services that make our companies money, we have to do a better job of thinking about these small details. And I’m talking to myself when I say that, over the years. That’s where stuff like this came from.

It’s such an important part of the job, making things organized. And you would think, with all the NetBoxes and the Infobloxes, and all those tools are so valuable, but somehow IPv4 is the gift that just keeps on giving. I’m probably going to retire, and this might still be a problem. Who knows?

The next thing, a really big lesson I learned, is, like we said earlier, you’ve got to start at the workload. There are two axes here. One is traffic volume, and the other one is endpoints.

When I say endpoints, what I mean here, by the way, is all the destinations in that public cloud provider you’re trying to reach. So, okay, I have a VPC or a handful of VPCs in this cloud. I have some stuff over here. I have this x number of endpoints.

When you really start thinking about this, you might actually have a combination of these. If you have a big enough cloud estate, you might have some combination of these at some point. In technology, it depends. One size never seems to fit all.

But if you think about a dedicated port, think cloud migrations. You need high sustained throughput, and you’re not multicloud. Maybe you have a few VPCs in AWS, a handful of VPCs or something. Okay.

A dedicated port works well for that, especially if you can wait for the LOA process and the lead time for the cross-connect and all these things. Perfectly fine. And then the site-to-site VPNs. If SLA is not a huge issue, maybe you have a science experiment or something you’re testing, and you don’t need the level of reliability that you’re going to get on a private network, IPsec is great. And then you have the partner-hosted on-ramps, the cloud routers.

This is great for that high sustained throughput when you have a little more complexity. Maybe you don’t have just a few VPCs in one cloud, but you’ve got lots of endpoints and a lot of clouds. So it’s a great fit.

But always start with the workload. You want to know what you’re trying to do.

Security models, this is another big one. And this is a hard one, by the way, because the whole point is the intent. A long, long time ago, we started looking at products, because we were like, hey, we want something that’s just going to do this for us. We don’t want to have to deal with this way of doing it in this cloud and that way of doing it in that cloud. But really, what we came back to is one of these things that we can’t control, but we need. We started looking at intent.

What is our intent? Building an architecture around the intent that could basically be spread across all three clouds, and then going in and learning from our friend Hermes about infrastructure as code. This is when we really started breaking down those silos between, I guess if you want to call it, traditional network engineering and the cloud teams. And this was really valuable.

So at the end of the day, we have three security models, but the intent across all of these models was consistent.

And then we have the blueprint. This one is so important. Say all the way on the left, you have your sites, and maybe it’s four or five data centers. Say you have a company that’s been around for a real long time.

You have some data centers, and you have some point-to-point circuits from your data centers to a colo, maybe, and you’re building out this on-ramp strategy. So everything is terminating at your edge there. And then you have a cloud router with some ports that span all these different cloud providers. When you think about that, one of the beautiful things the cloud providers taught us is about redundancy. When I grew up in network engineering, I was thinking Router A, Router B, these are redundant. If one goes down, I’ve got the other one.

The cloud providers did something interesting. Instead of thinking about redundancy that way, they thought about it in terms of data centers. You have availability zones. That is their measuring stick for high availability, going across multiple data centers. And so for a long time in my career, the North Star was to try to get as close to that model as we possibly could.

Multiple metros. You want everything to truly be redundant. And the reason why is, if you think of Icarus, I think it was his dad, Icarus’s dad said, “Hey, we’re going to escape. I’m going to build you these wings, and it’s going to be feathers and wax. But whatever you do, Icarus, don’t go to the sun. You will surely perish.”

What did Icarus do? He flew straight into the sun. The wax melted, and he fell in the ocean and drowned. So hopefully nobody drowns in a data center. That would be crazy.

But if you think about the availability aspect of this, a lot of times, and this is something I’ve come into in different environments and worked on and re-architected, we think, okay, maybe we have a colo. We have router A, router B, switch A, switch B, maybe a pair of firewalls. Okay, these devices are redundant. But what about everything else? Do we have provider diversity? Maybe the colo has a maintenance window that goes south and something happens. Well, two routers don’t matter at that point. Every link in that chain has got to be highly available.

We don’t want to have the same fate as Icarus. That would be terrible. So another thing: resiliency, picking a failure domain.

This is really important. And this goes back to the speeds and feeds, and focusing on individual components versus everything else. We want to know what our failure mode is. And we want to know, what is our business tolerance for these things?

And a lot of that is something I hated learning about. Once I did learn about it, I understood it so much more, and it made its way into one of my LinkedIn courses. That’s working with DR and understanding all these DR terms, RTO, and all the different things that are important to DR. Those have to grow with all the changes we’re making in the network and that we’re making with cloud. DR is such a critical piece. And so you have to know what the business’s actual tolerance is for these things.

So back to Hermes, platform engineer. When I went from enterprise to the startup space, I was really getting into Terraform and infrastructure as code, because the rate of change in cloud is wild. There’s a lot of changes that happen. I’m going to actually speed through some of this a little bit, because I’m running out of time. But on premises, when you think of automation, routers, switches, you have so many vendors, huge multi-vendor environments.

And automation is a little bit tricky. Hermes would have a heart attack, because not everything has an API. A lot of the automation you do, it’s screen scraping, it’s taking things, it’s massaging the data, it’s trying to normalize the data. And in cloud, because there are more layers of abstraction, they’ve figured out this easier way to do automation.

So, tools like Terraform, keeping track of state, what your desired state is versus what your operational state is. Creating a Transit Gateway, and we talked about Transit Gateway earlier, it’s not that bad. It’s that little resource block. Okay, so just because we create that resource block doesn’t mean everything’s just going to talk. We need to actually attach our VPCs to the Transit Gateway. So we have that Transit Gateway ID.

Every time you create a resource in cloud, it has this long, unique ID. So all we have to do is call whatever we named that ID. In this example, it’s this.id.

And we’re just referencing that. That’s all. And then we have the VPC ID that we actually want to attach to that Transit Gateway. And one thing you’re going to realize is that in cloud you’re creating a lot of resource blocks. But once you do it, everything is normalized so elegantly, and you have modules, and then you’re just worrying about the inputs that you’re putting in, and then you’re running a pipeline.

Now, what about segmentation? Well, with Transit Gateway, by default you have a default route table. I start connecting a bunch of VPCs. Maybe I have non-prod, prod, and shared services, a lot of VPCs. Well, they’re going to be able to talk to each other by default if I just keep everything with the defaults, which is not good. Again, Athena comes into the room and she’s like, “Nope, can’t do this.”

So what do we do about that? Well, we create custom route tables on the Transit Gateway. And it’s easy, because again, you’re just creating resources. So we create this custom route table. We reference the IDs again.

And then we actually have to associate each of those attachments that we created earlier, for each of these segments, back with the resource right there. And also the default route table association and default route table propagation, you want to disable those, because they’re enabled by default. That’s what allows everything to route to everything, and you have an any-to-any. And we know any-to-any in the firewall world, those things get snuck in sometimes for testing, and they just find a way to stay.

So we want to avoid that. This is pretty close to VRFs on premises.

But hey, just because we have all these things created, how do we actually propagate these routes? Say, in this example, that I want prod to be able to talk to shared services. And maybe non-prod can talk to shared services, but I don’t want prod to talk to non-prod. Just a hypothetical example. That would be a great use case for this. So we’re building segmentation.

And the VPC side of this has to be set up as well. All those VPCs that we’ve created, they don’t know how to route through that Transit Gateway now. So I have to reference the VPC ID for the VPC that I want to route through the Transit Gateway. And then I actually have to populate a route.

Because it’s got to be able to route from both sides. So it’s almost static-like routing on steroids. I don’t know how else to describe it. But at least it’s automated. It’s consistent. We set it up. And everything is versioned, like infrastructure as code.

And honestly, the beautiful thing is, what if you could do all this for your on-ramps?

We’re doing so many things manually today in the network world, but this is why companies like Megaport exist. Having that single interaction surface for your provisioning, bringing some of the really great things from these cloud providers, and being able to operate your on-ramp in this way is just immensely valuable. Because Poseidon is getting salty, and he needs help. He doesn’t have time to go figure all this stuff out.

He’s on outage calls because his pager keeps ringing. And this is where Hermes comes in. This actually happened to me at one point. It wasn’t called platform engineering at the time, but I had a developer that came to me, and we worked together.

He took time with me, and he was very patient with me, a long time ago. And I was able to figure some of this stuff out on the cloud side. Now, on-ramp solutions like Megaport and some of the others didn’t exist at the time. But we were onboarding Terraform to the organization, and they were doing things that were affecting routing back on premises.

That’s when I started to have to get involved. So this is all I got. Aphrodite shipped. We have intent. Everything’s tested. And Poseidon got some sleep. He might still be salty, I don’t know. But we have a sound operating model. Everybody wins.

All right. Thank you very much.