EPISODE 1867

[INTRODUCTION]

[0:00:00] Announcer: The modern internet is a vast web of independent networks bound together by billions of routing decisions made every second. It's an architecture so reliable we mostly take it for granted. But behind the scenes, it represents one of humanity's greatest engineering achievements. Today's internet is also dramatically more complex and capable than in its early years. 

Erik Seidel is a network engineer at Cloudflare, where he focuses on automating global network infrastructure. He joins the show to discuss his unique journey into tech, the fundamentals of how the internet works, the border gateway protocol, peering versus transit, Cloudflare's architecture, networking in China, and much more. 

Gregor Vand is a CTO and founder currently working at the intersection of communication, security, and AI, and is based in Singapore. His latest venture, wyntk.ai reimagines what email can be in the AI era. For more on Gregor, find him at vand.hk or on LinkedIn.

[INTERVIEW]

[0:01:18] GV: Hello, and welcome to Software Engineering Daily. My guest today is Erik Seidel. Welcome Erik. 

[0:01:24] ES: Hello. Nice to be here. 

[0:01:26] GV: Yeah, great to have you here, Erik. We've met very briefly a couple of years ago. I say back in Singapore. I'm still in Singapore. You're now in Austin. You were then and still are working for Cloudflare as a network engineer. And we're going to be hearing all about how Cloudflare works as well as just kind of general principles of how the internet works today, which is going to be very interesting. 

But as per usual, we'd love to just hear a little bit about your background before sort of leading up to Cloudflare. I gather you have spent a lot of time in Asia, but also China specifically, which is obviously very interesting from an internet perspective. Yeah, can you just give us a brief history? 

[0:02:05] ES: Yeah, I think like a lot of people in the tech industry, and I found I'm not unique, is I have a rather non - what you might call non-traditional, non-standard background. Very varied background. I came into the tech industry a little later. It's not my first go at tech, or understanding tech, or working with tech. I mean, I've been doing tech-related jobs since I was a student worker back in university in the late 90s, early 2000s. 

And then I kind of drifted off away from computing and internet altogether and towards a lot of language learning. I was classics student, Latin, Greek. I mean, I was training to be a classics teacher at one point actually. And then I ended up going to China for a while teaching English there and learning the Chinese language. Spent a while there learning Chinese because I just really enjoyed a whole thing of learning languages. 

And then after a few years in China, I kind of drifted back towards tech. And within a year of coming back to America, I found in March 2020, about less than a year after I came back to America, I found myself at Cloudflare, and I've been working at Cloudflare ever since. I first kind of started as a customer support engineer at Cloudflare. Very much networking-focused, helping customers with their networking-related issues and how to use Cloudflare networking-related services, in particular, our Magic Transit product. And then from there, I kind of migrated into engineering as a network engineer. And since then, I've actually become a systems engineer on the networking team where I focus more full-time on just automating networks, or automating our network, I should say. 

[0:03:47] GV: I mean, I guess, through that time in China, were you sort of keeping up with tech or doing any kind of network-related things? 

[0:03:55] ES: That's how I developed my interest in networking actually. I started doing a lot of networking-related things in China. I ended up building up my own ASN in China and running it with public IP addresses and stuff while I was in China, and things like that. That's really where I got into networking to begin with. Yeah. 

[0:04:11] GV: Yeah. Because I think most of the audience are probably aware, but the great Chinese firewall. Just networking in China when it comes especially to anything that needs to leave the country or come into the country from a network perspective is just a whole different kind of ball game, right? 

[0:04:26] ES: Yeah. Well, I think we might be getting to this later in the podcast. But, yeah, networking in China works very different from the way it works - in mainland China, I should say. Works very differently from the rest of the world. And that very much influences how we deliver our products in China versus ROW, rest of world, as we say. 

[0:04:46] GV: Got it. Okay. Yeah. And as you say, yeah, we will get to that in more detail later on. Awesome. Let's start kind of with the fundamentals. Kind of we might term it sort of just internet fundamentals and sort of where Cloudflare fits into all this. Yeah, where do we start with this? 

[0:05:03] ES: I mean, at the most basic level, the internet is just a big collection of networks. We've got tons of different networks that make them up. I made previous reference to ASN, autonomous system number. The network is divided into what are called autonomous systems, which are kind of freestanding independent networks in their own right identified by a unique number called ASN, and they all connect to each other. They all peer with each other. And altogether in aggregate, that makes up the internet. 

Now, once you get a little lower, though, what you have is you have different tiers of networks, things like that. At the very top, you have kind of what are like tier one providers. These are very big networks like Telia as an example, GTT is an example, Nippon Telephone and Telegraph, NTT in Japan is a big example, and they are what we call transit provider networks. And they have networks that - though they might have certain focus on certain regions, like NTT might be more APAC-focused, GTT I think might be more EMEA-focused, Europe-focused, that area. Cogent might be more like North America-focused initially. They run networks that span the globe. Big networks that span the globe. And they engage - and they're called tier ones because they are networks that can reach the entire internet without having to pay any money. They basically engage in settlement-free peering with each other. It's like their mutual benefit. Yeah, we all connect to each other, exchange our customers' data with each other. And by connecting to each other and exchanging our customers' data with each other, that's the internet. 

And in addition, you have all sorts of other networks. You have smaller regional ISPs that connect to one or more tier one transit providers. They might have downstream customers of them too. Their customers connect to them. That then connect to the transit providers. Then you have tech companies like Cloudflare is an example of one. Google, AWS are examples of others. We're big content provider networks. We connect to tier one networks, too, for transit. In addition, we do lots of interconnections with all sorts of other networks on the internet to broaden our connection base and reduce the latency in and just reduce the general cost and increase the efficiency of delivery to customers. 

[0:07:20] GV: And so we've talked about tier one. And then if we, I guess, look at edge routers, edge routing, they, I believe, use the BGP routing protocol. Could you maybe just - what even is that, and how does that sort of then fit into things? 

[0:07:34] ES: I find the best way to explain BGP with a lot of things is to start with the problem that it solves. What's the problem that it solves? You mentioned edge routers. We have all these networks. These networks have the concept of an edge and a core. The core of the network is like it's all their network, like deep in the bowels of their network. If you're an AT&T customer in America like I am, I'm deep in the bowels of their network. My local internet connection connects to something deep in their core. 

The edge or the border, we'll call it the border as well, the edge of the network or the border, that's where it's actually connecting to other networks, where traffic is leaving and entering their network from other networks. And the routers that do that, we call them edge routers. Now, when we have that, we have these networks connecting - their edge routers are connecting to routers on other networks. They have to basically provide a roadmap to the other routers. 

We all know the internet is made up of IP addresses and IP prefixes. Even like the most basic examples, 192.168.1.0/24. That's, of course, a private IP network, but it works. Public IP networks are numbered and work the same way in that respect. With their public networks, they have all these public networks like your public IP address, like my public AT&T IP address. It is part of an AT&T prefix. 

Now, how do other networks know that I'm an AT&T? How do they know? Like, "Yeah, I have to send my traffic to AT&T. Not only AT&T, but maybe it's best to send it to this AT&T router in this region." How do they know that? That's where BGP comes in. BGP, it stands for border gateway protocol. It runs on these edge routers. And what basically do is you establish BGP - these edge routers, like AT&T edge routers, Cloudflare edge routers, whatever they might be, if they have a direct connection to each other. If the AT&T router has, say, a direct connection to a Cloudflare edge router, a direct point-to-point fiber optic ethernet circuit, then these two networks will run a BGP session over that point-to-point connection, and they will exchange their prefixes. 

In the Cloudflare case, these are our prefixes that you can reach here to the AT&T router. And the AT&T will say these are all of our prefixes, that you can reach these prefixes through me, through that same session. And that kind of provides the roadmap. And it's made for all of these point-to-points. We generally have one - with the exception of IXP sessions. IXPs, where we have like multiple sessions. Each point-to-point, it's one BGP session. And that's what BGP. Basically, it's building the roadmap of the internet. It's telling everyone where the IP addresses are. 

[0:10:21] GV: Got it. And this might be quite a basic question for some of our listeners, but not for me anyway. How often do this change? Because if this protocol is all about exchanging the road map, yeah, how often do they change? Do they change? 

[0:10:33] ES: You have to look at it in terms of individual sessions versus aggregate. Individual sessions, maybe not much. All in all, aggregate, it's changing all the time. The internet is - the IPv4 internet right now has, I think, over a million unique prefixes. The IPv6 internet has, I think, about 200,000 unique prefixes. That's changing all the time in aggregate. And it also can depend how far up the chain you are. If you're a small regional network, maybe you're a customer of - say, you're a small office network, maybe, and you're a customer of AT&T, and you've got a BGP session with AT&T. Or not even AT&T, just your regional ISP. Your session with your regional ISP might not change that much. It might be very stable. 

Once you get to your regional ISP, the sessions with all of their customers will be very stable, but some of their customers may be changing, which means their sessions with their upstreams might be a little less stable, a little more dynamic. Because every time one of their customers makes a change, they make a change. It gets propagated to their upstream, and their upstream sessions make a change. 

Once you get to the tier ones - Telia and AT&T is a tier one, too. Telia, GTT, NTT, all these, they're having a lot of changes. It's very dynamic. Because anytime - not just one of their customers. Anytime a customer of a customer of a customer of a customer makes a change, that'll get propagated up to one of their sessions. They've got sessions where like, yeah, there's lots of changes. It can be very dynamic. It really depends where you are on the internet. The closer you get to the tier one, the heart of the internet, if you will, and what we call the default free zone, the DFZ, it gets more dynamic. 

[0:12:15] GV: So many questions I could be asking, but I know we're always going to run out of time if I ask everything. But one sidebar question here is tier one. AT&T sounds like the main US player. Does that mean that they're just sitting on a massive asset that at some point someone else could come and buy? Could another tier one be created in some way or - 

[0:12:33] ES: The basic standard of tier one, from at least the way I understand it, and I think that's the standard way, is a tier one is a network that it can reach anywhere on the internet without having to pay another network. 

[0:12:45] GV: Okay. That's a good definition. 

[0:12:47] ES: Yeah, that's basically what it is. And AT&T can reach, from what I understand, if they're still tier one, I understand they are, they can reach any other part of the internet without paying. And the reason they can do that is because they have such a large network, such a large customer base. And I'm not really the finances person or the business decisions person to this, but it's reached a point where like, yeah, other networks, they've all made the business decision that this is a network that it's worth doing settlement-free peering with. 

[0:13:13] GV: Okay. Well, that kind of actually leads us pretty nicely then into peering versus transit. And you did touch on it briefly earlier. But, yeah, let's talk about that. And, obviously, we're going to get into what that means from a basic financial standpoint as well, I guess. 

[0:13:29] ES: We have our edge routers. They're connected with lots of other networks with fiber optic, ethernets, and those BGP sessions I mentioned, a lot of them are peers. We do settlement-free peering with them. What they do is we send them our prefixes. They send us their prefixes and maybe the prefixes of their direct customers, which means we can use that connection to reach their network and other networks that are their customers, maybe, right? 

What we cannot do through such a session? What if there's some other network out there that's not their customer at all? It's a customer of some other network. We can't reach it through them. They're only going to provide us with the reachability to their network and their customer networks. If, say, we've got some other network way out there and we're not directly connected to them, and that other network is not a customer of any of our peer networks, how do we reach them? That's where transit comes in. 

Basically, when you're buying a transit session, buying a transit service, you are buying a connection to the entire internet. You pay them, and they give you what we call a full table or a full view of the internet. They won't just send you maybe a thousand prefixes of just their network and their customers. They will send you a million IPv4 prefixes containing every other network on the internet. They will send you all 200,000-plus IPv6 prefixes of the whole internet. And they've given you that full view of the network's internet. So you can reach any other network on the internet through them if you so choose.

[0:15:01] GV: And I guess in Cloudflare's context, this is sort of table stakes because Cloudflare, to provide - we can maybe do a very brief history of Cloudflare in a second. But the product, provided initially, and then a massive suite of products now, Cloudflare has to assume it can know where to go across the entire internet to be able to provide its services effectively. 

[0:15:24] ES: Yeah, we have to make sure all of our data centers basic requirements. Yeah, we have to have a fully functional connection to the internet that has a full view of the internet. 

[0:15:32] GV: Yeah. Let's sort of move on to the global scale, the architecture of Cloudflare. I mean, I definitely didn't prime you to be a history expert on Cloudflare today. But maybe you could just from any basic information, where did Cloudflare start? Where is it kind of now, maybe sort of from a product perspective, or just product suite, I guess? And then we can talk more about how that's actually then manifested at a technical level through architecture. 

[0:15:57] ES: Sure. I'll focus on the part of Cloudflare I know the best, based on not only my networking team, but previous and customer support. The aspect I'm most familiar with for Cloudflare, I think, a lot is the security aspect. We provide a lot of security services. And I guess you could say the classic way we do it, which is kind of the OG way of doing it, Cloudflare-wise, is we provide like the CDN edge where basically we sit between our customers' origin servers, like their web servers, our edge servers are between them. And their customers will not directly connect to their web servers. They'll connect to our edge servers. And then we might provide services like caching. We'll cache some of that for them. 

We provide a lot of security services for them to make sure, like, yeah, if there's a DDoS attack not, only do we strive to absorb it, we strive to filter it as well as much bad traffic as possible, drop it as much good traffic as possible. Make sure the customer gets, or make sure it's served by cache or whatever means. That's the OG. Like I said, I worked a lot in customer support on a product called Magic Transit, which is kind of expanding that philosophy to layer three, which is basically networks with their own ASN, like I mentioned. They can put themselves behind our metals where we basically establish using PNI, or using GRE tunnels, or some other means to like connect from our edge servers to their edge routers. 

And we sit in front of them. And before any of their traffic layer 3 traffic reaches them, it goes through our metals first, and our metals filter it, and then send it to them. That's another product. And that's kind of where I came to Cloudflare. Yeah at least from the perspective I see in Cloudfare, we are very much a security-oriented company providing security services to our customers in addition to CDN services and even edge compute services as well. And speaking of edge compute, in addition to our workers' product, we're moving more and more into the AI realm as well. 

[0:17:56] GV: Yeah. Let's talk about then the sort of pure infrastructure that's required to drive this. I believe just a couple of numbers off the top of the charts, if you like, as over - I mean, this was what? 2 years ago at least. Was over 300 global points of presence. And I think it' be interesting to understand what those are in over 100 countries. 44 million HTTP requests a second on average. These are just sort of numbers to help the audience maybe just get a vague sense of scale here. And again, this is 2 years ago. So it's only probably gone up since then. 

[0:18:26] ES: We've gotten bigger. I'm sorry. I can't provide you with the exact numbers now. 

[0:18:28] GV: No, no, it's fine. 

[0:18:30] ES: We're getting bigger. It gets bigger all the time. 

[0:18:32] GV: Yeah. Let's talk about point of presence. What does that even kind of mean? And what are they handling - 

[0:18:38] ES: That goes back to that edge networking concept. We have these networks, they want to connect to each other, right? Where do they connect to each other? The point of presence, it's like a data center someplace where, like, yeah, we've got routers there. We've got infrastructure there. The other network has routers there. They have infrastructure there. We can connect to each other there. And it's not just case of points of presence. With us, it's not just edge routers as well. We'll have our fleet. We'll have some of our servers as well there. The thing about our servers, the way we've got them configured, is we like to have them configured generally to run the whole Cloudflare stack. Basically, any of our Cloudflare services should be able to handle any of our customers' needs and services. 

[0:19:18] GV: Got it. How does this then like flow through to data center, I guess, design, if you like? I believe it's sort of the concept idea of traditional versus multicolo POP or MCP, which is a different MCP to what maybe a lot of our audience are used to hearing about. Yeah, let's talk about those. 

[0:19:36] ES: Yeah. The way I like to conceptualize it to understand how we approach, is you kind of get an idea of the internet topology itself, is just with human topology. You've got areas where population density is very low, relatively low, like rural areas. And then you have areas more urban where population density is very high. The internet interconnectivity on the edge is kind of like that as well. You have areas like Austin as an example, even though it's a big urban area, where there's not as much interconnectivity happening, right? 

Not many networks connect to each other, peer with each other in Austin. Where a lot of the peering, a lot of the internet connectivity happens closest to me is the DFW, the Dallas Fort Worth area. There you have lots of networks coming together, lots of data centers in that area all connecting to each other and peering with each other. Now, the internet almost works the same with the city metaphor. If you have all these networks connecting to each other, guess what? You've got lots of companies, including us, including all sorts of other companies. They want to park their servers and infrastructure there. They want to park their infrastructure close to where the connections are happening. Because the closer you are to where you're connecting to other networks, the quicker it is to get your data onto the customer network and to the customer. 

If you're in some area where it's like you're far away from where interconnectivity is happening, then you end up at the core of your provider network. You have to go through your provider network before you reach the customer's network. Whereas at the edge, yeah, it's easier to just immediately start going to the customer network. And that's why data centers and points of presence will follow those contours. Like DFW, lots of data centers, lots of internet connectivity. We have a big point of presence there with lots of servers there. One of our biggest IAD, Ashburn in Northern Virginia. That's another huge area where there's just tons of network interconnectivity which leads to tons of data centers, tons of infrastructure. Everyone parked where all the networks are meeting each other to get to the customers quicker. 

[0:21:42] GV: I mean, I seem to remember this from a talk you gave a couple of years ago. This was a bit of an eye-opening moment for me to sort of understand, I think, why we were in Singapore at that point in time, and I still am in Singapore, is a listed region, for example, when you go to AWS or GCP. And I think I was always wondering why all the regions are effectively kind of the same across different products and across different providers. They can't always offer all the regions, if you want to call it that. But a lot of them do still follow Southeast 1 is Singapore, and Southeast 1 is the same in GCP, AWS, etc. And this is kind of I guess what you're explaining here, which is same building, pretty much. Just where all the providers kind of have their metals. 

[0:22:25] ES: Same building. Or if not, same building close enough that you can get a metropolitan network connectivity in the form of dark fiber or waves connections relatively inexpensively. 

[0:22:37] GV: Yeah. And it does still require agreements between everyone to kind of say, like, "I'll connect to you, you connect to me." And that kind of all helps everybody.

[0:22:44] ES: Yeah, it's still based on that basic system. And it comes together where this happens. It's most efficient if there are certain points on a map where it all happens. 

[0:22:52] GV: Yeah. I guess if we then bring back in the edge router aspect to this, which is that edge routers are, just so I'm getting this clear, not the opposite of a data center, but they are the bits that do sit outside of these sort of mass transit areas. Is that correct? 

[0:23:09] ES: Well, I mean, they're the ones that are actually doing the connectivity. We try to get them close to the most convenient places to connect them to as many other networks as possible. 

[0:23:17] GV: Okay. And just in terms of like redundancy, we did actually have an episode a couple of months ago just talking about subc cables, and just how when they get cut, and what can cut or damage. And we were focusing a bit more on how they actually get repaired. We had someone on from one of the big news outlets who'd gone and been on one of these boats and just seen what workers needed to actually repair these fiber optic cables. But either those or any other kind of problems, I guess, because we're still talking about a lot of cabling for most of this. How do you factor in redundancy? And how to even think about that? 

[0:23:52] ES: Yeah, I mean you pointed out a tricky question. Subc cables can be tricky because they carry a ton of capacity. There's only so many of them. And when they get knocked out, it can take a long time sometimes to fix them because they're under the ocean. You have to go out there with boats and things like that to fix them. We do run a global backbone network. And the thing about it, it does span the globe. It circumscribes the entire globe. 

Basically, we try to make sure we have a lot of diversity. We have backbone links going through multiple different subc cable networks. And then we have lots of transit providers as well, too. If we lose some backbone capacity because of a subc cable deal problem, we can move to transit. Where the trickiness comes in is sometimes with some of these bigger subc cables, not only does it impact us, not only does it take us out, take out some of our capacity, it takes out some of the capacity of our transit providers because they're running over the same subc cable. 

Other big networks as well, content provider networks and stuff, where some of our customers are based, they get hit by the same thing. Because subc cables has so many different networks running through them, it can be really tricky. And it does at times lead to real capacity issues and difficulties that impact the whole internet, impact lots of networks, and really are felt until they're fully repaired. 

[0:25:15] GV: Yeah. Let's move on to sort of, I guess, mini case study just around China specifically. We touched on this in the beginning that you had already spent a lot of time there. But I don't think you ever worked for Cloudflare in China. But, obviously, just that knowledge that you could probably bring to the table in that respect. But how does the China network part of Cloudflare work? Because it has to be quite different. 

[0:25:37] ES: Again, starts with a problem. The way the networking normally works here, the way we handle it, is it's not something that really knows national boundaries. By that, I mean, we operate a global network. And not just our edge colors, but our backbone network. When I talk about our backbone network, like the big connections we have between our own data centers, our own network. We operate this big global backbone network. And we're not the only ones. All those transit providers, those tier ones I mentioned, they all operate those big global networks. 

And other content providers like AWS, Google, etc., they also operate big globe-spanning networks. And when we manage these networks and when we move traffic through them, our general approach is unless there's special exception, and those special exceptions are few and far between, we're not thinking about national boundaries. What we've got is a whole system with lots of different paths we can choose to send our data. Like, okay, we've got a path. We've got data center A and data center D, right? And we have a path via data center B or data center C to go from data center A to D. 

But, say, the path via data center B or point of presence B, whatever you want to call it, starts to congest, then we might shed some traffic, move it to the path via data center C. If you notice, we're not really thinking about, "Oh, what country is it in? What countries are passing through?" No. We're just thinking like, "Yeah, we've got this globe-spanning network. What's the best path?" It's all treated basically equally without concern for national boundaries, unless there's some special case about how data has to be treated. 

China is a different story. Mainland China, I should say. Hong Kong itself still is very much - it's connected to the rest of the world network, where we've got data centers there. We've got our backbone running through there, and same with other networks. But mainland China is, one, you've got three major networks there. You've got China Telecom, China Unicom, China Mobile. And if you want to get in and out of China, unless it's a special case, you're going through their networks. Like you're not running your own network in there. You're not running your own backbone network there. You're not transiting your own traffic through there or anything like that. You're going through those big three networks. And that in itself creates a special case because, again, we've moved away from just operating our own globe-spanning network that grows into China. Now we're stopping and saying, "Okay, at this point, we kind of have to go into the Chinese provider networks." And that doesn't just apply for us. Of course, that applies generally. 

And then, of course, you do have the great firewall of China, I think they call it. That involves a lot of packet inspection. Now, the thing you have to understand about our edge routers is, at a basic level, they're kind of dumb. They're not very smart thing. They're not very complicated things. They basically - they learn each other's prefixes. Then they install it onto a hardware route table, a hardware A6. And basically, their only job is they're forwarders. They get an IP packet. They see the destination IP address. Or in the case of - in the backbone network, when we've already got it encapsulated in MPLS, they see the MPLS label. And based on that IP address or MPLS label, they say, "Okay, I need to send it to this next top router." 

And that's all it's doing, right? There's no real packet inspection there. It's just kind of mindlessly forwarding traffic to wherever it's told, programmed to forward the traffic. But when you've got something like the great firewall of China, you're adding a lot of overhead to that. All of a sudden, it's more at the Chinese edge or near the Chinese edge. It's more than just mindlessly forwarding packets to wherever, not caring what - no, you're starting to look what's inside the packets. What's inside the packets? What protocols are the packets? What's maybe the DNS address the packet was originally destined for? Things like that. And then that in itself adds so much more overhead. Because whereas we can just like do more with less, so to speak, because we're not doing the filtering and inspecting, they're doing less with more because they need so much infrastructure just to do that. 

And the result of that together, the kind of big three - always having to go to through those big three networks, plus the filtering and inspection they do on the edge, leads to a situation where, if you're in China, and I've experienced this for myself, connecting to the outside internet, the rest of the world, it's not the smoothest experience. When I'm in America and I connect to stuff in Europe, I might feel a little bit of latency. But generally, it's okay. 

And when you're in China going through that, even when the great firewall of China's not blocking it or anything like that, even when they say, "This is fine. We don't want to filter that." Even then, it can kind of like be a lot of loss, a lot of connectivity issues because of overloads and things like that. And it can be generally you know not the most enjoyable experience. Now that means we've got lots of customers who are global companies. They use Cloudflare to serve their customers all over the world. And Cloudflare works great for them, right? 

But then the exception is, well, we've got these customers in China. Our stuff, the Cloudflare edge, is outside of China. Our Chinese users are having a bad experience of having to get out of China. you know, having to send their traffic outside of China to connect to us. And the answer to that we give them, "Okay, we're going to put Cloudflare infrastructure in China." Now, when you put Cloudflare infrastructure in China, the way we do it, and I think it's the fairly standard way, is you partner with local providers, local networks. And we have our local partner, and we basically host our infrastructure on their network. And I should add at this point, yeah, I'm on the network engineering team now. I don't really do anything with China network because we do not manage the network. We've got Cloudflare servers and stuff there. And our SRE team, for example, will work with our partner SREs to solve any problems with the Cloudflare stack. But when it comes to the actual networking stuff, yeah, that's not ours. That is our partners' network and our partners' networking team handling it for us. And they have the China-specific experience to know how to handle that well. 

[0:32:24] GV: And just to clarify, I mean, this is anything - I think, to my understanding, sort of that does run, I guess, on this partnership model. It's really just an enterprise product. This is not something that your average developer would have any interaction with. 

[0:32:38] ES: Yeah, we do offer our free tier services. You can use Cloudflare as a free user. And we really get a lot from our free users. It's like a real big part of our network. But not China network. China network is an enterprise product. You have to basically buy it and sign a contract for that. Yeah, we're dealing with like enterprise people who can afford enterprise services. 

[0:33:01] GV: Yeah, exactly. Where sort of just the need is absolutely there. I mean, likewise, I've had a lot of - probably almost over 10 years ago at this point. Yeah, a lot of experience with - as you say, it's not even just the great China firewall, but it's just the sort of latency involved with the in and the outs. Yeah, which was just painful to deal with. Yeah. 

Anyway, I think that's been a great understanding of what goes on there. I think it's really great throughout this whole episode. You're always framing it as, well, what's the problem? Before we're even talking about what do we do? I think this is a problem that most developers are at least aware of, which is DDoS. And I'm sure most developers have either experienced it. I can't access a website. Or literally had it to happen to their own service. Or at the very least, read about it as to why something went "down". It's a big kind of - Cloudflare, quite frankly, handles so many of these attacks these days. And I think I was almost sort of analogizing it in my head to Cloudflare is a bit of a government service at this point in terms of if Cloudflare went away, I think we would have a much worse internet, at least in the western world, if you want to call it that. Let's talk about how does Cloudflare mitigate DDoS? And I believe we're going to talk about anycast? And I remember you saying anycast in the presentation. I'd love to go back there. 

[0:34:19] ES: Sure. First let's talk about a little more what a DDoS attack looks like. 

[0:34:24] GV: Right. 

[0:34:24] ES: Now, DDoS attacks are normally launched by what are called botnets. Botnets are basically like malware. A lot of the malware we run into is like viruses that, to a certain or lesser extent, take over your computer. Or maybe not even your computer. It can even be like your mobile phone, or even maybe it's something as dumb as like a smart fridge, or something like that. Whatever they can - if it's connected to the internet and it has a CPU on it, it has a computer on it, yeah, there's a chance it can become part of a botnet. 

Basically, the malware implants itself in your computer, or whatever thing it might be. And it'll usually try to do it without the user, if it's on your phone or whatever without you even being aware of it, because it doesn't want you removing it, of course. And the software, the malicious software maintains a connection with a command and control. And then whoever owns that botnet, so to speak, they can use that command and control to order maybe I think up to hundreds of thousands of device. It might be closer to millions. Now, I'm not sure what the exact numbers are anymore. But I think, last I heard, in the hundreds of thousands. Hundreds of thousands of compromised devices spread throughout the whole world. And each of those devices can just use whatever internet connection. It has to just send whatever it can to target this IP address. Just send garbage, some attack traffic to this IP address. And it sends in all these compromised systems, over 100,000 maybe. Just at the same time, send a bunch of attack traffic to the same IP address. 

Now, in aggregate, that can be overwhelming. You're talking like terabits per second of multiple - I think I've seen over a 100 ter - I think, yeah, maybe 100 terabits per second, or more than that, of traffic going at a certain IP address. Now this is where anycast comes in. The idea with anycast in our network is anycast is, traditionally, with unicast, one IP address for one computer, right? There's only one computer in the world that has that IP address. 

Anycast is different. And the idea behind anycast is we've got thousands. In the case of Cloudflare, our fleet, well over 10,000 servers spread throughout the world throughout hundreds of data centers that each have that IP address in data centers that are each advertising that same prefix to all of our peers. Which means, when you do that, when you have over 10,000 computers, servers spread out through the entire world behind hundreds of edge routers, each with dozens, maybe hundreds of peer networks, and we're all advertising that prefix out and that IP address out, means that when this botnet attacks, its strength, its distributed network kind of almost in a way becomes a weakness. Because the way the internet works, it's all spread out. It's not aggregated. It's not hierarchical. It's just best path. Whatever is the best path. All of these hundreds of thousands of devices will be taking different paths, ending up at different Cloudflare data centers, depending on where they are, and ending up on different Cloudflare metals. 

And whereas, like, "Oh one data center getting 100 terabits per second of traffic," or not even that because, because it overloads. No. What you get is hundreds of data centers maybe getting under a terabit a second of traffic. Maybe not quite that even. That's ideal. Some bigger data centers might be getting a few terabits, and some might be getting significantly less. And then lots of metals just getting in the order of gigabits per second of traffic. We've basically, with anycast, taken advantage of the architecture of the internet, just the topology of the internet, to kind of naturally disaggregate and unfocus that attack. So it just spreads out into smaller bite-sized chunks that we can absorb. And that's key. Because as an anti-DDos network, if we want to be able to filter our customers' traffic and make sure they get all the good traffic we can give them and as little of the bad traffic as we can, we ourselves need to be able to absorb that attack. 

If we can't absorb it, if our network is getting overloaded, then we can't filter it for our customer then because we're already losing it. But we've gotten to the point where I've done network edge on call, where if we get congested, we get a page, right? And we have to fix it. I've been on call before where we've had a 100-terabit per second attacks, and I never get a page. I didn't even know it happened. Because we were able to just absorb it without having any congestion events at any data center. There have been other times where maybe it's just like we see one link, one PNI congest, and like, "Well, what's going on here?" And I'll put aside that engine that sometimes you like, "Let's back out. Look at the forest." When it looks weird, I'm like okay, "Let's look at the forest for the trees." Look at our global DDoS. And I'm like, "Oh, wait. This is a big attack. Okay, we're absorbing it just fine everywhere except this one link somewhere." And that's the only reason I knew it happened at all. 

[0:39:54] GV: Yeah, that's fascinating. Just to read some of that back is I'm almost thinking like water is quite a good analogy here. In the sense, if you were to pour in water in the top of a container, and this container still has a sort of out at the end of it, and out would be where the water is trying to get to, it's trying to get to a - what I'm getting at here is like that's the site that somebody thinks they're trying to target. The problem is they've got all these rivers in the container, and these rivers do - a lot of them actually end up - they flow through. Effectively, what we're talking about is Cloudflare architecture. And so the water is just dispersed across all these different rivers before it even has a chance of getting to the end point. 

And so, I mean, let's talk about - because that sort of is like DDoS filtering pipeline effectively. Just sort of, I guess, briefly, what kind of happens then during that process of the data is naturally - it's hitting this one address effectively from the anycast principle. But it's then like filtering through and being filtered. Are we talking is it the same filtering process that goes on across all the data? Or is there any kind of differences, I guess? Yeah. 

[0:40:58] ES: I mean, the other side of the anycast, in order for anycast to work, you basically have all - if all of our, say, 10,000-plus servers have that IP address assigned to them and are handling requests destined for that IP address, they basically have to be configured identically. They have to be running the same services. Like, www.acme.com is our customer, and it has to be served from all of those servers. It has to be the same. You don't want an experience where like, "Oh, but depending on what server, you get a different website." You definitely don't want that. Unless the customer themselves does something special to configure that. 

And some customers will create regional specific things and like that. But basically, the idea is, yeah, we need a system that can propagate all of this out globally and keep all of those servers in sync and make sure they're serving the same thing. 

[0:41:51] GV: Yeah. Looking ahead, IPv6 is obviously a sort of big transition that's going to be sort of in the works and will be, I guess, propagated over the next few months, etc. How is that affecting how any of this works? How is, just maybe at a general level, your day-to-day? How are you having to sort of think about that transition? 

[0:42:12] ES: I mean, we're well into the transition. I mean, from our perspective, I wouldn't even consider us still in the process of transitioning. We run a dual stack network, IPv4, IPv6, and they're aligned. Our IPv6 network follows the same topology as our IPv4. And in many ways, for a lot of our internal services, we're already IPv6 first. 

IPv4, we support a lot of our customers - a lot of eyeballs are still on IPv4 network. We serve them with IPv4. We still have plenty of customers who their infrastructure is IPv4. We connect to them with IPv4. I generally find from my experience, a lot of the DDoS attacks still seem to be - at least last I check, last I handled a major DDoS attack was, I guess, six, seven months ago. But they do seem to be more IPv4 heavy than IPv6. But internally, we're fully dual-stack. We're fully go on IPv6.

[0:43:12] GV: Got it. And going back to DOS just for a second just because you sort of mentioned the latest - you at least had to handle. But I did see Cloudflare puts out a very good sort of - I believe it's quarterly. No, I think it's like yearly, maybe. But you obviously talk about it in quarters in terms of the DDoS attacks that have been witnessed by Cloudflare. And basically, Q2 this year was quite a way up on last year. And Q1 this year had an exceptional number. Do you have any sort of, I guess - I mean, there's some insights in that article. But I don't know. Just as someone who then actually is dealing with it, do you have any insights as to why this is just only increasing, I guess? 

[0:43:48] ES: I mean, I don't have any particular insights. I guess it just seems to be following - there's the same general pattern that, yeah, they've been getting bigger and bigger as time goes by. As more devices are connected to the internet, there's just more devices that can be compromised. 

[0:44:06] GV: Yeah. And interestingly, I believe in that blog post, one of the people that had experienced this, it was something like 63% of the respondents believed it was actually competitor that was initiating. I mean, they talked about specific industries like gaming, gambling. 

[0:44:20] ES: That's the thing. Yeah, gaming is a thing. And that's not a new thing, by the way. Back when I was at customer support, there were cases where like DDoS attacks and the suspicion was - I mean, I'm not in a position to confirm or deny. But the suspicion was that some competitor hired a botnet herder. Bot herder, I think, is one slang for them. But hired out a botnet to launch an attack on them. And that's one reason we provide one product called Spectrum, which is it's a CDN - it's an edge network product. It's a proxy, but it's not an HTTP, HTTPS proxy. It's a generic proxy for any arbitrary layer 4 protocol on TCP or UDP. 

And one use case for that, one problem that is fixing, is customers who are like gaming networks, and they want to put their game servers behind Cloudflare. And, oh, well, these game server protocols are not HTTP. Well, that's where Spectrum comes in. And whatever their gaming protocol is, they can use Spectrum to proxy, and that kind of help protect that from that kind of, yeah, attacking game networks phenomenon. 

[0:45:33] GV: Yeah. I mean, for anyone interested, that is just double checked. Yeah, it's a quarterly report. It's just Cloudflare's 2025 Q2 DDoS threat report. And that's on the Cloudflare blog as well, which is quite a great read as well. 

Erik, great to have you on. I often ask this question to guests just before we head out, which is a pretty simple question, but I always get a range of answers, which is what do you know now that you might have if you could tell Erik of, I don't know, coming out of college and going off to China, for example? What do you know now that you might tell that person? I think it's interesting to hear especially, because, as you say, you kind of left tech for a while, but you obviously kept quite interested in it in the network side. And then you sort have come back to tech in a big way. But yeah, what would you tell yourself? 

[0:46:22] ES: One thing I've learned that I wish I knew when I was younger is when you're in an engineering role, regardless of what company you work for, it's not like a good employer, bad employer thing, or like that, or regardless of what industry you're in, burnout is a real thing. That's something when I was young I kind of took like a sort of a blasé attitude to burn - I didn't really take it seriously. I'm young, I'm strong. I can do this. I can do lots of all-nighters. Well, yeah, by the end of - before I even was able to enter the industry, right out of university, I already kind of burnt myself out. And I spent some time doing studying classics and ou doing the Latin, Greek thing. And time in China learning Chinese thing. It's only much later that I came back to tech. 

And by the time I came back to tech, I learned like, yeah, you need work-life balance. The thing I've learned now is to have a much better work-life balance. I'm much hard - I'm still very hard worker, still push really hard. But the old days where it was like so much unbalanced towards 100% tech 100% of the time. Yeah, looking back, I think I'm glad I've lived the life the way I have. I really enjoyed all my time not in tech and learning all those other things. But had I done it again, I probably would have taken a more work-life balanced approach just from the get-go and not have gone through that odyssey of burnout and then kind of recovery and working my way back to tech from my excursion, my travels outside of tech. 

[0:48:06] GV: I think that's a great call out. And the burnout thing is - as you say, it is real. And I would also agree, just as you get older, you - one, at least, I think, starts to understand it more for you. Because burnout can mean slightly different things to different people in terms of it's not just, "Oh, are you working 12-hour days?" 12 to 18-hour days or whatever. It is actually often about sort of so many other factors that play into it as well. Yeah, I love that answer as well. I think it's great advice for many people. 

Yeah, I mean, fun fact, my grandfather was a Latin teacher actually. I'm also quite into languages. I'm terrible at most of them, but I love just trying to learn them. Try to learn a bit of Cantonese, which is a very tonal language, which is very difficult. 

[0:48:50] ES: That's on the hard end of the Chinese language.

[0:48:52] GV: That is on the hard end. Yeah. Yeah. No shock that I didn't exactly become fluent, but I was able to have myself understood. That's always a fun one. Well, thank you, Erik, so much. We've learned a ton today. And anywhere people can find you, whether it's LinkedIn or anything like that you want to talk about? 

[0:49:11] ES: Sure. I mean, I just got a LinkedIn at just Erik J. Seidel. I think it's just LinkedIn/erikjseidel. Erik J. Seidel.

[0:49:25] GV: Okay, awesome. Well, again, thanks so much for coming on. And if I'm in Austin, I'll come say hi as well. 

[0:49:31] ES: I'd be glad to have you here. Show you around. 

[0:49:34] GV: Thanks so much.

[END]