Anna Claiborne, SVP of Software Engineering at PacketFabric on the Exponential Evolution of Networking on a Global Scale
Welcome to the tech deep dive podcast where we let our inner nerd come out and have fun getting into the weeds on all things tech. At Clark Sys, we believe tech should make your life better, searching Google is a waste of time, and the right vendor is often one you haven't heard of before. Hi. I'm Max Clark, and I'm talking with Anna Claiborne, who is the cofounder and SVP of product and engineering for PacketFabric. Anna, thanks for joining.
Anna:Hi, Max. Thanks for having me.
Max:For the people that don't know Packet Fabric yet or haven't listened to the other podcast with Jezebel, you know, at a high level, how would you describe what what Packet Fabric is? Or actually, let me restate that. Like like, what was the impetus to create Packet Fabric, and then how did that influence what Packet Fabric is?
Anna:So the impetus to create Packet Fabric came from a lot of it came from the cloud revolution, right, that we saw. At least that most of us in the tech world saw. So we went from this old archaic model of having to rack and stack servers, you know, configure each and every one or, you know, ZTPM, and then sit there and give them all their special care and feeding to just clicking a button or calling an API and provisioning a 1,000 servers in the cloud. And, you know, that's great. It helped really advance a lot of things technologically, helped spur this whole new revival and, like, app development, but that same thing never happened for the network.
Anna:Right? And there's 2 sort of crucial pieces of infrastructure. There's compute and there's network at the most basic level. And I'm including storage and compute there, you know, just 2 big buckets, you know, the compute network. And so compute butts ahead in, you know, these massive leaps and bounds as far as, you know, what you could do with it and, how you could automate it and how easy it was to provision it, and network didn't change, didn't change, didn't change.
Anna:You know, provisioning a 100 gig point to point is still exactly the same as provisioning a frame relay circuit back in 1996. Like, absolutely nothing changed with that. And we realized that fundamentally, that's a huge problem for everybody. Right? Because you can only advances in technology can only move as fast as infrastructure advances.
Anna:Right? And as fast as that is as that gets easier and better because if you're waiting, months to get more bandwidth or more compute, your innovation is gonna stagnate. That was the driving force behind it.
Max:I mean, one of the big things you touched on is anybody who has ever configured or or ordered a circuit from a from a carrier, it go through a process. Right? Even if you're in a carrier neutral data center, that is a process of pain. Right? I mean, I don't I don't feel like it's it's, you know, so
Anna:A gauntlet, really.
Max:A gauntlet. Right? I mean, I was in a lot of forums where one of the big ones that came out was, you know, question to the carrier. Could you just automate the LOA generation process for me? You know, if I if I order the circuit from you, can you just automatically send me the LOA so I can send it to, you know, Equinix and order the cross connect without having to involve more people in the process.
Max:Right?
Anna:Yep. But
Max:packet fabric is fundamentally more than just how do you work through this pain of provisioning these initial circuits. Because, you know, I mean, if I'm ordering an Internet circuit, it's like, okay. It's a lot it's a lot of pain to get it, but then it's just there. Or if I have 2 facilities I wanna link, it's a lot of pain to get that up, but then it's just there, you know, and and we can make modifications to it. You you took a a pretty big next leap on this.
Max:And this is more than also just cloud. I mean, you you guys, I mean, it's it's it's network as utility, it's network as code. I mean, it's it really is something a little different here.
Anna:Yeah. And that's, I mean, a lot of that you can sort of trace back to how we, you know, how we think about building our product is that we think about it in terms of basic building blocks. Right? You have ports and you have VCs. And then on top of that, what else can you build?
Anna:Well, you can build point to point circuits, you know, from a to b. You can build connectivity to cloud. You can do data center connectivity, and then you can do all those things from a single port. Right? And if you go back to saying that, you know, you set up a a point to point circuit, It's great.
Anna:You're done, and you never have to think about it again. If only that was actually true. Right? If only anybody never had to think about their network again after it was set up, but that, like, that rarely happens. You know, what what is, you know, what is the path when you need to upgrade from 10 gig to 40 gig, or from 10 gig to a 100?
Anna:The path is exceptionally painful. It's basically hitting the reset button on going back to, you know, talking to telecom sales, renegotiating, going through the whole provisioning process again, the whole circuit acceptance process. You're, you know, you're stuck in a you're stuck in a treadmill, you know, going through another anywhere from 30 days to 6 months of activity to get from that 10 to a 100 gigs, and we've just we we can press that down to a minute.
Max:What I loved about Packet Fabric when I first saw this announcement and and you guys came out of stealth of this is what we're doing, a lot of the pain points that you also you get into of capacity planning, contract cycles, how much capacity do I need on what term, pricing based on term. You know, so there's a lot of that that comes into this that immediately you say, okay, you have to worry about capacity planning. What port speed do you want with us? You want a faster port? You want a slower port?
Max:You want this port? You have a you have a port, right? You don't have to worry about capacity planning between markets. You don't have to worry about term lengths. You know, you can get into these flexible agreements that are you don't know what you're gonna do with it.
Max:And that I mean, anything that makes my life better, I'm all for it. Right? Like, who wouldn't want this? Oh, wait. I don't have to think about what my 18 month plan is for capacity between these two markets.
Max:I just can change it really easily. I get all that, and that speaks to me a lot. How much of this though is is like a you build it and then see what happens? You know, if you build it, they will come or there's there's a certain amount of chicken and egg because the original idea for Packet Fabric versus how people are using Packet Fabric I mean, I'd I'd be shocked if you said you predicted all of these use cases now that have come out of it.
Anna:A lot of them. Yeah. I don't wanna I don't know that there's a good answer to that question because either you sound overconfident or you sound like you don't like you don't know your customer that much. But a lot of them were pretty well known even 5 years ago, because remember, we're not that the company is still fairly young, especially in terms of for an infrastructure company, 5 years is insanely young. And so every a lot of things that existed then, exist now.
Anna:I think the thing that we probably didn't see being quite as popular, as it is now is the is like the layer 3 use cases. You know, people wanting to actually automate, you know, setting up BGP between for the cloud scenarios. You know, who saw cloud being a bigger thing. Right? No one saw that coming clearly.
Anna:But, yeah, the, the hybrid clouds and, like, multi cloud use cases have definitely become much bigger in, like, the last just even year or 2, because I think there is this sort of people are going through a bit of a disillusionment period with public cloud because it's like there was this huge race to, you know, nobody uses servers anymore in a data center. Put everything in the cloud. You know, everything goes in the cloud, and you have people trying to put, like, their AS 400, you know, database in the cloud. And then it's like, oh, that's actually a bad idea. You know, like that.
Anna:Yeah. Like, that's not gonna work, and but also the cloud can, you know, be expensive. Right? Unless you're unless you're utilizing it effectively. And so there's this mad rush to put anything and everything in there without really consideration for fit or cost.
Anna:And then now is sort of the era that I think we've been in the last couple of years where people are stepping back and saying, how do we best utilize public cloud and our own resources and and make the best out of all this. And and with that, actually comes a lot more new communication challenges because maybe AWS has some feature set that you wanna use, and Google Cloud has some feature set that you wanna use, and Azure has some feature set that you wanna use. Then on top of that, you wanna use some Oracle or you wanna have a direct connection to Salesforce. Right? So the ecosystem got much bigger, and then on top of that, you have a couple existing data centers that you have compute in and you wanna move workloads around and and do all this stuff that's been promised for, like, years years now because that was the whole part of the whole potential of cloud.
Anna:Right? It's free the workload and, you know, be able to shift it wherever it's most economically advantageous to run it at the time. And so people are starting to go, well, now I want this dream fulfilled, and I wanna get my costs under control, and I wanna run things in the appropriate places. And the little bit that everybody forgot about in there is the network. Right?
Anna:Because if you if you want ultimate flexibility and compute, you're moving all this data over the network. And if you have that rigid that super rigid piece in between that can't be changed, it can't be dynamic, you can't create new connections on the fly to where you need, that's a huge problem. Like, that's that right there is the show stopper for that dream because you cannot move data without the network.
Max:I was thinking, I think it's an x kcd comic where they talk about the trough of dissolution in it and, like, the tech adoption. So I'm cracking up agree with that. So a lot of the infrastructure companies, of course, are pushing, hybrid cloud, hybrid cloud, hybrid cloud, hybrid cloud. And there are a lot of use cases. A hybrid cloud is amazing, and we and we see them a lot with our customers.
Max:But the other one, I don't think anybody was really predicting was this cloud to cloud completely virtual environment of, I don't have physical infrastructure anywhere. I just need AWS to talk to GCP or Azure or what or Oracle in your example, and I need something to actually tie all these things together for me privately, fast, and I can just click a button and make it happen. And and you guys, you're doing this. I mean, this is part of your, you know, your core product. And the other thing that's I I find very unique, I think with the early story of Packet Fabric was more, you know, this idea of, like, layer 2 as a service or this network automation as a service of not per se, hey, this is a cloud connection technique.
Max:This is a we want a network between point a and point b, and you want that network to be I mean, do you guys even sell it at 1 gig? It's like 10 gig, 40 gig, 100 gig, x by 100 gigs, 400 gig. I mean, it's
Anna:We do we do sell 1 gig. It's actually by percentage of customer allocated ports. 10 gig is the most popular. The next most popular is 100 gig. And then after that, 40 and 1 gig have about the same percentages.
Anna:And a 100 gig is getting pretty close to it it's a lot closer than I thought it would get to 10 gig demand. So that shows you where the world is going. Everything's going 100 gig.
Max:Well, I mean, optics for an Arista for a 100 gig is relatively inexpensive. I mean, you can buy a a 100 gig optic for 4 you know, $1500 if you're not going 3rd party. I mean, so it's it's a Arista, don't listen to that. But, you know what I mean? 100 100 gig finally got affordable, it feels.
Anna:Yeah. In the last, like, in the last year, the optic prices have plummeted, and and you can get really you can get sub $1,000 pricing, you know, at any sort of scale now for the for l r four. So and that's, you know, without even really trying. You can get better than that even. So, yeah, it's very it's very cheap, ubiquitous, affordable now.
Max:But, I mean, you're already but you're already past that point. I mean, you're at 400 gig and 600 gig and 800 gig. Right?
Anna:Yep. That's, that is that is where we will be going next.
Max:This is when I feel really old because then I have this moment of, well, when I got on the Internet, I had a 56 k actually, I had a 1200 baud modem, but I wasn't gonna say that initially. And now you start thinking ports like it's a 600 gig port. I mean, this is massive amount of data being shifted, but being able to gain access to that and be in a have of dynamic pipe that you can turn up, move data, turn down, reconfigure, do something else with, and and it no longer becomes a limiting factor saying, hey, I need to move stuff from, you know, from wherever they are, from two points just instantly. I mean, there I mean, this this turns into people are doing things that would you predict? Like, would you predict that a IP transit carrier would be using you to backhaul to customers in different markets as an original use case?
Anna:No. No. No. I mean, there's definitely a fair amount of stuff we did not see coming. Right?
Anna:And I guess that's part that's part of any, like, as a service model. Right? You know, compute as a service, word network as a service. And so part of it is you're just putting you're putting a generic framework out there and saying, hey, people, what are you gonna do with this? And so there there's always been surprising things, and you were mentioning before the cloud to cloud use cases, and and that's definitely something that surprised me when people first started asking about that because it's shifted.
Anna:And I mean, maybe this is this is how old I am, but, you know, back in my day, you always had you know, there was always network provisioners and network engineers and, you know, people who, and peer you know, people who do peering and so specialized in the network portion of it. It was this own little tight sphere of people that, you know, had deep expertise in this one area. And now there's DevOps SRE, just straight software engineers that are just like, yeah. Can I just spin up? Because they're, like, their mindset and where they come from is everything's as a service, and they're asking, well, why can't I just spin up network between these two things?
Anna:Like, why doesn't that work? Which is pretty obvious, you know, when you think when you think about it, it should work, and we're like, wow, that should work. We can do that.
Max:I, for years now, have advocated for higher bandwidth enterprises to, you know, take some small increment of data center close to their offices and their their primary locations and do a in, you know, in Metro Connect, because a 10 gig point to point circuit even 5 years ago was relatively inexpensive in a metro. And then, you know, now you're in a data center that has, you know, dozens or hundreds of different carriers, and you have extreme flexibility in pricing for what IP transit bandwidth you're gonna acquire or where you're gonna land your MPLS circuits or, you know, you basically, the world's your oyster in terms of competition with different carriers. And, you know, the big shift now, right, we talk about big packet fabric originally. When when I was thinking about it was, oh, I can connect my own endpoints easier. Right?
Max:You know? Or I can connect to cloud easier. But now there's also this other use case which becomes instead of deploying MPLS network, I can connect to other packet fabric customers anywhere on your environment dynamically, like instantly from any of these locations. And the data center still becomes a strategic, you know, asset for me of get to the data center, connect to Packet Fabric, have massively phenomenal port speeds, you know, with dynamic allocation of service. And I don't need to worry about talking to a telco for MPLS because any customer that's on your fabric, I can connect to, and that's interesting as well.
Anna:Yeah. And and talking about some of the old, I shouldn't say old, but so some of the in the old world, how it used to work, you know, peering used to be a highly specialized discipline. Right? Is, you know, figuring out who who to connect to based on traffic stats, you know, over your public IXs. Who deserves a PNI, you know, public network interconnect, which is just a cross connect to another network to do this traffic transfer and all that now, because there's this, like, fundamental limitation there, especially for PNIs.
Anna:If you're doing a lot of traffic to another network and you wanna do it in a private secure way, I e not over a public IX, you know, that you want something that is going to be more reliable and direct. That means that you have to be in the same location as the network you wanna connect to. You have to physically be in the same data center. And the great thing that we did is we eliminated that fundamental limitation. You no longer have to be in the same data center if you want to create a private direct connection to a network.
Anna:So it doesn't matter. You can do that anywhere on the Packet Fabric, platform, and you can also still connect to public IXs over over Packet Fabric too and peer with whoever you you'd like to over there. But, like, to me, that was always one of the annoying things when I was building networks. For example, back at you know, I'll step all the way back to, like, Prolexic here just for fun. So building, you know, the Prolexic network, which was a massive DDoS network.
Anna:There was definitely, like, we needed to take in a lot of traffic from a lot of different sources, and we had to peer you know, we we had peer wherever we could because we took in a ton of traffic over transit, and that obviously gets very expensive. And this is sort of like the first time when I encountered this, like, why can't you just get more bandwidth when you need it? Because when you're taking in, you know, an 80 gig DDoS, which at the time was big.
Max:Yes. Yes.
Anna:That was very big. And you are, you know, you have, like, you know, 8 by 10 lag to your, you know, to your favorite transit provider, and you can't just add another 10 gig quickly. It's it's very disconcerting. Takes, like, weeks to get that added. So anyway, we have this problem.
Anna:We know where we needed to appear as much as possible. And, you know, our footprint, especially when we were starting, was very you know, was pretty limited. We're only in a few data centers, and so we just couldn't get a lot of the peers that we needed wanted as private peers at that point and have something like this existed. But it made it a lot easier for us.
Max:Sure. And you still see a lot of things where it's, oh, if you wanna pair with me, that's great, but you have to pair with me in 3 time zones and exchange equal traffic. And there's all these other things that come into pairing that become, you know, all these layer 8, you know, negotiation tactics. Yep. My favorite my favorite is the ones when you send a peering request out and a salesperson who is monitoring that inbox responds to you with, like, the sales, like, oh, great.
Max:You know? Like, here's my sales pitch back here. And, like, no. No. This is appearing at oh, boy.
Max:Let's not get into this talk. Yeah. So walk us through, you know, we've decided we're gonna sign it with Packet Fabric, you know, and and and walk us through the process of, you know, what does it actually mean in terms of, like actually, your month you have a not month to month options. We don't even have to talk about contracts. Let's forget the call contract point.
Max:Like, from a technical standpoint, walk me through the process of what this actually looks like for somebody, and we'll pick that they're in a, you know, they're Ashburn, Virginia or Los Angeles, one of these major carrier hotels. So, obviously, you're on net, and they're on net, and everything works out great.
Anna:Well, you would just go to www.packetfabric.com, Click on register. Go through filling out, you know, some basic stuff, technical information, address, and then you click through the MSA. Just, you know, click and agree. Hit register. We on the on the back end, we just verify that the company is a real is a real company.
Anna:And the next 24 hours, you get your login, and you log in to the Packet Fabric portal. You would then go to order a port in the location that you are in. So say that this is 1 wheelchair since that is very local to you. We'll go ahead and pick CoreSite on 1 wheelchair. So you're like, great.
Anna:I wanna order port CoreSite on Wilshire. You select that. The port takes about a minute to provision. You click download the LOA, and then you take that LOA and you give it to CoreSite. They run the cross connect.
Anna:And then as soon as the cross connect is complete, you are connected to the PacketFabric network. And if you then say that your use case is to connect to Amazon, Google, and then also connect to your East Coast data center, you would create a connection to Amazon, which takes again about a minute. Pretty minor information to fill out your Amazon account ID, you know, what VLAN you're gonna use. Same thing with Google. And then you would also go and provision a port on the other side.
Anna:We'll say core site again since we're on that theme. We'll pick core sites, Virginia VA 1, go provision a port there. We would then follow the same process on that side, get an LOA, get the cross connect run there, and then click to create a VC from LA to Virginia, and we'll just say that, that's gonna be an 8 these are both 100 gig ports. You're gonna do 80 gigs across the country. So about a minute later, you'd have that 80 gig connection up, and then you're also we'll say that you have 10 gigs to Amazon and 10 gigs to Google too.
Max:And and if you're listening to this, it really actually is as easy. I mean, this is not this is not, you know, paraphrase in any way, shape, or form. The so VCs, virtual circuits. Right? That's what you VCs?
Max:Okay. I mean, so I mean, also, like, my my brain works with associations. So I think of VCs as, like, you know, this is a VLAN tag on a dot one q trunk, and you just happen to be in between, you know, point no. You're not doing that exactly on your network. But, you know, in terms of, like, anybody that's doing LAN networking.
Max:Right? That's that's a good way of explaining what this actually is.
Anna:Yeah. Yeah. And it does you you have a VLAN on either side that you're using. They don't have to match, of course. And so, yes, from it is very much that that feel is that and and on the interface itself, all all we are doing is provisioning that logical interface for that VLAN.
Anna:So
Max:And part of what you do for Packet Fabric is you guys have a big software defined network. We'll use the SDN buzzword. Right? Because the biggest thing is you know, I mean, you're right. But you have an application that's dynamically configuring and talking to, you know, a lot now of physical devices in a lot of different locations to say, you know, this customer needs to come up, and what's my inventory at this address?
Max:Okay. I need a 10 gig port. Okay. Here's the inventory for that 10 gig port. And then here's everything that has to be built now in order to connect from, you know, LA to Amazon and east or from LA to Virginia East or, you know, these sorts of things.
Max:And the explanation of what is actually happening is hiding a lot of very sophisticated things at the same time because there is I mean, you have a there's a lot of devices that have to be provisioned in order for this to work.
Anna:Yeah. Yeah. I mean, no. There's a device on either side. I wouldn't call it like a lot of devices, but I overall, the network is a lot of devices.
Anna:You know? And there's some pretty you know, and this is starting out, I guess, in my mind, it's always pretty simple because, you know, I built the original spec for it, so now the amount of things are very complicated to me. But at scale, there's some really interesting things that happen. Right? Because, you know, the the laws of large numbers start to apply, and that's where things get really interesting to me.
Anna:So yeah, we have, you know, we have, like, our in house SDN controller, you know, that we refer to. You know, we wrote all of it. You know, for anybody that that's curious, you know, this is all of our all of our own software. You know, we're using open source components in there, like, nothing actually having to do with SDN controller, but, you know, we use a Postgres database. We use Redis.
Anna:We also use RabbitMQ, and those are, probably, like, the 3 major open source components that we utilize in the controller. But other than that, it's every, you know, it's everything that we wrote in house. And, I think one of the more interesting sides is our metrics collection because we're dealing with about, you know, 500 devices now. And we collect metrics from those devices every 30 seconds to a minute. There's a little bit of variance in there depending on the device.
Anna:Some of them don't like to be pulled quite that often, so we give them a little bit more time. And, like, pulling that amount of information we have, you know, it's roughly as of a couple months ago, I didn't look to the I didn't look today. It's gonna be much larger. But as of as of a couple months ago, it was about a 45 terabyte database with all the stats from the network since its inception.
Max:So so please tell me you're joining, like, a time series database in Postgres so I could completely nerd out. Okay.
Anna:See that. Yeah. Yeah. So so we actually we actually started off we actually started off with HBase. So this is one of the systems that I started yeah.
Anna:So we started off running HBase. And let me tell you, I have a very particular set of skills that encompass being able to completely destroy an HBase cluster very effectively. I burned that thing down at least, like, 3 or 4 times. This is well before we were ever operational, so it wasn't a big deal. And after after figuring out that running HBase and zookeeper and all this is not for the faint of heart Uh-huh.
Anna:We decided to actually go with Bigtable in Google Cloud. And so we use that along with the open TSDB for storing all the for storing all the metrics, and it was super cool for a while, and it worked great. And then once we started to get to really big scale, we ran into a little bit of performance problems on Bigtable, which we could have corrected by just throwing a lot more money on it. But we decided, you know, not a big deal. We're gonna move it back in house onto time series DB, which is which is working great, and we've been running that for about over 6 months now.
Anna:So we migrated all of our data back out of the cloud. Thankfully, we were like, we were able to do that effectively because we're directly connected to Google Cloud from our own system. That's one of the nice things is that we're we're able to dog food ourselves there for transferring all that data, and it works very effectively, which is great. And so, you know, transfer all the data back out. It's in timescale DB DB now, and we don't.
Anna:We're running, you know, completely our own software collection, stack, storage, everything on it now.
Max:I wrote a billing platform in the early 2000. It was based on top of MRTG, and it was feeding data into Postgres. And it's it really I mean, literally, it was just a parser that was parsing. I I I was I read through the specs in the MRTG log format. I was like, oh, I could parse this and and shove this somewhere else, and it wasn't that sophisticated.
Max:But now when I read all these things with, with time series databases and Postgres, I get really excited of like, man, if this existed when I was doing this, it would have made so much cooler. And then you you read all these things of companies taking NetFlow data and shoving NetFlow into time series data. And and just what's available now is awesome. You touched on this really briefly, but I wanna I wanna walk back a second and talk about, you know, almost every company has a redundancy requirement. Inside of a data center, you're relatively stable, you know, cross connects.
Max:I mean, they get bumped, you know, and inevitably, somebody's working on a panel somewhere, and you have a cross connect that just rounds out for some no. And it always is the same thing. If you have to open a ticket, and there's good to go and un patch everything, and patch everything back in, it miraculously just starts working again. But you offer a couple of options. There's one, it's an option.
Max:Right? Get a second port in a facility for redundancy, but then you also do something else that's very fun on your network that not a lot of carriers will do for you by default. You know? And let's talk about this.
Anna:Yeah. So that's actually having redundancy by by default. And it's one of those things that is always astounding to me that this is a new concept because when you pay for something like, you know, a a 100 gig wave, and all it takes is a, you know, an errant guy with a shovel to completely destroy that for, you know, hours, days, weeks, or months, depending on the location of, you know, where this this mythical dude decided to go out and dig. It just seems like it's a really, like, it's a really poor design and it's really poor investment for your money. So every every service on PacketFabric is redundant by default.
Anna:And we have at least 2 different carriers out of each and every data center. In most in the vast majority of cases, it's much more than 2. And we actually go for maximum fiber path diversity. Like, when we are when we are when we are actually sourcing our paths, you know, we are looking for, you know, at what points do they crossover. Right?
Anna:And how much crossover is acceptable on those paths because it happens shockingly more than you think it's ever gonna happen. Fiber breaks all the time, which is just amazing. It's like, don't these guys ever call 411 before they dig? I don't I don't know because there's always a backhoe going through something somewhere, and if not, then it's like aerial fiber and a bird got caught in it. And and squirrel no.
Anna:Squirrels, like, is that is couple of the best RFLs I've ever got is, severe squirrel infestation on aerial fiber.
Max:I'm laughing because I know exactly. The other one that I really like the most is, you know, somebody that was bored hunting decided to use our splice box as target practice. And Yeah. The you know, shot out the splice box, and so you have the Yeah. I mean, at this point, the ship's anchor doesn't even really seem exciting to me.
Max:What was the other one that was really terrible?
Anna:Sharks.
Max:Sharks. Sharks are fun.
Anna:Sharks are good.
Max:Train fires. Train fires and tunnels. That was a good one.
Anna:Oh, yeah. Train fires and tunnels. Yeah. I forgot about that one. That is a very good one.
Anna:That
Max:was that was not fun for anybody in Philadelphia in just in terms of, you know, not having any Internet for a week.
Anna:Yes. And or man man there was, like, a big period too in New York City where there was a lot of fires and manholes for some reason. There no. It was it was like 3 or 4, like, right one right after another where
Max:epidemic of fires and manholes.
Anna:Epidemic of fires and manholes, taking out fiber and destroying everything. It's it's one of those things that's so I I'm always fascinated, by undersea cable. Right? And the cable lane ships, how we key when they ran the new cable from Hillsborough down to Australia, Hillsborough, Oregon down to Australia. You know, I saw lots of pictures of that in video, and it's fascinating to me because it's so easy to forget that everything that we're doing right now, you know, the fact that we're talking to each other over the Internet ultimately comes down to this weird cable, this weird physical cable placed somewhere that all somebody has to do is, you know, chew through it.
Anna:Not even somebody, a squirrel. All all that has to happen is a squirrel to get bored and chew through this thing, and we are no longer talking over the Internet. So much critical day to day activity occurs over the Internet, and it rests on this very fragile underlying thing. And, you know, people have a tendency to forget that because we're so abstracted from it, but we certainly have not forgotten that, and it's, you know, a major part of our planning.
Max:Hi. I'm Max Clark, and you're listening to the Tech Deep Dive podcast. At ClarkSys, we believe tech should make your life better, searching Google is a waste of time, and the right vendor is often one you haven't heard of before. With thousands of negotiated contracts, Clarkesys has helped hundreds of businesses source and implement the right tech at the right price. If you're looking for a new vendor and wanna have peace of mind knowing you've made the right decision, visit us at clarksys.com to schedule an intro call.
Max:So, I mean, you you say redundancy, which is the right word. Carriers talk about this in terms of protected circuits. Right? So if you're thinking about, like, the code words, if we're listening right, it's it's it's a circuit it's protected circuits. And the cable systems, you know, every Transoceanic cable system has protection built into the cable.
Max:There's usually 2 paths. And and on the West Coast of the US, right, we have, you know, Hillsboro, Oregon or, you know, Seattle. So, basically, Seattle, I think, is is probably the the dominant one, and then it's, San Luis Obispo and Redondo Beach in in Southern California. But these cable systems, you know, one cable runs across both paths. They go from Seattle, and they come from Southern California.
Max:So and they're protected. You know? The cable system's protected, but yet you can go to the cable operator. And when you buy it away from them, they don't give you protection by default. They're like, oh, sorry.
Max:You're on the southern path, and it went down. Oh, you want the northern path too? Pay us more money.
Anna:Yeah. Yes. And and there is an economic reason behind that. Right? They are in fact 2 different cables.
Anna:There are there are 2 sets of there are 2 sets of economics that play there, and it and it does make sense that it costs more. It's just the thing that especially for especially again as we get into this age of cloud and everything as a service and network becoming a service, you can't expect the average person buying that service to understand the depths of, you know, redundancy and why you need it and understand how cable systems work. So it just should be something that's there. You know? But you expect the network to be up, and that's how we built our network was to be just be up.
Max:Or that you know the magic code word of, like, I need to order a protected circuit or I have an outage on this circuit, and I can call up and I say, I need snap protection on the circuit. It's like, you've got, like, the magic code. It's like, oh, you you know the code word. Okay.
Anna:Oh, okay. Yeah. You can come in the Speakeasy now. Yeah. And yeah.
Anna:And that's just a layer, you know, our our goal. And I say that I say this in jest, but it is a good goal is that we don't want our customers to have to talk to us. I mean, we're there if you do wanna talk to us, but we don't want our customers to have to talk to us. Right? You know, they should just get the service that they expect without having to give us magic code words.
Max:There was a big moment for me a few a couple years ago with PacketFabric, and one of the transit networks that I love is NTT, and they have a relatively small footprint. And part of what I love about them is they have a small footprint. Right? There's not a lot of stuff. I mean, their network becomes very efficient in in how they control it and how they built it out.
Max:And they had this little, like, side note announcement of, hey, by the way, all all of you people that have been asking us to have network in Vegas, and we're never gonna build into Vegas, and you don't wanna build your own way. It was like, oh, by the way, you can get ports from PacketFabric now and come directly over to us. For me, that was a big moment. I had this like, oh, wow. Like, this is really here.
Max:Like, this isn't, like, theoretical anymore. This is really here. Because a transit vendor at that scale is saying, you can order 10 gigs or 100 gig circuits from PacketFabric and interconnect back with us at 1 Wilshire, or interconnect with us up in San Francisco, or interconnect with us wherever you want the circuit to go. And, oh, by the way, PacketFabric can provide you that protection or that redundancy with it, and you can just home into ports. I mean, that's like, come on.
Max:That's really cool.
Anna:Yeah. I and NTT Transit is is awesome, and I'm a huge fan of NTT. They do a lot of really great automation as well on their network.
Max:Their automation is freaky though when you get into what they're doing. I mean, it comes from the fact they've been riding it for for decades. But
Anna:Yeah. Yeah. They do they do a ton of cool stuff. Huge fan of them, and it's a great it's a great use case too because we ultimately help make them, you know, a great transit provider more available than they are capable of on their own just because of the the capital investment to go into every every data center that somebody might want is, you know, is a pretty big overhead. And in the customer base in some of those is relatively small.
Anna:So we provide access to that, you know, through all of our pops. And, it just like it makes their footprint, you know, it goes from you know, I can't remember what the NTT footprint size was before, but we add, you know, a 160 pops to that. So it's a or a 170 now. So it's not a non trivial number.
Max:I I think in I mean, they added Boston in the US. Right? So it's 6, size 3, 5. I don't know, like 8 pops, 10 pops. I wanna say cities.
Anna:Yeah. It wasn't Not big. It wasn't very many. Yeah.
Max:So I mean, 5 years ago when you were starting Packet Fabric, I mean, this was still a relatively novel idea. There was a little bit of noise in this market, and there was this idea. And I mean, you know, this kinda dates back to, you know, tier 2 networks that were started doing, like, went from IP transit, but the margin IP transit completely evaporated. So they moved into IP transit, and then it became sorry, transport. And then there was like this idea of like remote peering that was trying to be pushed as like this, you know, like, you know, what's gonna be the savior of all these networks, and that didn't really materialize.
Max:But so, you know, the cloud interconnect and the network as a service idea or or, you know, these these these these dynamically allocated networks, It went from conceptual to this is, like, really here. I mean, not I mean, there's there's I mean, look, data center carriers are getting into this with either in the metro or in regions, or they partnered with companies, or they're trying to build out their own fabrics. You know, it it's it's fun to watch that happen because it means, you know, this is a real thing and this is not going away. Like, this is just gonna become a standard for what what happens here. And I've I've been curious for a while, like, when do we see a carrier start doing this?
Max:And I don't really think we're gonna see a carrier do this, but, you know, when does a carrier actually start doing this? Because in theory, you know, they have the last mile, which is usually the hardest component to always solve. You know, data centers are relatively easy, but the last mile is the tricky part. And I'm I'm kinda curious what you think is, you know, you know, what you guys are looking at over the next 2 to 3 years in terms of planning and and and cycle coming down in the future here. Because this is more than just, hey, we're gonna see terabit network links.
Max:This is beyond port sizes getting faster and faster and faster. Like, what else is kind of like the crystal ball?
Anna:So I get asked this question a lot actually is, you know, when when are when are the carriers going to going to jump into network as a service? And I don't even know that we'll see it in the next 3 years at all. Because here's the thing, automation is really hard. Networks are really hard, and combining the two things is really hard. Like, we all we have this, like, joke that is a running inside joke.
Anna:You know, the line in Jurassic Park, you know, we have we have all the problems of a major theme park and a major zoo, which is when you're which is, you know, you know, we have we we are a software as a service provider, and we have a and we have a massive network. If if infrastructure and automated infrastructure is hard, and that's why there's, you know, that's why you see a couple companies that absolutely dominate at it, you know, like AWS and Google and Microsoft. That's why there's only a few of them because these are hard problems to solve. And when you combine the fact that, you know, this is a hard problem, you combine it with carriers have never had the the mindset to automate everything. Like, software is not a thing that they do.
Anna:Software traditionally has never mixed well with telecommunications pretty much ever. And then you layer on top of that the extra fact that, you know, all carriers have a ton of revenue tied to existing services. Their wave services, their MPLS services, their VPLS services. All those things that are running today, where's the incentive to do this? There's a huge knowledge gap.
Anna:There's a huge hurdle in terms of difficulty, and there's a huge disincentive, you know, from these are publicly traded companies that have a ton of revenue that is not tied to them automating a thing. So there's you know, they they have whatever the reverse of stick and carrot is. Like
Max:Innovator's dilemma is is I think the the popular term. Right? Okay.
Anna:Innovator. Yeah. Innovator's dilemma. They and that is a massive problem for them. So I just don't think it's gonna happen anytime soon, and I've had these conversations with some of the folks that are working at these big carriers who are who are making efforts to do this automation.
Anna:And some of them have done you know, they have implemented system you know, they have implemented systems. They've they've built some software, but the problem is they can't rule it out for a variety of reasons. They can't rule it out because of internal process. They can't rule it out because the network is just too mess. It's just the brownfield network, which we had the advantage of starting on a greenfield and easing into a brownfield network.
Anna:And that was a learning curve, and and we did it. And and now we're super confident that we could go into any brownfield network and and flip it around, but that's 5 years of experience there. And also us having the will and drive to do it and a carrier who just is like, oh, you know, we're not gonna do this in our Brownfield network because there's too much existing revenue at stake here. We can't go and meddle with it. So you've just got barriers to entry all around.
Max:I mean, for clarity, what you're talking about is, you know, you started out as a as a Juniper platform and then have added other, you know, network vendors into the mix as time has passed, and that will continue to accelerate, you know, based on just what's on market and who's doing interesting things that you actually need.
Anna:And that's the great thing about having the software translation layer that we wrote in between there is it's just adding the module for us to add another software or to add another hardware vendor. So it's a pretty trivial effort. And it's not just that about Brownfield, but it's also things like, you know, cabling. Right? We have been meticulous in keeping, you know, all of, you know, all of our cabling is actually stored in a in a database and is kept up to date automatically as opposed to the vast majority of what exists out there in in other companies, other carriers, anyone.
Anna:You know?
Max:So there's a a 2 RU Juniper device. I forget the model number off right now, but it's a 288 10 gig interfaces that'll land on this 2 RU. And it's I mean, it's awesome. Right? It's a it has a huge, you know, PIP table.
Max:It's it's like the holy grail of Internet edge devices if you're actually a carrier and and be able to do a whole table. And you're like, on the surface, this is great. And then I was talking with somebody, and they were explaining, like, how do you actually deal with 288 pairs of fiber in a cabinet at a data center wiring into 1 2 RU device? And, you know, so you have a device, you know, that's 3 and a half inches tall, and you've got a bundle of cable coming into this device. It's thicker than the device is.
Max:What do you do with that? And and it's such a strange problem to think about. You're like, oh, man. This is such an awesome box. And you're like, wait a minute.
Max:I can't actually physically cable this thing. What does
Anna:that look like? Yeah. I mean, we, like, we have a really excellent deployment team, thankfully, that makes all this stuff that makes all this stuff look look pretty and, you know, gets the initial information into the system right? Because that it's like, you know, the the point of breakdown is always the human interface in these systems, because there is, you know, there is no automated way to initially get that cabling information into the system other than for a human to do it. And that's where, you know, if there's ever any errors that happen, that's where they happen.
Anna:And that's the high barrier to entry there. Right? Is is having that, you know, having that human go and enter what every cable position is from the patch panel to the device. And, you know, when you think of when you think on I'm sorry. Going back to that because I I like thinking about this telco problem.
Anna:Like, you know, how would I take an existing telco and automate it. Right? Because it's kind of a fun thought exercise.
Max:Somebody might buy you if they hear this.
Anna:You know, how to go back that initial step, like, how to go back and actually get all that information in. Right? Because you're talking a huge existing footprint, thousands of locations, maybe more to do that. And the than the amount you know, the mistake rate, it's different when you're building. When you're actually building out these pops, we're doing it at a rate that is sustainable for us to find errors.
Anna:But if you're doing it on such a big scale, those errors are, you know, you're going to have, you know, again, large numbers coming into play. If you have a 10% mistake rate, that becomes massive. Massive.
Max:I only know a few companies that have done these kind of migrations well. I mean and so in Los Angeles, it's capacity. You know, moving from 1 wheel Sherpa to 900. People hit the point, and they they're just at the wall, and they have to move. And so watching a few companies make this transition or a few carriers make this transition, there's only been a couple that have done a very good job at it.
Max:I mean and, of course, I, to their credit, has done excellent with those, cross connect, you know, those hot cuts. And operationally, they've we're very impressive to watch this happen because it is very difficult. I mean, you know, to your point, I think this is this is the hard stuff.
Anna:Yeah.
Max:So you had an early mover advantage, you had a greenfield advantage, and you're definitely now getting to the point where you now have a capacity and size advantage. I mean, you guys are becoming the, you know, larger gorilla where you have a lot of locations, you have a lot of infrastructure, it'd infrastructure. It'd be very expensive for somebody to replicate from scratch what you have in place now. Your relative scale of being able to add a new PoP or a new facility is relatively, as you said, trivial. It's relatively easy for you to do at this point.
Max:And each one of those gives you incremental value to your to your network and to your fabric and where you're located. And and it almost feels like you're you're approaching critical mass at some point. And again, another big moment for me was when you finally expanded into Europe, and that was, you know, a long time coming. But watching that actually happen, I mean, that's that's a pretty big you know, going going transoceanic is a is a big deal as well.
Anna:It is. It is. We went transatlantic and transpacific at the at the same time and achieve achieving that reach is a big deal. And again, all the all the problems with doing this are not where you are not where you think. You know, at the software layer, it doesn't matter.
Anna:We could add locations on Mars, the moon. We could extend out of the galaxy. You know, it's to the software layer, it doesn't matter, but to the actual, you know, building out the infrastructure, there's always just so much involved in going to new companies countries. Right? You know, gotta worry about studying setting up business entities.
Anna:You know, how we know what is your tax structures look like, dealing with shipping and customs. These are all the things that you encounter, and that's where that's where the hard part comes in.
Max:And And so the the big takeaway there is is don't do all the hard part. Just use somebody like PacketFaber to do all the hard part for you and make your life really easy in the process. Just say, hey. I have got a web page or an API. I wanna I wanna circuit from here.
Max:Click. You know?
Anna:Yes. Yes. And that and that really is the beautiful thing. I mean, I wish that we had a Packet Fabric that we could use to build Packet Fabric on. It'd make it a lot easier.
Max:I but but at the same time, you're being I mean, this isn't something where you're massively marking up or taking advantage of people for the flexibility of ease of time or time to market. I mean, you you really are driving cost down for people at the same time as well. I mean, this is as your costs have driven lower, you're using that as an advantage to drive your cost to your customers lower. And and and cost of this this, you know, layer 2 network being driven down, I mean, that I mean, that's gonna open up doors for people to do things that were never anticipated as well. I mean, at some point, it just becomes so cheap.
Max:You're like, well, why wouldn't we just do this thing?
Anna:I I agree with you. I think people should I think people should think exactly that. And that is one of the interesting things is we made this decision really early on that we were gonna make all of our pricing public because this is, you know, this is as an as a service and any as a service, whether, you know, network or software, you have public facing published pricing. And this is a pretty bold move in, you know, in terms of telecoms. You know, I I I can't think of another, you know, telecom anywhere that does this.
Anna:And so it was it was one of those decisions that was that was made at the beginning that turned out to have a really a really broad long term impact because people would look at the pricing, and and some some people were just blown away. That's it. That's all you're charging, you know, for something that's Yeah. There's no cash.
Max:There's no cash. This is really the pricing. Yes.
Anna:Yeah. And we actually did face a little bit of that at first as it looked too good to be true. Right? You know? So you're telling me I can provision a circuit instantly, and it only costs this much.
Anna:It doesn't sound like yeah. It didn't sound like reality to a lot of people. And so that was something that was certainly something we didn't anticipate having to face, but it was interesting to say, no. We'll show you. Like, it really is true.
Anna:This is not hypothetical. We've built it. You can do this now. We'll show you a demo to show you that it's real, that you can actually provision this and and that is the cost.
Max:I I I you know, for context. Right? I mean, these are these are use cases where it is cheaper to provision ports on packet fabric to cross connect data centers within the same metro than it is to go to your local dark fiber provider and and take fiber from dark I mean, and that's and you and you take a step back from that and say, well, you know, these two buildings are massively interconnected with each other, and there's there's thousands of strands of fiber in the ground between them. Not only is it complicated to get access to those things, then they want 5 year contracts usually for those things. And but just the price, you're like and in some cases, you know, so LA specifically, the market there are some players in the market that's relatively inexpensive to get dark fiber between buildings, especially in the downtown area.
Max:But it's still cheaper just to go to Packet Fabric and just turn it on. It's
Anna:Yeah. Yeah. Yeah. And it's not it's it's it's cheaper and not just the respect of, like, what you're actually paying, but the time investment that you have to put in. You know, to actually procure and use dark fiber is it's a pretty big time investment, right?
Anna:Because you're dealing with you're dealing with typically fiber companies are more, you know, wholesale companies. They're selling to businesses, they're not selling to, you know, individual consumers. So the business side is just is not set up to make those sales, and, so you're spending again weeks, months chatting to salespeople, doing all this overhead thing. So it's not just the actual cost, but it's the time savings that you're looking at as well because it's it's pretty significant time savings and just sheer pain of doing business savings.
Max:One of your investors has actually, one of your investors, a very major investor has this vision of telemedicine. And and but it's not telemedicine in the sense of, like, I'm gonna talk to my doctor over a Zoom session. This was more about, you know, can you get into a situation where you have low latency network with high capacity, and you can do things like remote surgeries. And that's a very bold vision for the in terms of what actually has to happen for that network to support that. But, you know, there's a lot of infrastructure that has to go into place before that's available.
Max:I mean, I'm I'm a lot you know, not just infrastructure in terms of, like, you know, cables in the ground, but infrastructure in terms of robotics that have to be at a certain sensitivity to support these things. But, I mean, obviously, short term, near term packet fabric, you talk about expanding to more pops, more countries, more continents, you know, more services, more clouds as they come online, more networks. Right? I mean, are you starting the planning of pushing down into the last mile? I mean, is there a hit list of, you know, let's go get into every hospital at this point?
Max:I mean, I mean, what is you know, how does this evolve?
Anna:So I'm really glad that you brought up the the, you know, the whole entire medical field because this is one of this is just like a hobby of mine. I love to I love to dabble in love love to dabble in medicine and and genomics and biomed, and all these things are super fascinating to me. And it, again, there's there's this thing that has happened, right, is people have been sold this this future and a lot of these cool things coming. And we don't really have the way to deliver it because the underlying infrastructure is so bad. Like for example, there's a lot of AI out there now that does readings of MRIs and x rays.
Anna:And it's already been shown, you know, through numerous tests that AI performs way better than a human, way lower error, you know, like, much lower error percentage, much more cost effective, much quicker. You know, imagine if you could go in for an X-ray, and instead of having to wait a day for it to go to an actual doctor to look at it and give you result, AI could respond to that in seconds, and and let you know where you know, if it's a break, what it is, you know, all the information that you would need to treat. But we can't do that because the facilities where you're getting these x rays done are, you know, connected by 3 meg DSL, off in the middle of a cornfield or something. Uh-huh. And there's just no way to communicate back to say the closest data center, you know, in an effective way to to train because images like x rays and MRIs are actually really big images.
Anna:Yeah. They're huge images. They're sending a lot of data. So there's no way to effectively move that data back and bring this, you know, what is like a mind blowing advancement in terms of medicine to the consumer, all because the network sucks. And we we start to look at things like this.
Anna:Yeah. It's pretty obvious that, you know, there's some places that we need to push into to help make this dream of, you know, a better healthier population actually happen. Because when you think that all this stuff is you know, all these medical advancements are being throttled by the network, it's pretty sad that we have you know, the infrastructure is the pinch point in this. And there's also a lot of cases like that with genetics. Like, there's massive genome databases that sit in a lot of different institutions, both private and public schools.
Anna:You know, like University of Chicago has a big one, and there's NCBI and a bunch of them out there, and sharing the information between these databases is incredibly hard because a lot of these facilities just aren't very well connected, and a lot like so many of them have options that if you want to do a big data export, you are literally like, they have they it says, like, on their websites, oh, if you need more than, you know, x amount of data, we'll ship you a hard drive. We're still shipping they're still shipping hard drives in this day and age. And a lot of them also use, like, AWS as a FTP intermediary to to move to move this data. And so when you think about because what's being what's this data is primarily used for is cancer research. And so, like, you could kind of if you want to do a big, hyperbole here, you know, you could say that maybe one of the reasons why we don't have a cure for cancer is because we just can't move this data.
Anna:I mean, the cure is locked is locked out there somewhere in all these different little silos, and because they can't communicate well, we still have this problem. And those are just two examples. I mean, there's there's a a million there's a million more of, you know, advancements that are being held back by by basic infrastructure at this point.
Max:Let me let me take this down a notch for everybody. So one of the funniest funniest one of the I'll stick with funniest. So Chick Fil A actually has a very long technical post on one of their engineering blogs. Yes. Chick Fil A has an engineering blog.
Max:It is awesome. Why? And it talks about Which is the one
Anna:on the street?
Max:It's it and it and it's it's relevant because it talks about machine vision and machine learning on how they actually structure, predict how much food to produce in their locations. So on a real time basis, based on trends hyper specific to that location, this application can say, hey. This is how much chicken you have in a basket ready to go, and this is your peak demand, and this is what our prediction is based on all these other data points that we have. And they have a machine vision coupled with machine learning platform that now at each physical location could say, you need to make more chicken, or don't make more chicken. You know, really basic things.
Max:But when you look at that in scale and you say, okay. How many how many locations do you have across, you know, your your footprint? And what is the cost of your actual raw goods? And how many people do you have in line? And and so for the average person going through a drive thru, their experience is, was it quick or was it slow?
Max:And you're like, well, if you have a positive experience because it's faster, now who would actually ever think that behind all of that, there is a computer application running with a camera looking at a basket of chicken applying a machine learning model, you know, and a data set against it. It's like, okay. That was awesome. I never would have thought that that was a thing. Right?
Max:But, you know, when you have access to more and more data, and it's not restricted, and you can move it around, and it's it's it's like you've set it free. All these things come up that you would never think of before because somebody had some idea of, like, I can make I can make our drive through line goes faster.
Anna:Yeah. That is amazing. And I absolutely want to go read this whole blog now because that is fascinating, and it's the perfect it really is the perfect application of that whole promise of promise of tomorrow. Like, well done, Chick Fil A. And it's running like so thinking back to buzzwords, it's kinda died down a little bit.
Anna:The, you know, the the buzzword of big data. Right? You know, big data. It's all about big data. What, you know, what can we find out of data?
Anna:Right? And data is amazing because you like, the insights that we can gain from it can do great thing for, you know, so many applications. But I think now we're going into the era of, big networking, because big data is great. But if nobody can access it or do anything with that data, it's not valuable. Data is only as valuable as how you can access it and what you can do with it.
Anna:And if that data is trapped, it's not very valuable.
Max:Anna, it has been so fun for me to chat. I actually been a big fan watching you guys develop over the last few years and coming out of stealth. I remember the announcement all of a sudden, this this little this this group got together, and everybody updated their LinkedIn posts of of we're all in hot you're all in stealth mode and and, you know, starting to get, like, rumors on the of what was what was going on. So it's been it's been really fun watching you guys have, developed. And like I said, there was a couple of really big moments for me of, you know, this is here, and and this isn't going away.
Max:And that was really I mean, for somebody who's been provisioning telco circuits the wrong way for 20 years, you know, having an option to do it differently is always very exciting for me. So thank you very much for your time today. It's been a it's been a pleasure.
Anna:Yeah. Thank you for so much for having me. This is really fun, and, I can't even tell you how happy I am to learn about this Chick Fil A blog. Like, that I feel like I've got my rest of the day planned out now. I'm gonna be reading this thing.
Max:It's it's excellent. You're gonna love it. Thanks for joining the Tech Deep Dive podcast. At Clarkstas, we believe tech should make your life better, Searching Google is a waste of time, and the right vendor is often one you haven't heard of before. We can help you buy the right tech for your business.
Max:Visit us at clarksys.com to schedule an intro call.
