The One Network That Saved 1500 Franchises From a $10M Tech Meltdown with Craig Collister
Craig, what is act what's your title?
Craig:Sales architect, actually. It was a solutions engineer, but they updated our titles probably six months ago.
Max:Craig, let's start with the customer when they reached out to you. So can you give me some background on industry size segment? Background, if you can disclose publicly their name, that's great. If you can't, just fill in some of those blanks. But more importantly, I'd love to know what they were dealing with that started this conversation.
Craig:So this is a quick service restaurant company. They've got, you know, 1,500 approximately 1,500 locations throughout The US, and they're growing exponentially. You know, probably 200 to 300 new locations per year, I believe it is, franchise models. So, you know, they've got the you've got the corporate, of course, owned location. Majority of the locations are owned by franchisees.
Craig:So there was no directive suggestions from corporate as to what they think would work best, but it was really ultimately up to the franchise owners to decide what they needed in term from a network perspective, security, etcetera. And keep it in mind, the these franchise owners, they're not IT folks. They don't really have IT staffs. They just the owner or someone that has some kind of experience with computer savvy, call it, whatever, they'd be responsible for getting what they thought would be adequate. The issues came up where they would have an LTE backup.
Craig:They'd have a broadband circuit, but really no security real security infrastructure in place. Nothing really handling a failover in an optimized way, and they were experiencing a lot of outages.
Max:When you say so fast food, I think, you know, you say quick. Right? I think it's fast food. Right? I'm I grew up in that in that with that brand name.
Max:In franchises, you've got a little bit of a push and pull, right, where the franchisor is gonna control brand and maybe look and feel and have some perspective control the menu and what's being served. And then the franchisee has a series of decisions that they get to make as well. So what you're talking about in this case is the franchisee probably had a point of sale terminal or some sort of was that unified across the entire environment, or did each franchisee could go out and get their own POS terminals as well and figure out what they want?
Craig:They could get their own POS terminals. They're the corporate said, hey. This is what you what we think is gonna work best, but it's ultimately up to you Okay. What you wanted to get. So I think later on, they decided, hey.
Craig:It's probably gonna make more sense to use the same POS vendor. And so that was put into place, but initially, it was not. And so what's really kind of interesting about that directive is that when we and and maybe I'm jumping ahead a little bit here, but when we started to propose our solution, it's really important to remember before proposed solutions, the most important thing for us is to understand the requirements, the needs, you know, act as a consultant, a trusted adviser. And that's really where I come in in terms of the technology and things like that. Because at SpectroTel, we have a lot of tools in our toolbox.
Craig:We're not locked into a specific vendor. So we're very agnostic when it comes to what it is we deploy. So when we understand the requirements, that's absolutely critical, you know, knowing what it is they're dealing with. And so because these franchise owners, you know, kinda were just doing what they thought was best, maybe taking the advice of corporate, maybe not. It was just coming out.
Craig:They just had so many disparate services. Everything was kind of a hodgepodge, and there was they were lacking reliability. And, of course, quick service with fast food, whatever you wanna call it, they'll lose not being able to make credit card transactions is that's a killer. I mean, it literally stops the service in its place. And so understanding that, we started to really talk about to them, what are your pain points?
Craig:How can we help? And then digging in a little bit, you know, further as to the specifics. And so when we went through and started proposing specific options, of course, cost was a factor. But Yeah.
Max:If corporate, if the franchisor is it was in a state like, we're suggesting a POS terminal. We're suggesting this. We're suggesting the technology standard, we're not enforcing it. Things had to have gotten pretty bad in order for it to let raise the level where corporate was making a decision of we have to impose a standard now, and we have to fix this problem. This isn't like a little bit of a problem.
Max:This has gotta be a pretty massive problem.
Craig:Yeah. Absolutely. It was, and they were getting extremely frustrated to the franchise owners where it was they're trying to troubleshoot. And, again, these they're not technical people. They wanna focus on providing food services to their customers, and they don't wanna have to worry about routers and firewalls.
Craig:Just they just need it to work, and it was becoming a significant issue. I mean, they were experiencing significant downtime, inability to process credit cards, which, of course, is critical in today. Very nobody very few people use cash now.
Max:Could credit start because of technical issues being pushed up to corporate of, you know, we need help with this? Was it or was this a very much of like a P and L? I mean, franchise models, there's a certain percentage of revenue that's going back up to the franchise or to the corporate entity, right? So they just were getting overwhelmed with phone calls and people saying, My Internet's down. Can you help me?
Max:Or, can't process credit cards. Can you help me? My point of sales terminal is Or somebody in the finance department literally looking through and saying, why are sales down? Why are numbers down? Like, why is growth not here?
Craig:Yeah. Yeah. To a certain extent. I mean, there was a back and forth between the franchise owners and the corporate headquarters that say, hey. What's going on here?
Craig:Just the constant you know? And it was consistently having failures in their failover solution. So there was big laxes in the credit card transaction capability, sometimes being out for a day at a time. I mean, that's unacceptable, obviously. And so it got to the point where our trusted partner that we work with was engaging with the corporate the folks at corp and they were like, we need help, and we need to come up with a complete solution that takes this burden off the franchise owners because they're not equipped to be able to handle this.
Craig:Even though it's not an overly complex network, it's still something that it's not what their focus is. And to be able to provide something that takes that away from them and is being able to be managed by us or a trusted adviser is really what they needed to do. And interestingly, you know, a lot of times and I've worked on quite a few franchise type solutions where you've got large number of locations. There's there's a headquarters. And a lot of the times, the corporate entity will say, you don't have to do this, but we really suggest that you do that.
Craig:And it still ultimately ends up being up to the franchise owners what they wanna do. In this case, which is really interesting, corporate said, no. You are going to do this. There's not an option. This is a requirement now.
Craig:You can't go out and start choosing your own stuff. You are going to work with this solution that, you know, Spectra is putting together. And when that happened, I mean, it was really amazing because, again, you don't often see that. You still the franchise owners, especially the larger ones, because these are big businesses too. They still have that clout and the the ability to be able to, you know, make their own decisions, but in this case, it was not.
Craig:And So it was a directive from corporate that this is the stack that you're going to use. And they tested it. They had a lab. It was really exciting for us to be able to show that we're able to provide that service, and then they had the confidence in us to be able to send that directive out. So that was really it was really exciting for us.
Max:And I was just like getting back to this thing. If you're Home Depot and you own all of your locations, of course, you've got integrated IT systems and inventory and ERP and planning and point of sales talk like any major retailer. Right? Like, there is a corporate IT umbrella that gets extended down to retail locations to actually process transactions. In the franchise model, there's a very loose coupling within these things.
Max:And so for an organization to say, not only are we going to mandate what you're going to start doing here from an IT perspective, since we're mandated, we're now taking on the responsibility of it as well. Like there's a couple parts of that, right? It's like we're now giving you a directive, but we're also now taking on this as our problem. You know, that's part of what's nice about the franchise model, right? You kind have like a loose coupling.
Max:Yeah, exactly. So that's a pretty big shift. And I'm thinking through like the background of that and the decision making process of what that actually entails from a corporate perspective to say, no, we are going to give me a number of 1,500 locations. We are going to take on responsibility effectively for a decision across 1,500 locations plus several hundred per year in growth. Now this becomes our headache.
Max:You know, that's a lot. So they had some serious pain here if they decided like the remedy for this was to go out and define a provider that could help them with this.
Craig:Yeah. Absolutely. And we've been deploying these at a mad pace. I mean, we're constantly they're growing so fast as well. We needed to have a replicatable stack of services that could just be cookie cutter, and we can knock them out as quick as they're deploying and turning up these new locations.
Craig:And we talked to one of their larger franchise owners. It's not available yet, but it's a testimonial. And we're looking at probably two to three hours of downtime per week, maybe on average. Maybe that's a little high, but obviously significant, you know, across the locations that they were responsible for. And now after deploying our solution within the last three months that they've had service, they've had zero downtime.
Craig:And that just speaks volumes. I mean, think about the money that they're saving. Maybe they paid a little bit more for the service, but think about the value that brings to them and, you know, just being able to concentrate on their business, not having to worry about the network infrastructure always dropping, going up and down and like, oh, here we go again.
Max:I mean, that's a really it's an interesting thing. Right? Because you start talking about, like, quick service restaurants, even if you're serving breakfast, lunch, and dinner, and you have extended hours, you know, your opening window is sixteen hours a day, maybe. Right?
Craig:Yeah. Yeah.
Max:And if you're losing three hours of that window per day, I mean, terms of, like, a percentage of revenue, like not even accounting for busy times, like 2PM in the afternoon is gonna be very different for you than, like, noon if you're serving lunch.
Craig:Sure.
Max:So if you have an outage at lunch, like, the volume of transactions percentage wise is gonna be much higher. But even still at, like, an absolute percentage of if you're taking two to three hours a week of downtime, and that's a nightmare scenario.
Craig:It is. It can really affect the business. Obviously, it's negative, but, I mean, it can really affect them to the point of, wow. You know, we're having some trouble here. I mean, this is really causing us a lot of pain financially, time wise.
Craig:You know? And, of course, the reputation. I mean, can you imagine if you're a customer, you wanna go get your food, and get out. You know? You're not gonna sit there and look for the long two hour dinner kind of a thing.
Craig:You want in in and out. And if you're sitting there and they're like, oh, sorry. Our credit card hang on. We gotta automatically, you're just getting irritated as the customer. I'll go to the other place that's just down the road.
Craig:You know? So having that reliability, it really it's so important because it's obviously their reputation, not even obviously all the financials that go along with it, but, you know, just and anyone can put up a review and say, oh, don't go here because whatever. It's just really important, and it's just eye opening, you know, to hear when something's actually working well, and we're able to do that for them and help them with that. It's great, and we love it. And that's what we, you know, that's what we live for.
Max:Did they come in with a defined was there a defined target, a metric? We want it to be better. Like, that's great. But what you know, in order for this to actually be successful, it needs to be better in these ways. Had they already formulated that of what success was gonna look like?
Craig:We I think they knew they needed to get better, but the direction that they weren't sure what direction they wanted to take. We had a cost point, obviously, which, again, we don't like to focus on cost. It's always about the solution, but let's face it. Cost is a factor. It just it always ends up being one, maybe not the primary, but, you know, it's important that it was sustainable financially for them.
Craig:So they know they needed to fix the problem. I mean, it was obvious. But then it was like, what you need to tell us, SpectraTel, please tell us what you think is going to be the best solution for us because this is what we have. Different things, you know, hodgepodge, disparate connections, things aren't working. Please help us tell us what you think is going to be the best solution.
Craig:And so we started going through that process of designing a solution, sort of a a cafe in a box, let's say, with a with a stack of gear, a process that we could do quick cutovers. So they leaned on us very heavily to design the solution and to take them to where they needed to be because they didn't know. And that was really another important factor, our ability to be able to help them and consult with them and show them, hey. This is where the industry is. This is where we're headed.
Craig:This is how it works. This is really what's gonna make the best solution for you. When we put that together for them, it it made sense. And let's just sign up a dotted line. It we had to prove that would work.
Craig:So it was had to be tested. It had to, know, went through POC, labbing. And so it wasn't like just a quick these larger opportunities, obviously, it's never gonna be just a quick sign. Oh, yeah. This is great.
Craig:You've really gotta be able to demonstrate your ability to provide the service rather than just drawing it up and talking about it. That's critical component as well. So your question, they really didn't have an idea of where they needed to go. They just know it needed to get resolved. And they asked you know, they needed someone to really lead them into the next phase or the next or the direction into where to provide the services that they needed.
Max:So you have 14 1,500 locations. Fourteen, fifteen hundred locations. Each location could have a different POS terminal. Each location could have, like, different infrastructure, like wired, not wired, Wi Fi, not wired, different Wi Fi Yeah. Different types of equipment.
Max:So that's a pretty big kind of disparate. Was time spent trying to understand what the common at like, do we have a blueprint here of 80 you know, 60% already running this? Fill in the blank, and lean into that path? We're gonna push out this POS system, and this is how we're gonna need to integrate with it and what it actually needs. Or was this, like, more of a almost like a greenfielding of, okay.
Max:This is gonna become our standard going forward, and now it's just gonna be pushing us pushing the standard out to the field.
Craig:Yeah. It really was almost a greenfield environment because they they didn't have the expertise to be able to drill down and understand the specifics of what was gonna be required. So we were able to come in and just bring in a brand new service, a brand new environment for them that, you know, completely standardized the model, you know, their technical model. It was nice because, again, they didn't really have the services that you know, they didn't have a plan of, oh, this is what we want. This is the vendor we wanna use.
Craig:This is the technology we wanna use. There was none of that or very little of it. And so it really it gave us a lot of freedom to be able to build out what we really truly thought would be the best solution for them. And then, of course, ultimately, it ended up being that.
Max:What were the base requirements? There's a point of sale terminal or multiple. Right? So I go immediately to connectivity for that so it can process credit cards.
Craig:Yep.
Max:Was the design, these terminals are wireless. These are wired. It's gonna be this type of connectivity. We need this kind of infrastructure. We also wanna provide other services.
Max:Like, do we need to have a PC in the location for for the franchisee or the manager that's on-site to be able to do work? Is it employee time sheets and clock in and clock out added into this as well? How deep did this go? What were the requirements that you got? And then how did that then influence the design and, like, what tools you were bringing to bear?
Craig:Yeah. Absolutely. The first thing and there are a lot of segments to the network. Let's say there's a lot of devices on the network. So, you know, everything from desk phones, DVRs for security cameras, five to six registers, wireless POS systems, label printers, APs for guest Wi Fi, corporate Wi Fi, and laptops, PCs.
Craig:And so there were a lot of moving components to it.
Max:That's actually I mean, I love this part. Right? Because everybody's like, oh, it's like just whatever. Just a vanilla fast casual quick service restaurant. Right?
Max:And then you start rattling through. It's like, oh, we have phones and time clocks and registers and point of sale and cameras and access points and guest Wi Fi and printers and this and that. And you start, like, rattling through the list all of a sudden. I mean, just in this line, you've got, you know, two dozen critical pieces of equipment in order for this function for one location.
Craig:Yeah. So, you know, we started going through that inventory, and we first started to look at, obviously, connectivity. We wanted always up mentality, a % uptime. We were that was our focus. But in today's world, you know, when you're talking about connectivity and then you're talking about failover and then you bring SD WAN becomes a part of that discussion usually.
Craig:But another part, a big part of it is security. And the security component really is so important now. And, you know, things like SD WAN architectures and all that, they're and security are really becoming synonymous. I mean, they're becoming one and the same. You can't have one without the other.
Craig:So it wasn't that long ago where, you know, you'd have multiple devices or multiple pathways for, hey. I've got my failover, and then I've got my security, and I've got cloud security, whatever. You know? So it was our goal knowing that there were all these components and there's gonna be different segments to this network, you know, trusted network, which is untrusted networks that we needed to provide a standardized security system or environment for them along with being able to, you know, keep the network component up as much as possible. So that was really what we started to do is understand, okay.
Craig:We we really need to focus on the security piece, standardize that, but also be able to provide as much uptime as possible.
Max:And by the way, have a guest network too, which complicates things pretty significantly. Right? Yes. Outside of, like, what people can do on your guest network and how do you segment the guest network, you then run into issues of, like, well, how much bandwidth can the guest network consume? And how do you prioritize traffic so your credit card terminals can actually process transactions without somebody, like, streaming Netflix?
Max:I'll just pick on Netflix.
Craig:Yeah. Right.
Max:You know, off in the background. It could be YouTube. It could be something else. TikTok. You know?
Max:Everybody's uploading their TikTok videos.
Craig:Yeah. Of course. And that was one of the things as we started to, you know, dissect the network and the different components of different segments. We put together a topology that basically segmented that traffic out. And so we controlled just kinda talk about our SD WAN infrastructure.
Craig:Our FortiGate SD WAN solution is we actually have our own for virtual Fortinet SD WAN gateways throughout The US built out at Equinix data centers. So in a traditional because this was a Fortinet solution. We decided that was gonna be the way to go. It was very cost effective, very powerful, gave us a lot of control to be able to segment out these different components of the network. And so that is the direction we decided to go and ultimately proved beneficial.
Craig:Just to kinda talk about that architecture a little bit when you need to segment guest Wi Fi and POS transactions and things, we are able to provide that true SD WAN service for critical applications. So, the voice service, for example, which we're also providing. You know, that's gonna be a priority application that goes to the cloud gateway. POS transactions, obviously. You know, number one, traffic flow.
Craig:But for the guest Wi Fi comp components and things like that, that's not gonna be considered a priority traffic so that, you know, egress is out locally. We're not using it across the SD WAN, you know, overlay. And so we really got very granular as to how we're gonna segment this traffic and optimize the traffic flows to make sure the critical components of that environment are the focus. And so for example, if there's a failover condition because we have, you know, one broadband circuit and then one LTE backup. And so, obviously, you're not gonna want if there is a failover on the primary circuit, you're not gonna want guess Wi Fi cranking out on that LTE service.
Craig:That's, you know, not optimal. So we have that control as well where we said, okay. Only critical traffic, you know, credit cards, you know, UCaaS, voice, of course, that's gonna be you know, in a failover event, that's gonna go across the LTE. Other traffic would be restricted to go that. So we're very, you know, detail oriented when it came to understanding, you know, the critical points of the traffic flow and how it needed to be segmented and how it needed to be prioritized in the event that there was a failover to to the LTE service.
Max:I guess the concept becomes like hub and spoke. Right? And think of, like, a traditional network architecture and design. When new POS terminals were going out and DVRs were going out, UCaaS was going out, was this then a push into, you know, centralized resources where we have centralized data processing of some sort or UCaaS platform doesn't necessarily mean centralized location. They're usually, like, distributed where the phones are registering to.
Max:But, you know, DVR has to go from somewhere, for instance. Was that then rolled into a corporate umbrella as well, and was that part of this design?
Craig:Yeah. So there were different flows of the traffic. So some had to go to the, you know, the corporate. Most of it was cloud based, but there were some resources, you know, at the corporate where we had to filter that traffic. You know?
Craig:But most of their environment, you know, the POS environment is cloud based. So most of that traffic really gets segmented out based on where it needed to go. So, I mean, there were are some hub and spoke components to it, but most everything is, you know, in the cloud. So we were able to gateways were able to we peer that were part of the the peering partnerships that we have, you know, through Equinix and things. So we're able to optimize the, you know, the routes and minimal hops to those resources that they needed to have access to.
Craig:So their POS, their, you know, UCaaS, obviously, you know, those types of things.
Max:Did you help them with the PC equipment refreshes, the laptops as well? Or is that now corporate taking and influencing and pushing that down?
Craig:Yeah. No. We typically do not get involved in the PC support, you know, printer support. That's kind of where we draw the line. So we'll build out the LAN and WAN infrastructure, create the connectivity, monitor, you know, those environments.
Craig:But when it comes to the actual support of POS, PCs, printers, that's something that we do not do not do.
Max:1,500 locations, one broadband circuit, one LTE gateway, two firewall SD WAN appliances per location?
Craig:Just one. Just one? Okay.
Max:So firewall, a switch, at least one access point. So you've got four boxes per location plus Internet circuit times 1,500 locations. You're rolling out 6,000 devices for this customer. Yeah. Just from a logistics of ordering Internet circuits and managing 1,500.
Max:You know, like, the whole, like, network in a box concept is, that's great. We come up with a design. We're gonna have this firewall. It's gonna connect to this thing. We're gonna plug a switch into it.
Max:You come up with, you know, like, we have a mandate, you know, always up. We know we want credit card transactions always processed. Label printers are really important to us. Right? So we get into this idea of broadband, and I would assume it's cost of performance evaluation for broad availability.
Max:Right? Broadband and then an l a cellular backup. Right? Okay. Makes perfect sense.
Max:Then from there, you start working down. You say, okay. Now we have an SD WAN appliance for using Fortinet. We've got our virtual gateways. And then you round that out, you say, okay.
Max:We need switches and access points to fill this thing out. Right? So now you've got this idea of, like, a kit, you know, an architecture. How did you what was the validation process in the POC or POV process with the customer?
Craig:We put together basically a lab kit for them, and we had basically a success criteria. They had a store that was essentially a lab, at a headquarters, and they needed to be able to basically run a store right out of their labs. So testing the failover, that was, of course, a big one. Getting the we staged everything, had everything built, and then it came down to, let's see if this really works. So unplug that unplug that primary.
Craig:Does the failover work? Well, yeah. Great. It does. We could still process our credit card transactions.
Craig:And so there was a pretty detailed success criteria list, everything from security components, making sure content filtering works, again, making sure the failover worked properly, making sure, you know, traffic was segmented properly. Does you know, is the Wi Fi being segmented as it's as it should? And so we went through that, and we treated it just like an order. I mean, whenever we do a POC, we push it through. We process it just like it's a new onboarding of a customer.
Craig:So we start from the beginning. We go through the the official onboarding process, which includes implementation engineering, project manager, account manager. It goes through the complete order process of entering orders. Really, again, treating it just like a production order, not not a POC. And that way, you know, the customer is actually able to see, oh, this is how it works.
Craig:This is the cadence I'm gonna have with my project manager, for example. This is the cadence I'm gonna have with implementation engineering. This is how they process and document all the configurations that need to be recorded and configured. And so we really take them through that entire process so they understand it, and then we can check off all of the critical success criteria as far as what their requirements are.
Max:So go through the POC. They said this meets all of our, you know, tweaks come out, adjustments, whatever. They say, this is fantastic. Like, let's do this. Now you get into the real meat and potatoes stuff, which is you have 6,000 devices that have to go to 1,500 different locations plus Internet circuits have to be ordered in some cases probably and provisioned as well.
Max:That is not a happens overnight thing.
Craig:No.
Max:As that project kicked off, I immediately think and my brain goes like lots of different directions at the same time. Right? It becomes, are you taking and ordering equipment from OEM, ODM vendors and bringing it into a warehouse and prestaging it and then putting it into assemblies and shipping it out? Are you shipping it directly on-site? Are you providing instructions for somebody local to do it?
Max:Are you sending out your own engineering resources to go to the assembly? How do you determine timeline of how many stores at the same time at what geo markets? I can't help it. Like, my brain immediately just goes to, like, that. Walk me through that process for this with Spectra Tal.
Max:Like, you know, it's one thing to say, okay. Great. You know, we validate we can do this, but now you gotta get into we gotta actually make it work and push it out.
Craig:Yeah. Exactly. And, you know, being able to deliver on all of the what was designed and discussed, that's important. If you if you can't do that, then it's all for naught. So yeah.
Craig:So it all starts with introductions. You know, we assign premier account manager right off the bat. We assign a project manager right off the bat. And it's important because having that account man in lockstep with the with the project manager, that person's already aware of, you know, the project. It's not like, hey.
Craig:We've deployed 750 locations. Now it's time to bring in the account manager, and and they have to try and get up to speed after six months of work. So that's not really efficient. And so what we do is we get that account manager involved right at the beginning. So everybody's aware of what's going on.
Craig:And so it all starts with the project manager bringing all the stakeholders in. You know, that person coordinates with the key contacts with the customer, local contacts, etcetera. And then it comes into in parallel, what we do is we actually bring our implementation engineering folks in the deep dive design. And so they're working in parallel with the customers or franchise owners, you know, IT or technical contact, I should say, and really going through and documenting all of that. So once we get everything documented, we have the, of course, the circuit install.
Max:Mean, that's a lot of work. 1,500 locations. I mean, you're gonna have some franchise owners of multiple locations, but you're not talking about 50%. Right? I mean, it's, you know, so you're probably dealing with 1,200 different ownerships, you know, within this roughly.
Max:Right? Like
Craig:Yeah. I don't know if it's that many, but it's, yeah, it's a good number. I mean
Max:Even I mean, even 50. Seven hundred and 50.
Craig:Seven hundred
Max:50 ownerships. Right? Like, it's still 750 conversations you have to have.
Craig:Yeah. That's right. That's right. And so it starts with as the project manager is going through this, you know, we've got a lot of circuits being delivered. We've got a lot of, like you've mentioned, a lot of equipment because, you know, once the overall design is completed, because it's, again, we're replicating this package, this, you know, solution in a box, you know, so to speak.
Craig:Once that design is replicated, then it becomes, you know, kind of an assembly line of, okay. Here we go. We know it needs to be done, so we'll start staging the equipment. We have a lot of equipment in our in inventory before but for a solution like this, obviously, we need to, you know, purchase additional. So we get all that gear in.
Craig:Once all the configurations are documented, we'll start pushing in pushing the configuration, staging the equipment. We test every device before it goes out. So in our staging area, there's a, you know, a lab where we, you know, hook everything up, make sure it's working because what we'll do once it's ready once we're ready to turn up a location is we ship out that gear, and then we roll out a field technician to actually do the install. And they have punch list. I mean, a very literally step by step list of, you know, how everything needs to be configured.
Craig:So red cable one plugs into port one on this switch. Red you know, blue cable two. I mean, it's literally that documented and that, you know, detailed. And so literally any technician, any field tech that goes out there can, you know, open up the gets the punch list and, oh, okay. I can do this.
Craig:And then they coordinate, of course, with the our engineering folks in house that are doing the remote work and, you know, doing the configuration and and things like that.
Max:Taking equipment out of a box, putting it on a desk, plugging it in, turning it on, testing it, doing documentation. I mean, you have a standard config, right? But you still have to get the red cable and the blue cable and the black cable and put it into the box. Right? So you're going to have like you get pretty fast and have a thirty minutes per store Yeah.
Max:On a low end, right, up to an hour, probably. I mean, that's a lot of work if you're to replicate just the staging. Absolutely. So maybe you get really efficient at it. You're talking a thousand hours worth of engineering time just to stage all this equipment to go out the door.
Craig:Absolutely. I was up at our, you know, headquarters in December where our primary warehouse is, and I went back there. I always like to go back there and, you know, talk to the folks that are working there because I I work with them quite a bit. It's always nice to see them, and they're just testing equipment. They've got boxes.
Craig:They're putting them in. I mean, it literally is an assembly line, and they've got a group. Some groups focus on one thing, another group focuses on focuses on another. And it's really fun to watch because it's so efficient, and everyone really knows how it works and knows what needs to be done. And that's why I like to go in there and, you know, being a technical person.
Craig:It's always fun to see what's going on. And yeah. But it is. It's a lot of work. And I would say, you know, I've worked been in telecom or managed network services.
Craig:You know, it started out as telecom back in '99, you know, I've been doing this a while. But easily, this is Spectra tells, you know, implementation folks and project management is is absolutely unbelievable. I mean, they're so good, and they're so seasoned. We've brought folks in that are they're not rookies. I mean, they've been doing this a while for large carriers, and so everybody knows what needs to get done, and everyone gets their hands dirty.
Craig:It's not like, oh, well, like, that's so and so's problem. You know? Let me see if I can get ahold of them, and it just kinda holds things up. If there's really none of that here, everybody just gets in and gets it done because we know it's critical for us to be able to provide these services in a timely manner. So it's really it's a real joy to watch the folks work here.
Max:Can you go out and acquire 6,000 units of equipment, configure them, and install them? And how long is that gonna take your team? And just from a staging process, like I said, I mean, at a low end, you've got, let's just, I don't know, like, again, I'm sure it's north of a thousand hours of staging time. Right? Like how long is it going to take you in a thousand hours of engineering labor, by the way?
Max:That that thousand hours of engineering time can't be doing anything else at the same time. This is their sole function right for that thousand hours. Right. But like they have other jobs and for people, I try to explain this, even on the small end of like people being like, oh, I've got like, you know, a headquarters and then we've got, you know, fifteen, sixteen, 30 satellite offices and we roll out some network refresh. It's like, that is a lot more work than you anticipate it's gonna be.
Max:And, oh, by the way, you also have to fly to each one of those locations Yeah. To get this thing installed. So it's always impressive to me when I see these things at scale.
Craig:Yeah. Perfect example of that. I was working at a different opportunity really just a couple weeks ago. It's been going on for a while. And this customer, that's what they did.
Craig:They went out and bought a whole bunch of gear, a whole bunch of firewalls and switches, and it's literally been sitting, in boxes. Yeah.
Max:Yeah. It's in the box.
Craig:Yeah. And it's just been sitting there. And think about it. They purchased a year long license. So that license is already six months gone, and they haven't even used it yet.
Craig:And so they realized, yeah, we need some help. And so
Max:If you come from an engineering background, like, you wanna build stuff. Like, you just can't help it. Like, you're it's like, oh, I got this problem. I can go solve this problem. I'm go build it.
Max:Right? And the build versus buy, it's such a dangerous decision if you're when you're making it, if you're not really thinking about what comes next. Because we haven't even really gotten to the other stuff. Right? So I wanna actually keep talking through this a little bit here, we can circle to the other side of the next step of this.
Max:Right? But once a decision has been made, it's like we have a problem. It's creating pain. We want to fix it. Right?
Max:So there's like the first thing. Okay. We want to fix it. How do we fix it? And you go through the process, we figure out how to fix it.
Max:And then there's agreement like, okay, we know this fixes the problem. Let's fix the problem. Then you get into it. That turns into how quickly can we get this problem fixed? Because now everybody wants to fix the problem and then you just, you know, nobody wants the answer of like, Oh yeah, we'll fix this problem.
Max:It's going to take us three years, right, to fix this problem. What was the design goal from that point of like, we've got our POC, we have our kit, we know what the thing is. Like, what was your customer telling you in terms of timeline and roll out what they wanted to or trying to see? And what did you guys end up committing to? Like, what was
Craig:Yeah. So the first phase was going to be kinda target the locations. We're having the most issues in terms of uptime. So that was critical point was, hey. We've got a set or a subset of locations that really need to get this resolved ASAP.
Craig:Not every location was having trouble. I mean, let let's face it. You've got 1,500 sites. Some of them are gonna be working. So the focus was to hit those sites that were just constantly having problems.
Craig:And so that's what we did. We focused on those.
Max:How was that determined, like, ranking and ordering? I mean, were they at this point, it wasn't unified corporate. I guess not like they had, like, a ticketing system or some kind of help desk that was tracking. Was this survey? Was this, like
Craig:Yeah. So project manager was working with the corporate folks, and they did have a list actually of Mhmm. You know, some of the, you know, more challenging locations. Order at, like, a hundred of them, maybe, something like that. And so that was the target.
Craig:And so they knew which sites were having the most issues. And so it was actually pretty straightforward to be able to kinda target those. Then we developed a an overall project plan, typically twelve to eighteen months to get everything rolled out. But what we'll do is we'll decide we'll get the customer space. You know, we can pretty much go as quick as they want within reason.
Craig:So figure 10 sites per day, sometimes 15. And so we kinda develop the pattern per se or the schedule based on how they feel comfortable doing it and working with the franchise owners as well as the, you know, the corporate folks. And so we determine it that way. So the, you know, the biggest focal point, of course, was were those, you know, hundred or so I don't know if that's the exact number, but those hundred or so sites that were having the most downtime, the most issues. And then once those get rolled out, then we start focusing on the deployment based on because some franchise owners were a little reluctant.
Craig:They didn't really want to go this direction. They're like, my stuff works great. I don't wanna do this. But that was, again, the most exciting part about getting that directive from corporate that said, no. You have to do this because it works.
Craig:We wanna get this fixed right now. Put whatever in you want versus the other ones that were, like, a little reluctant. We scheduled it that way based on priority need.
Max:10 to 15 sites per day. Again, resources necessary in order to pull that off.
Craig:Yeah. And keep in mind that this is not the only project we have like this either.
Max:So, you know, we've got other projects. I'm I'm thinking about from, an end user perspective. Right?
Craig:Yeah. Yeah.
Max:You know? I mean, that's a lot of bodies in the field.
Craig:It is.
Max:A lot of bodies in staging a lot of coordination and a lot of project management. I mean, it's a simple thing that gets exponentially hard at scale. You know? One location, no problem, whatever. But it gets it's impressive.
Max:It gets this is an exponentially hard at scale problem.
Craig:Yeah. And this is really where we shine, and this is really where our sweet spot is. You know? Do we do the one two location opportunities? Of course.
Craig:But we really can show our value managing these large scale deployments, bringing everything into that single umbrella, and it just makes the customer's life so much easier. And this that's really where our value is, providing that single shoulder to hug, as we like to say, consolidated information through our new portal that we're rolling out, visibility into the network, proactive notifications, just handling everything from a proactive and predictive mode, I guess, rather than, you know, just kinda reactively like, oh, no. This thing is down again. Just kinda takes it to that next level and just opt allows them to operate so much more efficiently. And so they don't have to worry about the back end infrastructure of their network and POS and things like that.
Max:Okay. So as these things start rolling out, you move into operations and keeping them running. What does the requirement then look like and support? So NOC, SLA, equipment replacement. I mean, you're testing everything, so it shouldn't ship out broken, but you have a certain amount of infant mortality at a certain you put a thousand boxes in the field, and some percentage of those are just gonna smoke and explode on It's just statistics.
Max:Right? Now the good news is modern enterprise gear doesn't happen very often. Right? Like, is in the dark ages of network gear. But now you go into, like, an ongoing operational requirement to keep all this stuff running that now you have to deploy.
Max:What does that look like? I'm curious from in two angles. What does it look like from a SpectroTel standpoint? Like, what is your NOC services, your team, your SLA? And then the second part is how does that interaction with the customer then look like?
Max:Because you've got corporate and then all the franchisees. Right? So you've kinda got two masters also within that world that you're servicing.
Craig:Absolutely. So we monitor everything 24 by seven, three sixty five. We have two NOCs. Our primary NOC is located in Neptune, New Jersey at our headquarters, and our secondary NOC is located in Columbia, South Carolina. 1 of our biggest, if not our biggest initiative over the last eighteen months here at SpectroTel has been our AIOps platform and integrating the AI engine to be able to take in all these different data points from nodes out in the field.
Craig:We've got integration with the power grid. We're integrated with, you know, the weather service. So we're monitoring not just these nodes, not just these endpoints, but we're monitoring everything. So we know if we see, you weather alert, like a blizzard coming in in an Upstate New York or something, we're prepared to understand, like, hey. We're gonna see some issues here.
Craig:And so not too long ago, before we implemented our AIS platform, we had a literally a group of folks just sitting at desks trying to correlate all these automated monitoring events that were coming in and generating tickets. And so you could have multiple things going on at the same time, all for one all for one customer. And so we had to kinda figure out or just one location. We had to try to figure out, like, oh, okay. We had a node go down here.
Craig:Let's bring that up. This dropped because of this. And so trying to manage all that manually just through human beings was becoming unscalable. So what we've done and what we are continuing to do is automating so much of this and using, you know, artificial intelligence to be able to kind of proactively and even predictively monitor, you know, the footprint of all of our customers. And so and then once that data is put together through so we've got the AI engine, you know, monitoring the power grid correlating.
Craig:Hey. We've got a power event here, which led to, hey. This LTE modem dropped, which led to, hey. This Fortinet dropped. Putting all of that together in a single case, and then that gets moved over to our technicians to be able to look at.
Craig:And so you were very you were we're trying to move to, you know, not just a proactive model. You know, we're past reactive. We're we don't do reactive really much anymore because we're more proactive. And now we're looking at predictive and prescriptive, meaning wanna get ahead of issues before they actually happen. So for example, the perfect example is we saw six months ago a hurricane coming in.
Craig:We knew it was gonna be pretty catastrophic down in the Gulf Area or, you know, through where the Carolinas. And so we actually got, we sent out some, you know, rapid deploy kits with UPSs to be able to say, hey. We're gonna provide this for you because we know you've got a critical locations getting that out there because we're pretty certain there were just gonna be some pretty catastrophic outages. We do think we're doing things like that. We send notifications to, you know, customers that can be, you know, folks at the headquarters, for example.
Craig:In this case, it could be the franchise owners, and you basically tell us how you wanna be contacted. Text, email, who are the important people that are gonna be contacted. And then once that case is opened either via automation or if it's like a manual entry, the end users have access to everything, almost everything. So they'll be able to see the technicians working on the issues. They'll be able to see all the notes in real time, all the events.
Craig:And so we're very transparent when it comes to that. So they have almost complete visibility into everything that we see. Giving them we're working on this, and it gives them, you know, the peace of mind to know that, hey. Spectra Charles is already on this. I don't need to call them and say, hey.
Craig:We can't process credit card transactions. What's going on? They know where to who to call. They know that we're working on it. It makes it a really good experience for the customer.
Craig:And we also take we've also implemented in this particular opportunity that we've been talking about. We also put in smart power devices. So these devices, they're basically Watt boxes. So they're essentially smart power strips that we plug into you you plug it into the LAN, and it monitors the nodes, the points on the LAN for connectivity. So if it detects a failure, for example, on the primary circuit, it loses connection to that modem.
Craig:It'll automatically reboot that that power outlet to say because a lot of issues can just be resolved with a single reboot. Right?
Max:I think that it's true. I mean, it's probably the most frustrating thing as somebody in IT space of, like, the reboot actually does solve most of the problems. Yeah.
Craig:And it's just like And we're still there. It's it's still
Max:a
Craig:thing.
Max:So true. I hate that it's true so much. Yeah. Yes.
Craig:Yeah. So in that talking about that, we've put as much effort as we could to kind of, again, be ahead of the game, not being reactive. And so they're not waiting for the customer to call, and it's like, oh, you know, something's down. And and all that's all time interval. Right?
Craig:So the communication, calling in, the tech has to look at it. You're talking about an increased time to repair. Right? So with all of these things that we've built in with our AIOps platform with the smart power, it just cuts way down on any issues. And so it means more uptime, less back and forth communication, quicker times to resolutions, and so forth, which really just, again, just makes the customer experience as good as possible.
Max:I grew up in Southern California and worked with a lot of West Coast based businesses for my career. And in SoCal, we deal with the impending, there's always going be the 10 earthquake that's going to take down and create a new island, all these different things. The ocean's going to move into Phoenix.
Craig:Yeah.
Max:But if you're not operating, you're not used to operating at scale distributed throughout The US, you don't realize, like, how much weather in different areas really does consistently impact your operations. And the big ones, of course, are hurricanes. And we have hurricane season. And if you're in certain parts along the Gulf or on the Atlantic Seaboard, maybe it's just a travel question, right? Like a little storm that comes through that's still really disruptive or Denver has bomb blizzards.
Max:Like it's the Northeast has weather and it's strange to say this, but I see that being very valuable also just from a, maybe you're not thinking about it. You can see situations where people are just going about their days and not thinking about the fact that there's a storm brewing off the coast. And that's going to impact 300 locations because that's the predictive track. Getting a text message from somebody that says, we're monitoring the storm, and this is the track, and this is the assets that can be potentially impacted. And this is what we're doing to mobilize resources in advance.
Max:I mean, that's gonna be a really comforting feeling for somebody to receive that message from you.
Craig:Absolutely. And, again, this is it's all about us getting ahead of that, getting ahead of what's coming and be being able to provide the best service to the customer. And it is. It's comforting to them to know that, hey. We're monitoring this.
Craig:We're on top of it. Are we gonna be able to resolve every issue caused by a storm? Of course not. But for those critical those critical sites where we need to have absolute uptime, whether it's a hospital or some of these emission critical life saving type services, we can get ahead of that, and we can take steps to support the customer before it even happens or as it's happening. And so that's really important today.
Max:So what does, like, post deployment cadence look like? And when I say that, some companies operationalize this as a KBR. I think of things like this location is having these issues consistently or that we're remediating or we're seeing this pattern or bandwidth at this these sets of locations or this, or we're seeing this kind of pattern in your network. Or what is that, like, dialogue turn into after these things start getting rolled out? I mean, is it, hey, franchisee, I don't know what's going on in your location, but you're always saturating your broad this broadband connection because of your guest Wi Fi, and you should we need to do something about it.
Max:Or is that does this become like aggregated med meta policies that can apply at a corporate level in discussion? Because you're not the IT team. You're not making the decisions, but now you've got to empower people to make informed decisions that are really like your recommendations. So how does that cadence work?
Craig:Yeah. That that's a great point. And a lot of that now we're building into our next generation portal. And so all of that visibility and look. I totally understand that these franchise owners, and this kinda goes back to what we were talking about, they're not IT folks.
Craig:They wanna focus on their business. So they're not we don't expect them to be in our portal looking at everything every day on their footprint. It's not realistic. But what we're doing is because we're armed with that information, we can send out notifications to them either passively, like through an email or whatever. But and then we also have the premier account manager that is literally their point of contact for specific items.
Craig:And so that account manager in real time, not just quarterly or twice a year, in real time will have this information basically saying, hey. You're peaking at you're running at 98% capacity on this on this access circuit, or we're seeing that you're getting a flooded data flows into the firewall eating up the CPU time. We need to we need to look at this. And so that happens in real time with the premier account manager. We also do the more formal QBRs talking about business business in general and just running through how general operational type things.
Craig:So we do that as well. But the most important piece, in my opinion, is having that premier account manager working in lockstep with that customer every day. And it's not just a, hey. We'll contact you every four times a year to kinda go over things. We like to stay on top of things, and that's really important to provide, you know, providing that fantastic customer experience.
Max:Have you pushed outside of, like, IT operations into sales and marketing? I mean, now you've got like guest Wi Fi opens up interesting avenues, repeat visits, average duration of visit, density of visit. I mean, without even without doing anything else, you get a certain amount of you get data, right? Like you get a certain amount of data that just comes out of these platforms that can be aggregated and distilled and turned into business intelligence.
Craig:Yeah. Absolutely. We didn't do that specifically here, but we do offer that as a service. We have that implemented for some of our other larger customers. We're tracking the presence data.
Craig:We're tracking the what are they looking at through a Wi Fi overlay platform. So we didn't do that here, but we do that. And it is can be extremely valuable to get that data, to be able to provide their marketing teams competitive data, localized data for foot traffic, things like that. But, yeah, we do that, but it wasn't a part of this particular scope.
Max:For somebody watching or listening to this, what else comes up in this conversation for somebody that hasn't been down this road before? There's the core of it, right, which is I can't conduct business. I can't transact credit cards and I can't. And I'm now having a significant percentage of downtime and lost revenue from it. Right?
Max:So it's not just lost revenue becomes lost revenue, but it's loss of goodwill. It's adjustment of consumer behavior. They go somewhere else and maybe they don't come back. Those things are pretty big issues for our business.
Craig:Sure.
Max:What have we not talked about or what have we not covered that somebody who's not in the IT or telecom space really, like, cares about in the sense of, like, when you're going through this from start to finish and deploying these things and you're walking somebody through this process for the first time that people should know about.
Craig:Yeah. I think just understanding where we are from a networking infrastructure perspective, I think, is very important because I think, especially for these larger multi location customers that are not really focused on security, for example, that time's obvious. Right? Everybody wants to be up. But I think security really is very important, and I think it's an afterthought for a lot of customers and for a lot of end users.
Craig:And I haven't been I haven't been hacked, and it's not a big deal. It's just all in the news. And it that's totally not the case, and it's very important. I would tell anyone that kinda doesn't really pay too much attention to their security posture, whether you're a two location business or whether you're a 2,000 location business, having that security posture in place that protects you from all of the different threats that are out there, and there's thousands of them. I mean, there's literally businesses in other countries and even here that all they do is focus on hacking sensitive data and ransomware.
Craig:And it is so absolutely critical that you harden your network. And it happens very frequently where I brought into an opportunity, and it's like, oh, we don't really have any security in place. And I'm just like, wow. Okay. So, yeah, that to me is literally it's just so important because all it takes is one event, one ransomware, one DDoS attack, and Mhmm.
Craig:Your credibility is gone. Your financials can just go through the roof to try and resolve it, and it's just it's so important today.
Max:The industry is now kind of referring to this as like a risk and resilience. So backup disaster recovery, security, all these things kind of fit into that pot into that thing. And I was in a conversation last night about it, and really, it's those have had the event and those that are about to have the event. Right? Like, there's it's unfortunate to to say it that way and to sound I don't sound, like, fatalistic about it, but at some point, you're gonna lose data.
Max:Like, you don't have a and the question is, do you have a way to recover that data? And if you have a way to recover that data, it's a much better situation for you. Right? And the thing with hacking, especially dealing and talking with, like, smaller businesses or people operating smaller footprints, Right? In this case, a retail location.
Max:They think about hacking like somebody's gonna break into my server and try to steal data from me or, like, break into my bank account and, like, well, like, if they get access to your system, they can get access to your credit cards that are flowing through, which has a real risk to you. Or, you know, they can just do a denial of service. People think, like Yeah. Denial of service is also like, I lock you out of your platform so you can't do business. And how many days does it take you before that's a really big problem?
Max:Right?
Craig:Exactly. And a lot of the a lot of the bad actors out there are just doing that for fun. Even doing it to get money after, like, oh, let's see who we can mess up today. You know what mean? It's just unbelievable.
Craig:I started a server twenty years ago when I was doing online stuff, and the person told me, like, oh, be careful. You're probably gonna get hacked. I was like, yeah. Whatever. It's not gonna hack me.
Craig:I'll tell you, in one week, they had already hacked my server. I couldn't believe it. I was like, wow. Okay. So and this was twenty years ago.
Max:Yeah. Yeah. It's just if you're connected to the Internet, it's
Craig:there. It's definitely running. Like the the Wild West.
Max:I don't wish bad outcomes for anybody. I really don't. Again, if you're watching or listening to this and you haven't had an event yet, I hope your first one is relatively minor. I really do. Because the experiences and stories that I've been I've seen them in part of, like, they get really intense really fast.
Craig:Yeah. They do. But,
Max:thank you for sharing this with me. This is amazing. I love talking about this kind of stuff. And it's really impressive to be able to zoom in, but then to zoom out and say, Okay, here's a deployment. 6,000 devices at 1,500 locations, a clip of 10 to 15 per day.
Max:You know, but not only is it getting installed properly and tested so people can conduct business, but then monitored and maintained and proactively cared for. That's fantastic scale. It's very impressive.
Craig:Yeah. Thanks. I really appreciate you having me here, and it's been a great experience, and I really appreciate it. Thanks for having us.
