Protecting Your Business: Preventing CircleCI Cybersecurity Breaches
Hi. I'm Max Clark. Let's be a little rant on cybersecurity. And I mean, we'll just call it what it is. Don't let CircleCI happen to you.
Speaker 1:Let's just use that language because I was talking about the other day. And, yes, I'm calling on CircleCI and I feel for CircleCI and the point here is CircleCI, if you don't know who they are, they do continuous integration and deployment. It's a it's a hosted CICD tool. It's really good. We have tons of clients using it.
Speaker 1:It's fantastic. If you're looking for a CICD tool, take a look at CircleCI. They had a breach, and there's a lot of things about the breach. But let's talk about the first thing about the breach. The first thing about the breach is they found out about the breach from one of their customers.
Speaker 1:One of their customers notified them that they were seeing strange things and they had tracked those strange things back to I believe it was tokens that they had released and given on the CircleCI platform. And so they were pretty sure that something had happened and it's come from the CircleCI platform. And, basically, hey, what's going on? Right? And CircleCI to their credit investigated, identified, and resolved this very quickly and then published a, like, I don't want to call it root cause but, like, basically, I I what happened blog post, by the way.
Speaker 1:Getting all this information from. I don't have any information about what happened here other than what CircleCI is published publicly. But if you read through the what happened post and I'm not reading it right now, so I'm paraphrasing from memory, so I'm probably gonna get some details wrong. Go ahead and flame me for it in the comments. But the point being is a developer executed a payload on their device.
Speaker 1:Right? Wasn't clear whether I don't remember if it was email or if they clicked a link and ended up on a web page or whatever happened but something happened in effect to their computer which became the source of the compromise. Have you ever heard of this happening before? Somebody having and clicking on something or opening something and something bad happening as a result of it? Like, rate basically, like, over the overwhelming majority of all, you know, security incidents going back is when I got into IT 25 years ago.
Speaker 1:Now that in itself we we can we'll come I'll come back to this. The next description is that their antivirus tool didn't catch this. Okay. By the way, the language is really specific around their antivirus tool And this is a huge red flag. Like, if you're running something that identifies itself as an antivirus tool at this point, throw it out.
Speaker 1:Turn it off. It doesn't do anything for you. It's worthless. Get rid of it. Antivirus tools work by applying signature matches for, you know, an executable.
Speaker 1:Right? So, hey, we scan this file and it matches a signature. We know the signature is bad, so we're gonna prevent that from running. Right? That's antivirus.
Speaker 1:Problem with antivirus systems is people that are writing and creating these bad software have access to the same data. You know, the the actual underlying, you know, databases there that are going into and doing pattern matching on antivirus. And then just all you have to do is use modify the signature a little bit and you get past it. Okay. It's overly simplistic but that's the point.
Speaker 1:Okay. Anyways, their antivirus didn't catch it. The attackers then were able to exfiltrate data and credentials off the developers machine. They were then able to connect into the production platform using these credentials. By the way, the connection to the production platform and as these systems came through a VPN.
Speaker 1:Right? They masked their actual source and location and it connected to a VPN and they connected in and then they were able to get into the platform and do all sorts of bad things which resulted in ultimately one of circle c i's customers saying, hey, this has happened. What's going Them doing the investigation, massive amount of trauma and discomfort for them, massive issues for their customers. By the way, like, my clients using CircleCI, I mean, they had to go through and and rotate every single key and token that they had that was deploying inside of the CircleCI system. So, I mean, just identifying and figuring that out was a lot of work, you know.
Speaker 1:So, collectively, I mean, you know, a a million man hours. I mean, who knows? Like, what actually was invested, you know, in terms of the downstream effects of this in terms of cost and impact? I think our average client remediation was something in terms of, like, you know, 3 to 4 months. And this multiple sprint cycles and engineering time and DevOps time and then having to go through dealing with the SOC 2 controls and auditors and all these different things to deal with this.
Speaker 1:So just from, you know, our vantage point of this, the actual downstream effects of of this were pretty significant. Okay. Anyway, so bad things happen. Now when I say, like, there's lots of things. And and, of course, again, reading what was publicly disclosed, you know, and reading between the lines and deferring things.
Speaker 1:Right? You know, some probably correct, mostly correct, some probably wrong, but let's just talk through it. Right? So the first thing is is doesn't seem that there was any active protection running on their workstation to do things. Right?
Speaker 1:So was an email source compromise? Were they running a, you know, so we call that more the industry calls it business email compromise or BEC. BEC. Sounds like a cat vomiting. BEC.
Speaker 1:And for the people watching this, they go, oh, like, oh, we're running Google Workspace. We're okay. Or we're on Office 365. We're secure. No.
Speaker 1:You're not. I'm sorry. You are not. Send me an email. Give me a call.
Speaker 1:We'll get you on a demo of a, you know, an overlay tool on top of these platforms, and you can, you know, run it. You don't actually have to run it in line. You could just run-in journaling mode for 30 days and get the report afterwards of, like, this is what got through the platform's filters that we actually detected and blocked and it's gotten caught and see it. And it is frightening. Okay?
Speaker 1:So, yes, Google Workspace, Microsoft 365 as a baseline is better than nothing in terms of, like, compared to what's actually going on. It's a massive problem. Okay. So there anyway, so there's an email vector. And then there's also the browsing the Internet.
Speaker 1:Just clicking on links vector. Was there secure web gateway? Was it doing remote browser inspection? The answer to both of those are probably no. Why?
Speaker 1:Because they didn't mention it. A, you know, I mean, they they call it their antivirus but they don't call out the fact that there's secure web gateway and RBI system didn't catch this thing and they got a payload deployed on the machine. We're in a secure web gateway. There's really good ones. By the way, if you don't wanna go all the way into, you know, a SWIG with Sassy infrastructure, you can do it DNS based.
Speaker 1:And here's a free tip, and it's free. Quad 9. Like, 9.9.9. Quad 9. It's a nonprofit.
Speaker 1:It's DNS based filtering. You can drastically improve your security posture of your users and their devices just by changing their DNS entries for their resolvers. So is Quad 9 as good as a commercial swig? No. It's really good though.
Speaker 1:And as an entry point, like, just turn it on and run it. Like, again, free pro tip, you know, like like Max's tip of the day here. Go out and just turn on run quad 9. Like, do it. Like, right now like, press pause.
Speaker 1:Go out and do it. Okay. Don't run an antivirus. I mean, seriously. Right?
Speaker 1:Don't. You need an EDR. Right? You know, some of them brand themselves as advanced antivirus because it's easy for, you know, lay people to, like, understand, like, oh, there was, like, old antivirus. Now there's new antivirus.
Speaker 1:I want the new one because it's better. It's endpoint detection response. You need an EDR tool. Right? We can get into an argument about which EDR is better or why or or all these different things and and, honestly, it almost doesn't matter.
Speaker 1:Pick when you like, run it or even better if you need help deploying and managing these things which you probably do because it's a lot of workload, your MDR vendor, and we can help you pick which MDR vendor you're gonna you know, service provider you're gonna use, your MDR vendor preferred EDR integrations. If you're on, you know, Office 365 and you wanna run E5 Security and run Defender, great. You know, there's a whole ecosystem of really good service providers just to help you roll out E5 Security for your platform. If you wanna run CrowdStrike, if you wanna run Sentinel 1, if you wanna run Carbon Black, if you wanna run I mean, the point is get one and get help implementing it and managing it and making sure that it's doing what it's supposed to be doing and you're getting the data out of it that you wanna do. And by the way, EDRs give you great information like, hey, something's running on this box that has never run before that's doing something really weird and you should take a look at it.
Speaker 1:Right? Like, so even if you get that infection or that compromise, even if you don't, like, for some reason, catch the fact that something's going on, at least you can figure what the heck happened and you can deal with it as opposed to your customer calling you. And and this is part of the reason why I'm giving so much, you know, like, I'm I'm making this a point, without your customer calling you to tell you that something happened. Right? You don't ever want that to happen.
Speaker 1:You don't want somebody else telling you you've got a problem in your house. You wanna know about it. You don't want them telling you about it. Okay. So that's the second thing here with this.
Speaker 1:The third thing with this is, o m g, they were able to use the credentials over some rando third party VPN and connect and get into production infrastructure. Like, oh, seriously? If you're running a SaaS tool, you can limit access to your SaaS tool by using a CASB. Right? And if your SaaS tool doesn't support using, you know, a CASB integration and and limiting, you know, remote IP addresses, get a different SaaS tool.
Speaker 1:Like, I'm sorry. It's just not it should be thrown out. Right? Any adequate SaaS platform is gonna give you the ability to integrate and use. And not even do you know it has to be, like, full CASB.
Speaker 1:It can be CASB lite. Right? But and I think it's incredible about what I'm talking about here in terms of layered infrastructure. You know, having a CASB limit access to your SaaS tool. If you go into an SSC or or Sassy platform, you're getting the CASB as well as the SWIG, the secure web gateway, as well as the ability to do RBI and that usually is gonna integrate with the seats gonna like, we're building here in terms of, like, layering up solutions.
Speaker 1:Okay. So maybe it wasn't, you know, like, they're something else was going on and they were, you know, the it's not as it wasn't a SaaS tool. Right? It wasn't just public access where everybody was connecting into their system and they actually had, like, partition, you know, like, some other remote access. It's called ZTNA, 0 trust network access or a SDP software defined perimeter.
Speaker 1:The fact that it was, like, anybody could authenticate from anywhere on the planet and connect into this thing and it wasn't noticed, like, hey, something weird is going on here and we're blocking access to it. Like, come on. You know, like, again, you know, Google wrote, you know, the BeyondCorp spec, like, I I mean, a decade plus ago at this point. Like, this isn't like, ZTNA is not some acronym that came out of nowhere. Like, get a secure remote access system that has ZTNA functionality into it, integrate I mean, and you can do it again, SSE and Sassy.
Speaker 1:They can all integrate this stuff together or out you know, or go out and get a standalone. And make sure you're getting policy enforcement, you know, for your entitlements. Why the heck was this credential allowed to connect? A, from a source that had never been seen before wasn't authorized, probably from a different geo that wasn't where this person with the credentials were. Why didn't that then have to authenticate through their IDP?
Speaker 1:Why wasn't there MFA in place? Let's, like, count up the steps here. Right? And I'm using this as an example just in terms of like, if you come back a few steps, probably any one of these things would have prevented this from happening. Now, if you're a SaaS platform and you have public customer, you know, like lots of customers and public data, like, your threat profile goes way up.
Speaker 1:You know, if you're just an enterprise maintaining your own intellectual property that's massively important to you in terms of trade secrets, right, or if you got, you know, protect your bank accounts or your manufacturing equipment or whatever it is, like, okay, fine. If you have customer data, that customer data becomes valuable. Right? So, you know, CircleCI, maybe it's like, okay, is there a bank using CircleCI to do deployments? Right?
Speaker 1:Is there a pharmaceutical company doing, you know, drug research? Is there a manufacturing company? A defense I mean, like, go through that list here. And now all of a sudden you say, okay, what's your actual risk profile? And then you say, okay, well, why didn't you have anything in place?
Speaker 1:And this is why I'm calling it out so much. Right? But the point here is don't be this. Do not be this. Right?
Speaker 1:Like, yeah. Okay. Some of the stuff I won't even put it in the in the category of expensive. You know? It adds cost but in the grand scheme of things of how much cost as per ads, how much cost it adds per user is pretty minimal.
Speaker 1:Oh, it's a pain to have to use MFA to authenticate, like, deal with it. You know? I mean, seriously, whatever. You know? Like, if you can't be bothered to look at your authenticator app or whatever you're using and, like, type in a 6 digit code or plug your security key into your USB port and press the button on it to, like, authenticate, like, you know, you can't use that as an excuse with me anymore.
Speaker 1:This isn't an undue burden for you. It's just not. Don't claim it is. And you don't have to do everything. Like, would it be better for you if you had an EDR, plus a swig, plus RBI, plus, you know, CASB plus ZTNA, plus, you know, plus plus plus plus.
Speaker 1:Of course, the more things you put into the mix, the better off you are because now you have defense in-depth and layers. Right? One system doesn't catch it. You've got other systems behind it to catch it, and you have layers of security to protect yourself. But even without defense in-depth and having all these different tools, like, start down the journey.
Speaker 1:Like, the best time to do this was yesterday. The second best time is today. You know? If you're not running an EDR, if you do not have an MDR, if you do not have a service provider helping you with your MDR, like, get one. Today is a great day to do it.
Speaker 1:If you need help with it, we will help you. Call me. Call me. Send me an email. Anyways, I'm gonna stop harping on this at this point, but don't let this happen to you.
Speaker 1:You know, like, it's not gonna happen to us. Yes. It's gonna happen to you. At some point, it's gonna happen to you. At some point, something bad is gonna happen to you.
Speaker 1:You're connected to the Internet. It's just a ticking time bomb. You know, it will happen to you at some point. Don't be the, like, oh, we didn't do anything about it because we didn't know any better. Of course, you know better.
Speaker 1:Just don't let it happen to you. Don't be that company. Anyways, I'm expecting hate below. Just go ahead and go for it. You know, tell me I'm wrong.
Speaker 1:You know, whatever. Hate below in the comments. If you if you wanna chat, reach out. Give me a call. Send me an email.
Speaker 1:Happy to help. Happy to talk to you. Hope this helps you. Seriously, go out. Get the tooling.
Speaker 1:Get the help. Don't let this happen to you. Have a great day.
