EPISODE 1617

[INTRODUCTION]

[0:00:00] ANNOUNCER: Ransomware attacks involve the deployment of malware that blocks access to a user's, or organization's computer files by encrypting them. The attackers then demand a ransom payment in exchange for the decryption key that will restore access to the files. These attacks are often directed at governments and corporations, and can be costly. Veeam is a data storage system that was designed specifically to provide protection against ransomware attacks. Object First is a system that works with Veeam to increase its safety and security.

Anthony Cusimano is the Director of Technical Marketing at Object First, and he joins the podcast to talk about the growing sophistication of ransomware attacks and the emerging technologies to block them.

[0:00:44] ANNOUNCER: This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale, is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments. Lee is the host of his podcast, Modern Digital Business, produced for people looking to build and grow their digital business. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com and see all his content at leeatchison.com.

[INTERVIEW]

[0:01:31] LA: Anthony Cusimano is Director of Technical Marketing at Object First, and he is my guest today. Anthony, welcome to Software Engineering Daily.

[0:01:39] AC: Thank you so much for having me. I'm so excited to be here.

[0:01:42] LA: I'm so glad you're here. This is great. I've been really looking forward to this conversation. I want to start off just with a basic definition here. What is a ransomware attack? Really, how does it differ from other types of attacks?

[0:01:56] AC: When we were doing a little bit of pre-show, I was thinking about this question specifically, because I think everyone has an idea of what an attack is. It's always the whole like, oh, yeah. Viruses lock down your files. Prepare to pay some Bitcoin. It's changed a lot in the last five years. I was actually a victim of identity theft seven years ago. This is seven-plus now. Things have just become different. It used to be, they go after people like you and I.

Ransomware, just breaking down the words. You're paying a ransom and some kind of malicious software is generally the cause, right? It used to be, how can I get that malicious software into yours, mine, our parents, whomever’s computer and extort a little bit of cash out of there? That has changed drastically. That's why I love this question. It's no longer about the little fish. Ransomware is synonymous with big organizational attacks. How do I spread and become the worst possible thing to ever happen to your company in the entire history of your operating business? Truly, it is just, I almost feel like we need a new word for it, because it's not the ransomware of yesterday. Now, it is just so much more pervasive.

There's a lot more arms to it. It's not always software now. There's a lot of just bad actors in the system, either converting local employees to do no good, because they're disgruntled, or they're cleverly integrating and getting files and malicious entities in the environment. It's so huge and it's so big and it's so nasty.

[0:03:34] LA: It sounds like, it's a lot more encompassing, right? I mean, so –

[0:03:37] AC: It is.

[0:03:38] LA: It's not just, like you say, software that shuts down your system and prevents access. Still, fundamentally, it's some form of denial of service, or denial of access to information. Is that what it normally is?

[0:03:51] AC: Sometimes. Sometimes it's not even a denial of information. I mean, recently we've seen a lot of just – it's general extortion. “Hey, you still have access to your files, so do we. We're going to leak this list of sensitive customers, or medical records, or what have you if you don't pay.” Just getting information and extorting a company is a form of ransomware. You're still holding them hostage and trying to put their feet to the flame to pay. Nothing too malicious has happened on the encryption side. You could actually go about committing ransomware type attacks in lots of different ways and not all of which require a malicious software tied to it.

A disgruntled employee is a perfect example. If somehow, they have the ability to say, “Hey, I'm not happy with the company. I'm going to leak your files, or I'm going to do something.” They could pose as some online attacker. Only infiltration is happening with human personnel. It's scary. I think it's going to get a lot worse with AI, but time will tell on that one.

[0:04:50] LA: I think we've just introduced three or four different sub-topics here that we can get into a lot more depth. Let's start with some of that and say, so given that ransomware is a lot more than just keeping you from getting access to your files, are the techniques you use for stopping ransomware different depending on the type of attack?

[0:05:09] AC: Yes, it really is. We talk a lot about zero trust in our industry. The idea that you should just assume there's always a breach. You should do your best to have as many points of encryption and protocol checking and ensuring you are who you say you are. When you're developing your software, or putting together architectures and infrastructure, there's lots of little checks along the way that can help prevent it. It's a sum of the parts as a whole. Even that is not good enough sometimes.

It's like antivirus software, right? That used to be all you needed. Hey, as long as I was running some paid antivirus software, I was fine. Now, the definitions can't keep up with the latest ways that people are getting in and infiltrating and attacking. What I say is actually, live in a state of constant skepticism. Be a little bit paranoid. Assume you are breached at all times. Then keep up with the news. Look at what's going on. We've had some pretty major attacks this year. We saw a casino amongst other organizations. I think an airline organization was also attacked. They could go after a really big fish.

You look at the way they get in. If they publish how they get in, it's always interesting because it's never what you expect. That's the nuance of it is if you assume you're always breached, which you should assume you are, they're going to be coming in through the vector you least expect. They're not going to try and go through your firewall, because they know you've done everything in your possible power to keep that firewall up to date. They're going to wait for someone to plug in something into a USB port, or they're going to wait for someone's password to be compromised, because they posted too much information on social media. It's always the thing you least expect, and that's when it spawns and does the damage.

[0:06:50] LA: That's actually part of the problem with most of the preventative mechanisms is that you can put all your effort in. Unless, you fill all of the holes, which is impossible to do, all it takes is a one little hole that the bad actors know about and everything else doesn't matter anymore, right?

[0:07:09] AC: That's right.

[0:07:10] LA: That's one of the problems with the brute force. I'll just put up a firewall. I'll just put up this. I'll put up that. I'll block people from getting into my systems. What you're advocating instead, which I completely agree with, and I know it's really the best practice nowadays is just assume that somebody has gotten in and deal with things correctly that way. That's more of a reactive stance. That's really what's required here, because you can't proactively stop people from doing things that you don't yet know what they're capable of doing.

[0:07:41] AC: Exactly. It's a terrible piece of advice to give. Like, oh, assume you've already lost the fight. I do believe when you have that mentality, it forces you to take inventory on what they could get. That's what's important is prevention. It's like a virus that we experience as human beings. You could do everything in your power. Wash your hands, wear a mask, do everything. It's the one time you walk into that one room that someone sneezed in and you let your guard down, and that's when it happens. It's better to be prepared for the worst. Assume they're in there. Assume they're going to get everything. What's the worst thing they can get? How can you make that less bad?

You get into a mind shift of let me implement data masking. Let me make sure that all of my data has some level of encryption. Even if they have it, if they don't get through our points of protection and ensuring that we validate who they are, they say they are, all they have is just a cache. You're not going to be able to hold me hostage if you just have a cache of information you can't do anything with, that the best encryption tools can't crack. You start to think, okay, if they take everything, what does that mean? Then, how do I make everything not valuable to them?

Many times, what this means to the IT admin, or whoever is in charge of this is you're introducing lots of points of contention in your workday. Because it's just, I got five-FA on everything now and I'm ensuring that anytime I get access to my databases, I have to go through some validation and I have to de-encrypt, and it's more time and it's more energy. Does that offset the cost of paying 8 million something dollars in Bitcoin to ensure they don't leak your data to the public? I would probably guess, it's yes is the answer.

[0:09:26] LA: Yes. Yeah, that makes sense. Let's talk about why ransomware though. Your bad actors exist. I think the internal versus external bad actor, I think we want to definitely get to that conversation point and what's different between them. Just bad actors in general, why – bad actors trying to choose some way of attacking and why would they choose ransomware over some other form? What is it about ransomware that's making it so popular nowadays?

[0:09:54] AC: Well, and I think that goes to a little bit of the point I made earlier is ransomware has become such a nebulous phrase. It's not just malware. It's not just a worm that's infiltrating your email. It's lots of different vectors. Ransomware is a catch all term. Personally, I think that it's why I think we need a new definition. We're going to start to see attacks that come through the vector of human puppetry through AI. I truly think that what we've seen the last couple of months, people being puppeted, or using some kind of deep learning face AI where you're just faking someone's identity and you're going to be able to convince people to do things as a fake human being. Is that ransomware, or is that some new, terrible way of getting through? I think it'll probably fall into the catch all term.

Bad actors are always going to do whatever it takes to get the money. There was a study that was published this year. It basically said, the number of successful attacks has actually gone down, but the amount of money netted via ransomware has gone up. You just do the math there, all that means is they're charging more.

[0:11:05] LA: Fewer [Inaudible 0:11:06] attacks.

[0:11:07] AC: Right, right. They're looking for the organizations that are likely to pay and can pay and they're trying to figure out the best way to do it. What that means for the little guys, or when I say the little guys, I'm actually talking about the ransomware attackers, there's different scales. This was something that blew my mind when I learned it is a lot of the attacks that come through email, they're going after guys like you and me. Those are just little fish. They bought something off the dark web, just black-market software, trying to get through, trying to get the one fish to bite on their crappy email chain. Hey, you click this PayPal link. It looked legit, but it wasn't. I got you. I got your information at least. Now I can get into your PayPal account. That's the small way.

The big attacks now, when they start doing the investigation, all the three-letter agencies start looking into it, a lot of them are nation state. We're starting to see these big attacks that come from countries that are looking to actually extort money out of large organizations to put into their own coffers, and they'll go through third parties, so it never goes back to the country directly. It's not like, we're seeing some military force that's been generated to attack.

When you connect all the dots, it does lead back to nation state bad actors that are trying to fill their own banks. It is a type of warfare, cyber warfare. It's just a different way to attack your enemies and get some money out of them.

[0:12:35] LA: Earlier, you said that the reason why ransomware is increasing is because a lot more things are being considered ransomware. Does that mean it's a definitional issue as opposed to an actual issue? Is there actual increase percentage-wise of attacks in ransomware compared to other types of attacks? Or is it really about the same as it always has been? Obviously, more larger attacks, but are the types of attacks that are being done still, basically, the same things that were going on 10 years ago, or 20 years ago?

[0:13:09] AC: Well, the form factor has definitely changed. I think the way they're getting in and I use the phishing email example. Most organizations, if you're in a Fortune 100, Fortune 500, even Fortune 1,000, if you're an enterprise today, chances are you and your employees are going through some kind of phishing exercise in your email once a month, if not once a quarter, right? An email will come in. It'll say, “Hey, log in to this account and make sure that you change your password to keep it up to date, because we use – this is our CRM and it's been compromised.” You click that link and then it says, “Hey, you failed the phishing test. Now go take cybersecurity training, because you didn't investigate where the link came from, the authenticity of the sender and all that.”

They've trained and they've done a good job, right? I think phishing exercises, quarterly cybersecurity training, all of those things lead up to better companies being more resilient to the attack and at least being able to cut off a lot of the damage. Those vectors don't work as well anymore, which is why we see more attacks that either come through disgruntled employee, where it is external organization knows that this person is unhappy. Maybe they've posted something on LinkedIn that's the suspect. They get reached out to. Now, they have, basically, an inside man that can help with the job. All they have to do is give away some credentials, or to give access to a single port to get into the infrastructure. That's when the compromise can start happening.

Then there's the third way. This way, I think is going to get far, far worse is social engineering. I mean, I guess the second way is a type of social engineering, but this one is we put so much of our personal information on the internet freely, willingly.

[0:14:56] LA: How do you know my mother’s maiden name?

[0:14:59] AC: Yeah, exactly. Facebook can define a person based on 365 different points of data and figure out how you're going to vote in the next election. We are very predictable creatures when you start to break down what the human behavior is. We love to share information with everyone, which this is a big thing to me is embrace privacy. Take advantage of the ways you can implement cyber security in your personal life and what you put out there. Don't tell everyone your vacation plans. Don't tell everyone the street you live on.

Enjoy the fact that some things are your secrets that you get to keep, because all of those little secrets build up you as your cyber self, but the data that comprises you on the internet, you feed that into an AI. Eventually, you're going to be able to crack passwords, security questions. You're going to be able to pose as this person as the deep fake face technology gets better. Even me being on video and podcasts, I'm putting my data out there that we're going to be able to feed to generative AIs in the future.

People can pose as me. It's going to be scary. That's the third way that I think is the scariest is there's just so much of us on the internet. All of that leads to an entry point into a company's ecosystem, or just your personal life, right? Your bank accounts, your PayPal accounts, what have you. It's scary. I think we need to be more cognizant of it.

[0:16:26] LA: A little personal story. I use ChatGPT, as I'm sure a lot of people do, and a variety of different things. Just for the fun of it, somebody suggested I try this. Tell me about blah, and the name of a person. You can tell the difference between how outgoing those people are by what type of information you get. For the most part, ChatGPT is pretty good. It says, that person's not a public figure. I can't tell you anything about it. But that's only because of rules I put into ChatGPT.

I put in my name and I get a six-paragraph list of all of my books and all the things that I've done. It's too much. This is not good. But it filters. Artificially filtering, so it doesn't give information out. Not all AIs are that way. ChatGPT maybe have safety filters, but you don't need safety filters and AIs for them to be useful, right? You get a lot of information. If you want information about private people, it'd be very easy to get, because of social media and all of that.

[0:17:27] AC: It's exactly it. OpenAI, who's founded ChatGPT, they've taken a very good stance on the importance of privacy and bringing in good data into their platform. That's just one vendor. I think the thing we've learned is any company can put together a generative AI today. It's all about, do you have enough data to feed it? It's not going to be long before the bad actors figure out, “Hey, I can just go scrape the internet. I can put together a couple scripts. Start scraping Facebook, Instagram, TikTok. Let me start putting together my own little AI. I'm going to ask it all the questions that are going to get me information out of people.” They’re probably doing it right now.

[0:18:08] LA: I'm sure that’s already going on. Yeah.

[0:18:10] AC: Exactly. Yeah. If we could think about it, they're already doing it. It's a good way to operate.

[0:18:17] LA: We're here talking about different ways they can attack. I want to get back for a second before we go too far into the type of attack, or the reason for the attack. I think, if you go back many, many, many years and figure out why people broke into other people's account, in very early days, it was because we can, and it's a fun thing to do. That's most of the reason why. I'll leave a message saying I was here and that's it. It progressed from that into data destruction. Was a big reason why people did attacks was to destroy data, to destroy companies, or individuals, or whatever.

Ransomware is a shift away from destruction and into net gain, right? The idea behind ransomware is not to try and destroy a company, but to profit from their woes. Is that a fair comparison?

[0:19:13] AC: It's the perfect. I love profit from their woes. That's poetic. Yeah, that is exactly what it is. How can I inflict pain? Because it doesn't have to be a lockout, right? Extortion is great. Just general, “I am in your infrastructure. I can see all this. Do you want me out?” They don't have to do anything. They can just say, “I am here.” Suddenly, that is a pain point, because you're not a HR employed human being. You're in their ecosystem. How do we get them out?

[0:19:42] LA: The change is from trying to destroy someone to try and profit from them. Is that changed? Is there an increase in that that's going on? Is there really a shift towards that, or are they both going on?

[0:19:54] AC: I don't have concrete data to back up the claim, but I would say, no one is doing – well, not no one. There's always someone. The grand majority of attacks today are profit oriented. They even call it ransomware as a service. You can literally contract out hits in hopes that you can get, “Hey, I'm going to attack 300 companies. I'm going to pay these three people to go do it. We're going to split the profits 60/40.” I'm not a criminal. I don't know how they split it. It's very like mafia-esque. It is totally a business. It's just a black-market business. It's one that I don't think is very well regulated by the world governments out there. It's one that the attackers are definitely, they've got the home team advantage. They're all technical. They all know how the software works. They know what they're going after. They know who the big fish are.

Businesses are constantly playing catch up, because businesses are not in the business of hiring black hat hackers. You might contract out a gray hat, or even a white hat. But the people that get employed by the big companies are not generally your hacker types. They're not doing this stuff on the weekend. They're to either engineer to orchestrate the IT, to be the security team. They only can keep up with what they know. These guys over here, they're living it, breathing it every day. They're on forums. They're in Discord servers, where all they're doing is what's the latest best way to run a tax scheme? Can I get paid to do it? What am I going to get out of this?

It is a business. It's a sickening business. I think the attacks we used to see that were more just either for the lulls of it, or the, “Hey, I want to do damage to this person's reputation, or the company,” those, we don't see so much anymore. There's always a monetary amount tied to it.

[0:21:49] LA: Monetary, but it may be monetary meaning dollars, or maybe monetary meaning some political gain in the case of state actors, for instance.

[0:21:58] AC: That’s true.

[0:22:00] LA: Yeah. It's not just money. It can be other things as well.

[0:22:03] AC: That's right. Yeah.

[0:22:04] LA: What are some of the things that we're seeing besides money that people get, or try to get, I should say?

[0:22:11] AC: Yeah. The one that always comes to mind is, and I mean, ultimately, at the end of the day, it was about money. But the Sony leaks that happened, I think it was eight years back. They were just putting out all kinds of IP and information, just blasting it out to the world, which really damaged future plans for the organization, because they had lots of things under wraps that became very public very quickly.

Their organization, I think it was the Lazarus group that was responsible for that attack. Would go on to end up attacking a Filipino bank for 8 billion dollars. This was all part of their ability to prove what they could do, the kind of attack they could get. What started off as non-monetary, or “We're going to damage this company's reputation, or we're going to leak their information to the public and hurt them financially.” It ended up leading towards, essentially, a bank heist that always wraps around.

It's like the conspiracy theorists always say, right? It's like, follow the money and then you'll get to it. But in this case, I mean, they really are – they are, everything is done for some gain at the end of the day. I can't think of any activist-driven type attacks, which I'm not saying they don't happen, it's just, it doesn't make the news. It's not common enough. I think everyone is in on this, because they're trying to squeeze out as much as they can.

[0:23:32] LA: Yeah, that makes sense. How effective is this? Obviously, we hear when the big ones happen and those are very effective. Get that. I'll also say, this is obviously a profitable business to be in, because there are large companies, large black-market companies that mafia cell companies that are able to survive and build this. Is an arbitrary attack truly profitable?

[0:24:02] AC: There was a study that I did a lot of marketing around three years ago that basically said, by 2023, an attack was going to happen every 11 seconds. It's not cool, but the thing that happened was that came true. As of this year, there is an attack – actually, under every 10 seconds. There is just some attack that occurs. Whether it's successful or not, that's a different number that I don't know. They have said that every 10 seconds, there is some attack that is lobbed at a business in this year. The success ratio, it's driven by how weak is this organization's security? How trained are they and how ready are they for the –

[0:24:45] LA: Industry-wide, I’m thinking here. How effective is this? Yeah.

[0:24:49] AC: The Biden administration put out a statement. I think it was in – it was either September, or October. Basically, it was very good. They're talking about cybersecurity and how important it is that this is actually an attack of war and businesses that are attacked need to be forthwith and forthcoming. These things need to be reported to the police and reported to national institutions. If you're a business and you're attacked, you managed to stop it. Are you going to report that is the question. This gets into the political, capitalistic gray zone that everything always leads to is we can only see what we can see with the information we know.

Many organizations do not report that they've been attacked until months after the damage has been done. We've seen that time and time and again. It's very frustrating to see your name in the list. I've seen my name in the list numerous times like, “Oh, I'm part of this. This was six months ago. My password's in there,” which is why I set different password for everything. Just pro tip in life. Different password for everything. That way, when that happens, because it will happen, at least you don't get the hit on that. We don't have a good standard.

I don't think there is a, of good will that comes from the company saying, “Hey, I've been attacked by this worm and it happened at Tuesday the 12th at 11.30. Here's everything you need to know.” Because the second you say that out loud, your customers lose faith in you. They assume, “Hey, well, maybe they don't have the security that warrants my business,” which is not true. This could be any company. This literally could be any single business on the planet. No one is immune from this. Even the top cybersecurity firms can be attacked. There is that back of the mind fear, right? Like, “Oh. Well, I do business with this company. Maybe I shouldn't anymore, because they've been hit, or they were an attempted hit of an attack.”

That leads to cloudy data. We only have the data we have. While it's disturbing, there are way more attacks than you'd ever think that would happen every 10 seconds. We don't know the success rate. We don't know how many actually get through, or get bounced back. Usually, we only hear about it when it's really, really bad, like the casino this year and the airline. When it makes the news, everyone knows.

Generally, that's because they didn't do the ransom, or they decided, “Hey, we're going to try and do a recovery, or something and it becomes public very quickly, because someone leaks it or they have to get ahead of it from a PR perspective.”

[0:27:30] LA: Now, transitioning here for a second, the recovery aspect, this is where you guys come in.

[0:27:35] AC: That's right.

[0:27:37] LA: Your focus isn't on providing barriers to prevent people from accessing your systems. Your focus is on how do you recover from attacks? Is that a fair statement?

[0:27:49] AC: That’s a very fair statement. Yeah.

[0:27:51] LA: Why don't you tell a little bit about, I guess, both about Veeam and about Object First and how all that works together?

[0:27:58] AC: Absolutely. For the unfamiliar, Veeam, actually, it is the top used data management platform on the planet today. Data management, meaning backup, replication, and recovery for business data. It was founded by Ratmir Timashev and Andrei Baronov 17 years ago, I believe. At the time, they created the company, because there was a serious need to back up VMware workloads. They saw this need, they built some super cool software. What snowballed out of that is the most successful data management company on the planet.

Skip forward to three years ago, Ratmir and Andrei, who ironically said, they would never get into the storage hardware business, start to really evaluate the ransomware situation. They identify this very key gap, which is a lot of Veeam customers did not have secure and immutable backup storage. They were using what was available. They'd buy some cheap off-the-shelf direct attached storage, or they'd go with a storage vendor, but they didn't have the expertise, or ability to secure it to the level that it needed to be.

Ratmir and Andrei, being the innovators they are, despite our previous statements about never going into hardware, we're going to create an on-prem hardware storage appliance that delivers object-based storage, so very similar to what you have in the cloud, but in a box. It's only going to work for Veeam. We're going to do something a little bit different on the security end. What they did was they put the immutability and compliance mode, which means that no admin can touch, or change the data that lands there. They also said, “We're not going to give admins any kind of credentials or privilege.”

Your top user, you buy an OOTBI, which is the product is called OOTBI. It stands for out of the box immutability. If you put that in your data center, even you, the purchaser, does not have administrative rights. No backend, no route, no operating system access, and this is by design, because we realized, compromise can come from an individual. The data that comes from Veeam to our device that lands there, it becomes essentially, unchangeable and alterable, which is what the word immutable means.

Our whole purpose of being is to take the number one data management platform on the planet and give their users a place to put data that is ransomware proof, because everything that lands on our box, the Veeam user has to say, how long is the immutability window on this data? Once it lands there, it cannot be altered, or changed for the entirety of that window. We launched back in February of this year. Our product came out. We've seen some pretty great success so far for only being under a year-old product. But I think Ratmir and Andrei's name goes a long way with a lot of the Veeam users. They recognize the legacy that comes with that.

It's been exciting for us. The thing that really makes me happy to do the job that I do is, when I joined this company, I'm learning about the customers. A lot of them just didn't have the time, energy, or even the basic level of security expertise to go off and do this kind of thing. The ones who have went and bought OOTBI and put it in their environment have been like, “This is just great.” It took me less than 10 minutes to turn it on, set it up, install it, connect it, and I haven't touched it again, but my data is always there.

We've done security trainings and recovery exercises. Pull the data back, it's perfect, but we can't get the data ourselves. I cannot break the box. I mean, unless you submerge it in a tank of water, the data is truly there.

[0:31:38] LA: That physically breaks. Yeah, yeah.

[0:31:38] AC: Exactly.

[0:31:39] LA: This is specifically, you say it's immutable and unaccessible, but it really is accessible, but only in a read-only format. You can recover your Veeam database from this immutable storage and guarantee that you're able to do that. That helps if you're attacked, so that your data is either changed, or hijacked, then you can recover that data and push that aside. What about the data compromised attacks, like you were talking about? Where someone takes your data and says, “Hey, we have your data and we're going to make it public, unless.” What about those cases? How does this help with that?

[0:32:20] AC: That's actually something we rely on Veeam on. Because we are the storage box and we're immutable, we don't do any encryption of our own. Which is, again, a choice we made to say, “Hey, we're going to trust Veeam.” Veeam has actually done a great job of building out an encryption engine. Instead, we're going to make sure we accept their 1 megabyte block size and we're going to be the perfect fit for their data throughput to be as fast and optimized as we can.

Veeam does all the encrypting of the data, which is actually how it should be. Veeam goes off, the Veeam admin searches through and it says, “Hey, we're going to back up all these ESXOs. All these VMs are going to come along with it. Here's our encrypt. We're going to checkbox the encryption. We're going to utilize that.” That admin has that ability to then send the data where it goes. They've now implemented even a four-eyes, which I think is just a hilarious name because that's what kids used to call me in school when I wore glasses. It basically just means that it's two sets of eyes that have to then validate and verify when an administrative operation occurs.

Even if you wanted to remove the encryption, or unencrypt, you could put these things behind a two-person verification wall, which follows the principles of zero trust. Again, you want to make sure that any individual can be compromised.

[0:33:35] LA: No one person can do anything I don’t know.

[0:33:37] AC: Two people is such a great assurance. It's just like multi-factor authentication. I think, you start to put these technologies together and you get in a state where even if the ransom has gotten in, it's locked down your production data. If you've gone through and done the data masking, if you've removed any PII, or data that could be compromised, or you've encrypted it on that end, you encrypt your backup data, you put it on immutable storage. Even if someone gets in and destroys the Veeam environment, the data is still on OOTBI and it's protected and preserved. All you would have to do, respin up that Veeam environment, have those credentials ready, so you could pull your data back and you could reorchestrate this whole thing.

It's not fast. That's the thing I have to tell everyone is there is no fast ransomware recovery. On average, the good recovery scenario is three weeks. Bad is three months. What we do is we ensure that recovery is a 100% possible and we give the users and afford them the ability to take advantage of features that Veeam has, like instant recovery, which can dramatically speed up being able to run virtual machines on OOTBI, simultaneously instantly in that recovery and then move them back to where they belong. Because I think what a lot of people don't realize is just being able to put the data back, that could take a matter of hours depending on how much you have.

But if all of your hardware is bricked and you have to factory refresh and flash at all, that doesn't happen so quickly. That's not something software can take care of. That's an administrative hands on the technology process. It's never fast. What we try to do is make it as painless and quick as possible on the data movement and as we can, and ensure that recoverability is always there, regardless of the situation.

[0:35:24] LA: When you talk about read attacks, that's what Veeam protects against with its encryption models. If I talk about write are real, what I think of is ransomware attacks, where they lock you down. That's where the immutable storage of OOTBI comes in that allows you to recover, changed, or modified, or locked down.

[0:35:45] AC: That's exactly it. We like to think of it as, it's almost why you don't put all your eggs in one basket. The more segmented and the more fences you put in between your software and your hardware and the various places your data lands, in the industry, we call it 321. Three copies of your data. There's going to be one in production. There's going to be two that need to go either on an OOTBI and one to the cloud. Then keep it immutable. Last one. You need one immutable copy. I say, get as many immutable copies as you can. Maybe it's 3-2 and infinite. That's the trick to recovery.

Ransomware and these bad actors are smart. They go after the backups now. That's why we see a lot of these attacks come so successfully is they'll blow away whatever the data management platform is. Suddenly the backup data is just sitting there, exposed. If you have root credentials, you can delete it, even if it's immutable. If they have root access, it doesn't matter if the underlying storage layer is immutable. I can RMRF all day, and there it goes.

It's one of those things where I think everyone needs to step up their game on the level of access and credentials they have for certain levels of storage, or pository. If you're not implementing zero access and zero trust, or your backup storage hardware, it's time to consider how you get there. One solution is not it. That's the sad thing, right? Hey, I got to tell people to go buy more stuff. Truly, it's all the parts and pieces. It's your endpoint protection, plus Veeam, plus OOTBI, plus good network segmentation. It's a lot of different parts and pieces working together in conjunction that lead to a smooth recovery process, or even preventing it.

[0:37:28] LA: Your focus is definitely on-prem applications, or data center applications. Not cloud-based use cases. First of all, is that a fair statement? Two, do you have plans, or some ideas for how to do this for cloud-based applications?

[0:37:43] AC: We are focused in the on-prem today. We're an on-prem appliance, so that automatically makes it very difficult for us to take care of the cloud guys. Now, having said that, the challenge with cloud is the same challenge with on-prem. If you have an admin account, it doesn't matter how good your IAM rules are. I mean, you got to have multiple verification, right? There has to be multiple human beings who have to verify before administrative actions occur. Getting that cloud data, you can send it back. It's not cheap. That's the problem, right? With cloud data, it's always putting things there. Oh, so nice. Pennies. Pennies on the terabyte. Bringing it back, that's where the sting comes in.

[0:38:25] LA: Dollars in the terabyte.

[0:38:26] AC: Yeah. That's why, “Oh, this hurts.” If you want to send your data back from the cloud and put it on-prem in an OOTBI, we're not going to stop you from doing that. In fact, I highly recommend it. I think the cloud situation is one that it needs a lot of vigilant, vigorous security best practices and ensuring that you verify and get as much zero trust as you can there if you don't have that on-prem footprint.

[0:38:53] LA: Now, I know some of the cloud providers have their own mechanisms. The backups that AWS creates for the system databases are immutable and have a lot of characteristics we're talking about. My mind, the problem there is it's still a single vendor. As long as Amazon doesn't get compromised, you're fine. If Amazon does get compromised, now you have everything is gone there. Do you see a solution to that in the future that involves what you're doing? What do you see is the solution to that?

[0:39:24] AC: That's a tough one. At this point, we're basically asking some of the biggest businesses on the planet to play nicely with other businesses. If a data center – we’ll just keep it generic. I won't point fingers. If an East Virginia data center goes down and you lose access to that data center, it's not a ransomware. It's just a disaster, right? I love that we've forgotten that disasters occur, even though I live in the hurricane capital of the world. I lose power once a year. You just lose access to that data center for a single day. How much does that cost your organization, especially if you're cloud native? If you're operating in there?

If you're replicating to the West Coast and you have that failover plan, or you have that ability to start running, that's great. That's more effort. That's more work and that's also more money. What are you budgeting for? What are you planning for? We don't think these guys are ever going to go down, because that's the magical myth of the cloud, right? It's perfect. It's cheap. It's fast. Why would it ever go down? They've taken all the assurances in the world when all it would take is one network switch to on and off and suddenly, everyone's lost access for 24 hours. It's not a perfect situation. I think the real answer to your question is we need to accept the fact that clouds are not perfect. They will go down and we should have secondary, or some plan for when it does.

There's not really a solution that's cheap, I should say. Cheap and or reasonable, right? Because no one wants to exfiltrate their data into another cloud vendor. That's just feeding the competition. They want to keep it in the family. We've not seen an attack that has taken out a cloud vendor. That doesn't mean it's not going to happen in our lifetime. They're the biggest targets out there with the amount of data they have, right? You want to talk about the Fort Knox to go after. They got a really big target painted on them. We'll see.

[0:41:25] LA: They sure do. Actually, that's one of the reasons I hear a lot of companies, clients that I talk to focus a lot nowadays on hybrids. The idea of, “Yeah, we prefer Amazon, but we're going to use Amazon and Google, or Google and Azure, or whatever, because two is so much better than one when it comes to many different aspects.”

I always tell them, forget about things like, cost and all of that, because that's not the reason why you do it. I used to hear people say, “It gives me better negotiation rates on my bill if I have a second vendor.” No offense. Drop that. That doesn't matter. That doesn't help at all. What it does do is it gives you better access for security and other protections. When there is a disaster that affects the data center in Northern Virginia and not the one in Southern Virginia, where the other vendor is located. It matters that you have data in both locations and you're able to recover from that. That's to me, the big value I see from hybrid cloud is adding a second vendor add security.

[0:42:30] AC: I will a 100% agree with you. Three years ago, when I was a cloud technologist and I focused exclusively on the cloud, I had this waking realization that, oh, my gosh, it can't just be cloud. That's when I started looking for a new job and landed at Object First. We're definitely in mind, or same mind on that. It's why I think the hybrid is the true, best way. You cannot bank on your building having power 24/7. You can't bank on your cloud being up 24/7.

[0:43:03] LA: Yeah, it makes a lot of sense. Thank you very much. Anthony Cusimano is the Director of Technical Marketing at Object First. He's been my guest today. Anthony, thank you so much for being with me on Software Engineering Daily.

[0:43:16] AC: Thank you, Lee. Really appreciate it.

[END]