EPISODE 1697

[INTRODUCTION]

[0:00:01] ANNOUNCER: Security Operations, or SecOps, refers to the collaboration between security and operations teams to secure an organization systems, applications, and data. Maxime Lamothe-Brassard is a Co-Founder of LimaCharlie, which is a cloud SecOps platform. He has a background in security and has previously worked at the Canadian Intelligence Service, CrowdStrike, Google, and Google X. He joins the podcast to talk about modern security operations.

Gregor Vand is a security-focused technologist and is the Founder and CTO of MailPass. Previously, Gregor was a CTO across cybersecurity, cyber insurance, and general software engineering companies. He has been based in Asia-Pacific for almost a decade and can be found via his profile at vand.hk.

[INTERVIEW]

[0:01:00] GV: Hi, Maxime. Welcome to Software Engineering Daily.

[0:01:03] MLB: Hi. Very happy to be here.

[0:01:05] GV: Maxime, great to have you here. You are one of the Co-Founders of LimaCharlie?

[0:01:10] MLB: That's right.

[0:01:11] GV: Yeah, LimaCharlie, a very interesting product, very interesting company, but I think you also have quite an interesting history, just before coming into LimaCharlie. What was your path to founding the project? I think the project was actually founded as far back as 2015.

[0:01:27] MLB: Yeah, that's right. The name dates from about 2015. Yeah, it's recapping back in the day, I'd left government for a little while back then. I started in security in government, in the intelligence sector. I did a few things of interest. One of them being early CrowdStrike. Then eventually, I made my way to Google internal security, which was a fascinating place. Conclusion was that I ended up, essentially, starting an open source EDR project. EDR is endpoint detection response. Write the agent that you deploy everywhere out there and get telemetry and take action.

The reasoning behind it was really that I've had a lot of experience building EDRs in the past. Then we were finding yourselves in an interesting position in Google, where we were looking around in the marketplace for what EDR was out there. It was quite a frustrating thing, because what we found was that most vendors were really targeting down market. Meaning, here's the widget, the software package, you install it and now you're safe, right? That's the target. That's great for SMBs, but when you're talking about a size of an organization like Google with an amazing cybersecurity team, it doesn't work so well anymore, right? Because the thing that you're looking for as that Google isn't the widget that is magically going to work in the Google environments, which are super unique.

Instead, you're looking just for the tool that will allow you to apply your knowledge and customize it and make it work for you. We looked through the market, everything was SMB kind of thing. It was my background and I just got tired of it. I was young back then, young and naive. I was like, "You know what? Screw this. I'm going to make it." I built LimaCharlie, which was an open-source CDR and I got a couple of good lessons from it. One of those lessons being that you can make something that's great and that's free in open source. It does not mean that people will use it. Open source, you have to publicize it, build the community, put so much effort behind it to make it work. Yeah, that was the first spark.

Then when I left Google in 2018, so I left Google and it meant that I was able to keep LimaCharlie, which was, yeah, so LimaCharlie was the name of the EDR. Then I was looking at what was going to be next in my career. I wanted to go and build something. That was clearly the stage. What I had this technology, and it was really one of the core pieces and we kept the name of LimaCharlie with what it is now, but now it's a much broader scope. The EDR itself was still a big component of that. It's like, I mapped things to cloud providers all the time. It's our EC2, right? EC2 is not AWS, but it's a big part of it.

[0:04:34] GV: I'm also curious, what's in the name, LimaCharlie? I mean, it's phonetic alphabet, right, but it's -

[0:04:42] MLB: Yeah. It's a combination of two main things. I would say, the official one is it stands for loud and clear. My background, as I mentioned, was from intelligence. Loud and clear, LC is NATO phonetic alphabet. It just felt like a nice name to put for something that is there, to make sure that you get visibility into everything, into your environments, kind of thing. The unofficial part is that back in the day, when I started working on some of the early libraries that would eventually become that EDR, so when I love government, it was a renewed sense of freedom. I could write code out on the outside and nobody could complain. I would write some code, some libraries on no time. I did some of those while I was working out of France for a year. I spent a year in France.

I was working out of a cafe every morning. That's at the time where people working with a laptop in a cafe was not the norm at all. It was called Cafe LC. They were super nice to me. I thought like, "Oh, it's a nice different hat."

[0:05:52] GV: That's cool. That's really nice. Let's talk a bit more about EDR. I think some of our listeners base, maybe more than 50% might not actually know what an EDR is. What is an EDR? Why do companies, I think, to use them? Why is LimaCharlie different from the incumbents? Just to put it out there, I would name the incumbents as CrowdStrike, Sentinel One. If people are for those two companies, that's roughly what we're also talking about here. EDR and how does LimaCharlie, high-level, differentiate?

[0:06:26] MLB: Absolutely. I'll start by saying that those are actually two very different questions, and you'll see exactly what I mean by that, spoiler. EDR, end-point detection response, it says a little bit about in the name, has two core characteristics. One is visibility. It's an agent that you go and you deploy on workstations and servers and laptops and anything you can imagine. You deploy it and you do that, so you get visibility at the operating system level, right? The main target is really a question of if I take it, for example, Windows, what is executing on Windows, what files are there, what processes are there, what's going on, what network connections, all that stuff? That is one of the two aspects.

The other one is the response aspect. It's the idea of "Hey, evil.exe is running on my system. I want to go and kill it," being able to take action. Over time, the definition of EDR has shifted. I think in and of itself, that's a really interesting change in the industry. Back in the early days, EDR was not an antivirus. Antivirus was the thing you deployed and it just magically stopped the bad guy. Obviously, there's a bunch of limitations that come with that, right? If you're going to make a software package that will stop, like block things on all of those computers out there automatically, you better be sure that it's really, really, really solid. No false positive. Because if you have false positives, people are going to be frustrated. That's an antivirus.

EDR was this thing that was manually operated. Purely manually operated. Well, mostly, right? The idea, it wasn't the thing that would block automatically. It allowed you to have a much more powerful agent, something that you could really get in the guts if you wanted and really see the details. Because you weren't limited by this main parameter of don't mess this up, right? It was that big a deal. Over time, there was a shift where a lot of vendors realized that especially back in the days, there weren't that many people that were able to operate an EDR. It was a complicated piece of machinery in a way.

They slowly shifted down market, meaning towards the antivirus, like the thing you deployed does it all. Over time, the definition of both started merging and then you had vendors that came out and said, "You know what? We're next gen EDR, or a next gen antivirus," and it's all the marketing BS that came along with that. Fundamentally, there is that big shift. Yeah, so that's the EDR 101, what it does.

Now, to the second part of the question and the reason I was saying those are two different things is I'll go back to my analogy of AWS. LimaCharlie now is, I'm going to describe it in broad terms, is a cloud provider for cybersecurity capabilities. If you compare that philosophically to AWS, AWS has EC2, AWS is not a virtual machine company. Those are two different things. We see EDR in the same way. EDR is our EC2. It's a big product, big capability we have, but we're fundamentally a cloud provider, so we have a ton of other capabilities.

Honestly, just the statement I just made, it's in a nutshell, the main difference between us and all these other vendors. It's not exactly in the technology itself, it's mostly in the approach. Vendors and security today are what I would - most of them, again, broad strokes, are mostly promised-based vendors. What that means is if I go and I buy CrowdStrike, what I'm buying is I'm buying the concept that CrowdStrike is selling me that they are the best at protecting everybody with the same software package everywhere at the same time. That's their promise. If I buy that, I'd buy the software. Good to go.

Now, if I'm an organization that has a cybersecurity team that is growing in maturity, so maturity is going to be a big - is really a big concept in cybersecurity and that we believe in. The idea of just saying, "You know what? I have this black box and it keeps me safe." Is it super great? What you're really looking for is a cloud provider in the same way that if it's 2005 and I work in a big company, I have to buy the box software from the vendor. That's the only way that software is going across. Now if you go, you fast forward to 2012, well, now there's this AWS thing. I don't have to buy everything from the software vendor. I have the freedom to say, "You know what? I know what a VM is. I know what a load balancer is. I know what a relational database is. I'm going to assemble the right solution for my large enterprise, or for my company, or for my services company. I have that freedom."

It's the same mentality shift that we're bringing to cybersecurity. We're saying, CrowdStrike is great. The question isn't, are they doing a good job? Some vendors are probably not great, but most vendors are actually pretty good. The question is, do I want to buy wholesale that promise, or do I want to say, "Look, I'm a big company. I have a lot of different networks and I have idea acquisitions in different companies." The idea that this single black box solution is going to solve it all isn't really realistic. Really, what I want is, give me this EDR in cloud provider terms, it's a primitive, right? Give me this capability, and I'm going to take all those capabilities, assemble them together in a way that I know I needed, so that I can actually say at the end of the day, if I get a question from my CISO, is our outsourced IT in India defended against this type of threat? So, that my answer isn't, "Well, the vendor told us that, yeah, they defend us against ransomware."

Okay. Cool. Are you, are you not? Oh, it's a difficult thing to estimate. Instead, you're able to say, "Yes, we are. Here's how we are defended against." For me, I think in terms of a scientist. For me, it's like a mathematical equation, right? I can point to you to the terms and I can tell you, this is exactly what's going to happen. I can reason about it. Maybe it's not good enough and I need to tweak it, but at least have the knowledge and the ability to go and iterate over that.

[0:13:14] GV: Yeah. I like the AWS analogy for that. I think that's super helpful, where I think a lot of people that maybe outside the security world, or even inside think, yeah, there's one product that I'm looking for. What is the product? Line up all these products next to each other and pick one. When actually, the solution for how they, as you've been talking about, protecting themselves. It's a pretty personal thing actually for each company as to what their situation is and what are the different things that they need to, in theory, piece together in an AWS way that works for their situation. Who typically uses an EDR and LimaCharlie? Are we talking internal teams, external vendors? What's the makeup of that?

[0:14:01] MLB: I think we can talk about it in a bundled way, because it tends to be the same people. For us users, again, you're going to get sick of it. I keep making AWS analogies, because look, we're not reinventing the wheel. That's a natural evolution in tech and why would we not be worthy of that in security? The analogy is the following. If I am the hairdresser across the street that needs a website, I'm not going to tell them to go on AWS, spin up a VM, install. No, no. That's just non-starter.

I'll just point them to go to Squarespace and I'll tell them, "Click, click, click. You're going to have a great website. You're going to be happy. You pay at one amount a month. Easy." Now, Squarespace, now that's much bigger infrastructure and is where is Squarespace going to be built? Well, it's going to be built on AWS. We're seeing very similar dynamic in terms of users in LimaCharlie. It means, a small company with one part-time IT person is not really a good match. Instead, what will happen is that we have all kinds of different, and I'll go back to the types of users. But one of those big types of users is services, so MSSP, so managed security service providers. They're companies that do security for a bunch of other companies.

That means, they tend to have a good security team. They need to build this at scale. They need to have economies of scale. Usually, if we get somebody like that coming to us part-time, IT person, we'll just point them like, look, there's a bunch of MSSPs that do it, eventually use us. That's one of the big ways of thinking about it. But the three categories of users are the services. MDR, sometimes you'll hear a managed detection response, incident response. A lot of incident response really love it, because they're going from a world where they had to negotiate these capacity deals with vendors that the vendors themselves had incident response businesses.

They were competing with their own vendor. It's just a very messy, no good way of doing business. Now, from their perspective, it's like, credit card. Then I need 10,000 EDR if that's the thing in five minutes. Yeah, no problem. Right. That's big change for them. That's services. We have enterprise. That tends to be more enterprise that are tech forward. Companies that understand the cloud, they're comfortable with the cloud, right? That's an easy thing. They have a security team.

Usually for them, it's a big part of the tradeoff is they're looking for a capability and not a product. There's a subtle difference there, right? The product is the thing that tells you how it can be used. The capability is just the thing you use, how you want to use it, so they don't want somebody in the middle. That would be like, Snapchat is a customer of ours.

Then the last one, which I think is super cool is the builders. If you think of AWS, one of the huge use cases of AWS is for startups themselves. If I go and I start - I do a startup and let's pick something random AI today. Nobody's doing AI. Let's say, I was. I'm not going to go and rack and stack my hardware in an office somewhere. Because that's not where I get value for my work. I go to AWS. Nobody cares that I use an EC2 instance to do my thing. It's accepted.

With us, there's the exact same kind of change. We can accelerate the go-to market for a bunch of people that have really cool ideas in cybersecurity. The big canonical example for us is Blumira. They are a company that I apologize, I'm sure I'm not doing it quite justice, but they were essentially in the cloud sim space before that's a - event management to cross a bunch of different platform, simplifying things. They wanted to become an XDR. They wanted to release an XDR, which XDR again, think EDR, but across all kinds of different platforms. Not just end point, but also SaaS and all that stuff.

They came to us, they started using us. They went zero to GA, I think in about four months on their product. If you think historically, what would it have taken, a company that has this idea that wants to do that, okay day one, start writing a pitch deck, because you're going to need to go and raise some money to either build the thing, or you're going to go and use open source, and that's going to work great for the first couple of months, but then you're going to hit some scale problems, or you go to the big vendors and you sell your soul to their marketplace, and you're only going to work with that vendor. It's just not great. Where with LC, we get all kinds of people that just come to us. They're security professionals with really interesting ideas. They just want to go and test those ideas and build products. They can just do that, pay as you go that, that whole kind of - the cloud way.

[0:19:18] GV: LimaCharlie describes itself as a SecOps cloud platform, is that correct?

[0:19:22] MLB: That is correct. What's the famous quote? There's two hard problems in computer science, cash and validation, and naming things. That's the naming thing side of the equation.

[0:19:35] GV: Everything you've just described, I guess, rolls into that. I mean, at the same time, LimaCharlie does bridge the gap between on-prem and cloud. I guess, traditionally, security deployments tended to be on-prem, because that was the idea was that, yeah you're looking for something that works in your environment, very secure, doesn't talk to the outside world, but understands if something is coming inbound from the outside world. We want to be able to look at that, etc. But LimaCharlie very much, if anyone heads to the LimaCharlie website after this, there's a nice diagram where you're going to see how things can get set up. Can you just talk briefly, how does that work? Is that a big step forward in terms of in this space being able to bridge that gap, or, yeah, how does that look?

[0:20:25] MLB: It's been a big evolution, I think, in cybersecurity over the years. There was a time where it was purely on-prem right back in the day. Then, I would say, phase two was everything moved to virtual machines. It's in the cloud, but it's still contact a vendor, so they spin you a virtual machine for your server. Then there's a generation which, I think, CrowdStrike was the very early innovators there, where it was more the SaaS level. That has been the big evolution.

I think what we're doing that's really good is that we're not limiting ourselves to only the latest, greatest and online side of things. Like the SecOps cloud platform, we are a cloud provider, but it doesn't mean that we only work with the cloud. It translates in a bunch of more subtle ways, right? I'll give you one big example for me that stick in my mind. One is around compatibility. There's a lot of people that run point-of-sale software somewhere in some obscure network and it's running Windows XP. The reality is they need some security that goes with it. For us, it was a point of pride from the very beginning to extend our compatibility as much as possible and take a degraded approach. It's not binary, like support, don't support. It's like, here's the things that we may not be able to do on XP, but you'll get support through and through.

The other part is there's a lot of things that can be done better in the cloud. We wanted to go and put an emphasis on that, while still working with the on-prem world. We have a thing that is called the universal sensor protocol. We put out an adapter. Think like a Splunk forward, or kind of thing, although I'm not super familiar with the Splunk forward. So, it may not be a one-to-one match. It's this binary that you can go in, you can deploy anywhere that you'd like. that can bring telemetry into LimaCharlie. That telemetry is never gated by what LimaCharlie understands. You can specify your own parsing and that is done in the cloud.

We wanted to be able to go all the way down to the on-prem, this Linux proxy server that has this log that doesn't exist anywhere else on Earth, but is really important for you and your security posture. We wanted to be able to bridge that, bring it in and do the parsing in the cloud sounds like a small thing is actually pretty important, because there, we're able to abstract away a lot of the historical difficulties you might have had with trying to parse those logs on-prem, right? Like, I'm going to take them and I need a four-core CPU to run, just to parse the logs from this thing. Now, we can just strip all of those layers away and make it super, super thin, scalpel of this is what I need, bring it into cloud and then you blow it up to all the capabilities that you can only get by running in a cloud.

[0:23:33] GV: Yeah, I think that's a really good thing to call out. Again, this might not be clear obvious to those that don't work in security every day, sort of the two sides of the equation really with security is A, do you have something monitoring at all? Then B, what has it monitored and then what can we do with that information afterwards and what can we figure out, whether that's - it could just be that when we say monitoring, we're just talking about literally just a disk drive. Well, disk drive monitors files. When something bad happens, you go and grab the disk drive and you run a bunch of analysis on it.

In this case, this is an active way of looking at on any device where LimaCharlie is deployed, what is happening. Then as you say, I like how you phrase that, but there isn't a defined way that LimaCharlie is trying to understand things. It's actually then providing that capability to the user to then go and build up the understanding that they would like to have. I will get into that in just a second. I noticed there is a thing called sleeper mode and I'd love to just learn a little bit more about sleeper mode.

[0:24:36] MLB: Yeah. Yeah, for sure. The sleeper agent sounds ominous. Sleeper mode is actually pretty simple. It's funny that it hasn't been more pervasive, but it's for us, it's an emerging behavior from our approach. Very simply, you deployed those agents, right? Maybe you deploy 10,000 agents. Traditionally in the old world, you would have had to go and negotiate with the vendor for 10,000 licenses and God forbid, you have to go up 5,000, you go to go and renegotiate. In LimaCharlie, now at least it's, okay, on demand, you say, I want 10,000. I deploy it 10,000. I can go up and down how I want.

However, that is LimaCharlie, in full power mode. Meaning, I get all of the telemetry. You can take action. What we realized is that as part of our infrastructure was very easy for us to introduce a behavior where we could say, "Hey, this agent, all the way down to the single agent, do not load the security capabilities that you have." I still want you to be there. You're still going to be connected real time, bi-directional, real-time to the cloud. I'm going to be aware of you, but don't do anything. Don't bring back any telemetry. Don't do anything like that.

Because our approach is very infrastructure, I think another vendor would have seen this as almost as a risk, in a sense that, well, oh, if that happens I'm not selling my big license. We saw it as a feature where we said, okay, well, now we can introduce that mode and it drastically reduces the cost for us. Let's do that. Let's introduce a sleeper mode that when you turn that sensor into that mode, it doesn't do anything on a computer, doesn't bring back anything. Still there, and then it costs you pennies instead of the usual billing. That opens up a ton of doors. Again, this is one of the things that I love the most about LimaCharlie is that emergent behavior, right?

Things that we built in as a mechanic, and then people use those mechanics in ways we never thought about. This opens up scenarios like, incident response firms. Do really like that, because it allows them to go to people that they've worked with in the past, or they think they might work in the future and say, "Look, we're going to pre-deploy this agent." Right now, there's no fire. You're good. Everything's fine. Now is the time to go and deploy those agents under your own terms at your own speed." They're going to be pre deployed. They're not going to take resources. Everything's going to be fine. The day that something happens, instead of getting in contact with us and then we spend three days in a mad rush with your assessments, trying to get the things deployed, while trying to find a bad guy at the same time. Instead, we're going to be able to just flip a switch, light up the whole network, do the remediation, do the hunting, figure out what's going on. When we're done, flip the switch back off and we're good. It changes the whole life cycle of how they look at things. Because it's cheaper, obviously, it means that now it opens up different ways of structuring the revenues, which is super cool.

[0:27:56] GV: Yeah, it's very cool. Again, maybe not obvious to anyone who either doesn't work in this field, or is fortunate enough to have not suffered an attack and had to deal with this from being a customer side effectively. It does cost a lot of money. Often, that's to bring in a firm often to come and do this and have them install the respective EDR, etc. Often, companies then just refuse and they say, "No, this is too expensive." We're heading on. That's a real shame, quite frankly. I think there has to be better solutions out there that look at the cost, cost to the customer at this point. Clearly, LimaCharlie does that. Clearly, I mean, keeping costs low for the end user seems to have been quite a big part of LimaCharlie's success and growth. How else would you say that's been achieved and is that something that is baked into the mission of LimaCharlie? Or is that more, yes, it's low cost now, but in the future, we might have to look at different model? I don't know, just curious about that.

[0:28:58] MLB: Absolutely. That's a good question. I think it should be one of the first questions when you're thinking of a vendor, right? For those maybe not in the security industry, there's a really nasty behavior that exists in the industry, particularly in startups that sometimes will - like early on in the startup's life rush to sell various contracts, longer term contracts, where they're actually losing money in every contract. They're making the bet that we're going to get a bunch of contracts in, even though we don't make money, we can still refer to the full price. There's some weird accounting and then we're going to be able to raise and grow this way. I think it's not healthy, in my opinion.

We are absolutely not doing that. That was the very - the floor here. That being said, here's why we're doing it, and that will explain why that's not going to change. Because we're approaching what we do like a cloud provider, it means that we make very different pricing decisions on how we price what we build. What I mean is let's pick one example. It's just a simple example. Don't want to pick on them, but like a data lake, right? If I'm a Splunk, I sell data lakes. What am I going to be billing on? Volume of data coming in. That's pretty much the main knob. Now, you might disguise in a couple of different ways, but that's really the main knob.

If you do that, you will have an incentive to optimize as much of all of your billing on, how do I maximize the billing based on the data coming in? That's the very naïve, basic incentive there. If you're building a cloud provider, you can't do the same thing. Because imagine if AWS had come out with S3 and I don't know exactly the pricing these days, it's probably maybe around 2-cent per gigabyte-ish, right? Let's say, they'd come out and they said, "You know what? We'll get 10% of the enterprises stick with S3, but we're going to raise it to 15-cent per gigabyte, because that's the very stretch limit of what people will pay for storage." Then absolute number will make more money.

That works fine for, if you're very myopic and you look at S3. But now I'm introducing this cloud where I say, "Okay, but I have virtual machines and a lot of workloads requires, drives usage of S3. I have these databases that also drive usage into S3. I have all these other APIs and features." Then that approach is inviable, because the day that I want to introduce, I don't know, service X from AWS, the base price for me to use it is whatever S3 costs. That is at the roof, which means the economics just don't work anymore for me to build those - this building block to build a true cloud provider.

Going back to LimaCharlie, we made that realization, and so we're building the capabilities that we have with this in mind, right? Our data retention, I'll pick it as an example, because again, it's very straightforward. Very early on when we build that data retention, we realized, there's a bunch of features. There's a bunch of capabilities we want to release in LimaCharlie. There's a bunch of other capabilities that will rely on those capabilities you haven't even thought about. It means that we have an incentive to, yes, not go. We don't want to go into red, but also, to keep a very strict lid on the margins that we make on things, because we need to make sure that it keeps being competitive.

I think it's great, because it means that if you're a user that is looking for the environment of a type of LimaCharlie, cloud environment type, then the economics are really great for you, right? That's why the vast majority of our users, their builds go way down when they start using us. That's really, that's working as intended. That's part of doing it.

[0:33:11] GV: Yeah, that's a good way to look at it, because nobody says AWS is a fantastic value for money product on that basis. They have the whole suite of things and the more they use one thing, the more another thing gets used. I think that's a really good way to look at it. Of course, then you have, as a result, people know probably from, listen to some other episodes on, I'm a bit of a cloud run, Google cloud run fanboy, and yourself as well, great.

[0:33:33] MLB: Oh, yeah. Big time.

[0:33:34] GV: Yeah. Right. What is that? Well, that's Google is saying, "Hey, we know that if actually, we can sell a product that is a bundling of other things, that is exactly what you need, which is just to deploy a containerized app and scale it." You don't have all these dependency things, APIs that cost more here and there and everywhere. It works and people use it and love it. It's really interesting now to hear that you also like it, because whenever I meet developers that I like and I get on with and we all talk about cloud run. Yeah, it just shows you can build great products.

[0:34:07] MLB: Yeah, absolutely. A little bit behind the curtain, LimaCharlie, we have built an extension framework, which is open source, that allows people to very low-coupling, high-cohesion way of building capabilities outside of LimaCharlie and connecting in. We use it ourselves. As we add capabilities, our focus, our first thing you look for is, can we build it as an extension? It's webhook-based, which means it's driving thing us to use a ton of cloud run, because yes, at really large scales, there's some things you probably don't want to do in cloud run. You can do cheaper in different ways. For most things, getting this granularity of just here's a container that does one thing and I want to deploy it and I don't want to have to think about it and it's isolated. So, doing its own thing, it's scaling automatically is beautiful.

[0:35:01] GV: Yeah. Yeah. I find it really interesting. Yeah. I say, so many developers, the DX, the developer experience, that seems to be what drives it that this is a product designed for developers and developers just love it. Yeah. I've never heard quite so many developers talk effusively about a product that is that far into the stack, basically. I think it's really interesting.

Let's go back a little bit, actually just going back into LimaCharlie, the product itself. I was just mentioning how one side of the equation is just bringing in the data and we've covered quite a lot of that with things like the universal sensor protocol. The other side of the equation is the actual analysis of that information, because that's the whole point. If you can analyze it and then figure out what has happened, or what you're trying to just understand, then, yeah, not quite as useful a product.

LimaCharlie actually launched its own querying language back in 2021. Yeah, a querying language, i.e. to look through the data, you write in this language and bring things out. That's a very hugely, basic way of putting it. Why did you actually have to go that route creating a querying language and what has that actually unlocked for users?

[0:36:12] MLB: Yeah. The goal was never for us to create that language. Sometimes, I think, some people start from the perspective of, "Oh, I know what I can do better and that language and backtrack from there." For us, it was a different thought process. In LimaCharlie, we have an automation engine, which a role engine that will take an event, run the rules over it, say, yes or no and then take action, but very simply.

The format for it, it's a data structure format. We took an approach where parsing grammars is really, really, really hard. It's a pain. I hate grammars. When we built that automation engine, we went from a data structure perspective, because the advantage of doing that is that it's fairly easy to generate those data structures programmatically. It's easy to transcode between different things. You're not parsing like, is there a space between that parenthesis and none of that stuff. It's just data structures. That was the automation engine. When we expanded into the querying side of things, we found two things. One was at a technical level, the easiest path for us was to use that automation engine, but we're in a cloud running at scale.

Actually, it wasn't still is mostly on cloud run. We were able to essentially say, I'm going to take these events and I'm going to take that cloud run container and blow it up to a thousand instances, crunch through the data when they're gone, just go back down to zero. We wanted to reuse that engine that would allow us to do two things. One is it's a piece of infrastructure we had that was well tested. People understood it. Users in LimaCharlie understood it.

The other aspect is that if we built a query language that was based on it, it allowed for an easier transition back and forth for users between the UX of, I want to query historical data for this, this and that. Then usually, the next decision is, okay, I found what I want, I want to look for that automatically at scale across everything in my enterprise. We wanted to make those two experiences really, really similar. The approach that we took was to design a query language, very inspired, I would say, by the Splunk. We did not want to reinvent the wheel. We wanted to stick as much as we could. The way that we built that then, parts of the query were reusing operators that we have in our data structure. It works by, it's like a tree of operators, right?

If field A contains value B, then do something. We were able to really create a close mapping between them. It was a little bit easier for users to think, if you had written rules, you could write a query. It also meant that when you wrote your query, we could automatically spit out the rule that resulted in that, because under the hood, it's we just generated that rule anyway. Now as a user, I could just copy paste it and go. That's how we got to there. Query languages are, yeah, grammars. I'll leave it at that.

[0:39:51] GV: Yeah. Is that a part of the product that has to be quite actively worked on? Or is it, having never built a language before? I'm not really sure what. Do you build the language? Then, I mean, okay, if you're building, if you're part of the, I don't know, the Ruby community, of course, Ruby over time evolves Ruby version one, two, three, four, five. Or, you quite then content that you've created something that can just evolve with the rest of the product and not need too much attention?

[0:40:19] MLB: It requires effort to keep expanding it. I think it's in the more complex features side of things. What I mean is fundamentally, what you don't want to do is write your own literal code that parses the grammar. Don't do it. It's like writing your own crypto. Don't do it. Instead, you use a grammar parser. My God, I forget the names, but PEGs, there's a word that's an acronym for something grammar. We devised that and that maps into the various operators. That maps pretty well. The parts that are trickier are when you're talking about a query language, usually you're not just thinking about, find me all the events that match X.

Very often, you'll be thinking of, give me the top 10 events that Max X grouped by this ordered by that. When you start to enter that, the challenge, it's funny, isn't so much with the grammar itself, but it's with the back-end infrastructure that needs to do that. Doing that at scale in specific memory envelopes, and that introduces a ton of complexity. It's one of those, the MVP was fairly quick, keeping expanding on the operators, like that's fairly easy. It's really those advanced features. Those will definitely require ongoing tweaking in the future.

[0:41:54] GV: Yeah, very interesting. Not an area I am familiar with at all, but I feel I've just learned something right there.

[0:42:00] MLB: Yeah. We all get to start there. If you asked me a couple of years ago, I knew nothing about it either before.

[0:42:07] GV: We're coming to the end of the episode, but I'd love to just hear about something completely different, which is that LimaCharlie launched a conference last year. That's not a minor undertaking. What was the motivation behind that? Yeah, I think, so you had your first one last year, I believe, and what were some of the highlights of that?

[0:42:26] MLB: Yeah, absolutely. The motivation was that we go to a lot of conferences. We have to go to the conferences. There's a ton of security conferences. The thing that is generally not as well served are the builder side of it. If you are in cybersecurity and you're thinking in terms of building pipelines, doing systems, doing a little bit more heavy lifting, right? You're still in security. You're not looking at how does this exploit works, but it's still very security-centric. There wasn't anything addressing that, or around the innovation side of things.

Obviously, yes, all security conferences have innovation, but we wanted to have a place where we could get people that are building new, interesting things that you may not have seen anywhere else and they're just going for it. They're building that cool stuff. We had some really great talks last year, specifically around those kinds of areas. The two ones that really stick to my mind are Lennart Koopmann, who used to be at Graylog and a co-founder and is now doing Enzyme, which is all about Wi-Fi security, which is very underserved in the security space. It's totally different than anything you've ever heard. He's building technical solutions for that. Fascinating stuff.

We also had Anrew Morris from GreyNoise do a presentation and he was talking about a bunch of their early use of machine learning using AI. I think they're using OpenAI. It was really interesting to see a very ground level approach to how did they get value, and not only as this is the cool thing we can do, but actually going and all the way to delivering a product that really adds value. A real product. I think that's sometimes, it's underestimated in security, the delta between a proof of concept and a product. It was really, really interesting to see some real AI at the ground level of how they're doing it.

Yeah, lots of cool stuff. It was a great conference. We're doing it again this year in the same place at one of our investors in Science Capital in DC. Amazing offices over DC. It's going to be great again.

[0:44:49] GV: Very fun. I mean, it's not a minor undertaking to throw a conference. It's great that you're putting something out there that people can come to and just meet people. I think that's always the best part of conferences is just being able to meet super interesting people. That's clearly, there must be a pretty interesting set of people around the LimaCharlie community. Yeah, for anyone listening, maybe think about that one.

Maxime, it's been great to have you on today. I've learned a lot. Even though I was familiar with the product a little bit, but still learned a ton more today. Yeah, really appreciate you coming on. I know it's the end of a long work day for you, and you still come turn up on Software Engineering Daily, which we really appreciate. If anyone's looking to just get started with LimaCharlie, explore a bit more, where's the best place for them to go?

[0:45:36] MLB: Limacharlie.io. And we have a big YouTube channel as well, where we have a ton of content around it. There's lots of content there.

[0:45:44] GV: Awesome. Thanks so much and hope we get a chance to catch up again in the future.

[0:45:49] MLB: My pleasure. Thanks for having me.

[END]
SED 1697		Transcript

	(c) 2024 Software Engineering Daily	1