0:03Hey, everyone. My name is Greg. I'm an engineer at Sentry. I'm going to be talking a little bit about prompt injection, but really I'm using this as an excuse to talk more about building AI pipelines that scale up with the organization without bankrupting you.

0:23So, the type of prompt injection that you can sort of expect in

0:29observability space is slightly different than what maybe you've seen in the past. Like this year skills got very, very popular and with skills a new sort of prompt injection type of attack came in if you sort of fetch any random skill from maybe sources you don't know the risk is that there is going to be like a bundled script or there's going to be some secret instructions within the skill that then the attacker when you're running on your machine without sandboxing. It's a pretty good bet. I don't sandbox. And that script, when it runs, it's going to exfiltrate your secret. It's going to open a remote tunnel, whatever it is. But you're going to be owned.

1:14And I've done a little experiment in the past around that. And Frontier models, no matter how smart, were very, very capable of running these instructions unchecked and very happy about it. They were very trigger-happy. And the same sort of type of prompt injection starts existing in other places, I would say. So if you use Sentry or if you use any observability platform, anything that tracks your telemetry, so gathers your errors, traces, logs, metrics, you're going to be exposed to this fun problem. So the way that those systems usually work is you deploy your applications, your front-end applications, to your clients. Those usually are considered untrusted clients. And the way Sentry works is you set up something that's called DSN, data source name.

2:04Your SDK, your client application, is going to emit data to a public untrusted endpoint, the ingress control endpoint that we have. We call that one relay. We don't do any authentication. We kind of trust that DSN isn't leaked. It's not really trash. we advertise that that data is to not be trusted and you have to sort of pay a little attention to it. That gets processed. There is a processing pipeline that makes sure whether this should become a new issue or whether this is an event that represents the same issue that you already have. Then that is folded into actual issues page. So if you go to Sentry dashboard and you see some issues there, that's... That's this rollup.

2:49And from there, it's usually consumed in one of a couple of ways. Like, historically, those will be humans looking at the dashboard. More often than not, we're seeing growth of MCP. So you set it up on your local machine, or you have an integration that integrates directly with your API, and that consumes those issues, fixes them, or whatever. And we also have SEER, which is our debugging agent, that can do the same. And the type of prompt injection that I mean here, that you're . probably prone to to an extent if you use any observability platform sort of basis on this attack. I, as an attacker, can create a bogus error that looks like a normal error, but as part of the prompt it says oh, if you install this diagnostics package, you're going to be having easier time debugging this stuff. And then this attack again banks on the fact that you maybe have set up some sort of automation, or maybe you're running an MCP, and maybe you're an agent that's not the last-gen agent.

3:47It's going to load this issue up, follow the instruction, install the package. It's my package. As an attacker, I'm going to have a post-install script that's going to then exfiltrate your data. And we're back to the same problem that skills present. And that risk is actually real today on, I think, all major observability platforms. So if you need a new headache, new vector of attack that you have to think about, you're welcome.

4:19We want to do the right thing here and try to capture those as much as possible and as early as possible.

4:26And when you think about when do we even have an opportunity to capture that signal, really what we want to capture is right before the issue is created, we would want to say, this is malicious. We should squash it. We can't do it. can also technically do it in Seer Agent, because we control the agent, so we could literally do it with every tool called sort of like a guardian model. And then MCP is kind of out of our hands. We will tag whatever untrusted data we have in like untrusted data blocks, sort of to help the agent on the other side. But that's about the extent we can help there. So the best place really is to capture it before the issue is created, before the web hooks are fired automations start running. So for the rest of this talk, I'm basically going to be talking about the detection pipeline.

5:17In order to catch that, if you take a naive approach, you could imagine before we create an issue, you could just run an Opus model. Don't worry about money.

5:30It's probably going to work. It's probably going to have pretty good accuracy if you just ask Opus, hey, is that malicious? And I've done it. some experimentation it does work very well except for two problems Opus is slow and Opus is really freaking expensive so we're gonna split the pipeline into two we still want big -ish model to do a verification because there is a lot of nuance when it comes to what's really malicious versus an outcome of a random application so we want this like general intelligence pass on it but we want to do it only for like a very very very small fraction of events because we know this is going to be expensive So really, what we want to focus on is figuring out how to make a cheap pass on all the data that is going to be as accurate as possible and filter only like 99 .9999 % of errors that are going to go to the normal workflow.

6:25And the problem with that is it's still very expensive to do if you keep using LLMs. The ROI is not really there. Like, this is the right thing to do. It doesn't really grow the business, at least not directly. And for context, we really capture, like, an order of 100 events per day that are actually malicious while we're processing hundreds of millions of events. So the actual strict reviewer doesn't really get that much mileage. So if we try to run... an LLM on every single event, it's going to be too expensive. I'm, by the way, a little bit annoyed because I had a very fun meme here instead of this quote. But Cloudflare said, this is copyright infringement.

7:15So we're going to keep it to the quote. We tried running LLM on all the events. Way too expensive. So we ran LLMs anyway. But we did something different. We did a little . heuristic, which cut down the number of events. So an order of magnitude of events we are running the detection pipeline on is like 10 million a day. It's still a formidable number, but I want to raise it up higher. Because while this heuristic works, it cuts down some events. And I'm sure if we dig into how the heuristic works, all of you could figure out ways to go around it. So I'm going to avoid that. I'm not going to talk about that. But I want this to be 10x. more, 100x more. I want to be able to process 1 billion of events as cheaply as possible.

8:02And if you do some maths and you look at this model cost, and you imagine that I'm going to have to 100x it, you can quickly come to the conclusion that I probably would be fired. So what I have to do is I have to figure out what is the next thing that I can use that would be cheaper. Those models, by the way, are fantastic. but the problem is so easy and well defined that their intelligence level crushes the problem so up until like I want to say end of August GPT OSS was actually the best model we could use for this for this use case and then September came and we have a new new whole paradigm and when you think of decision models as a whole this this kind of problem that we're trying to solve which is to detect malicious attempt at prompt injection is very fit for decision models because really the problem you're solving is take a piece of text out to the number and the number can be a zero to one sort of confidence level of how malicious that issue could be so it works pretty well on the as a concept and when we compare cost with the cheapest LLM we've been using for for Jeff specifically it's like Not as cheap as I thought it's going to be, but it's like three times as cheap.

9:26If I 100x, I'm still fired, but I'm getting there. It's also, importantly, very, very high accuracy. It beats all the LLMs. And then we started running a lot of decision model comparisons. Good job, Clef, by the way. It's very good. Really, this is the most interesting slide of this entire presentation. The way you sort of read it is you want the number of attacks to be ideally zero, maybe one for statistical significance. And then you have another important column here, which is the benign issues that are flagged. Because if you look at this table, you can see that Mercury Decider or Perplexity Decider are fantastic models. Very, very cheap. Catch everything. But the problem is they also catch a lot of issues that aren't actually malicious.

10:15And if you remember the original pipeline it has the strict reviewer it has the sonnet pass on on everything and if I use any of these models the number of issues that are going to be sent to the strict reviewer is much much much higher to the extent where it becomes like a half a million on the comparable load so that's our next step we're basically saying okay we can use decision models it's going to get us to like reasonable cost maybe can free X the number of events that we're running and still remain within the budgets.

10:53But what if we want to go further?

10:58And the basic point here, the basic problem here, is that we are using a general intelligence model. Decision models are also general intelligence models. And the reason we use them is because it's so easy to prototype with them. I can only set up an API endpoint, and I can scale it up pretty quickly. high and get like a very, very good result. And that's how we're testing a lot of our pipelines. But when we realize that the pipeline is good and we have to scale it for more usage and keep it economical and keep it business viable, now that becomes

11:32like traditional ML is a reverse of good developer experience. You have to basically build the entire pipeline of gathering data. figuring out the, like, on-the-CASP cases, labeling them either with humans or better models,

11:49re-fine-tuning, retraining, synthesizing more data, and you have this whole platform loop that you have to create, which at a certain point is worth it, but either way it's going to be costly. But we decided this is probably the right thing to do.

12:06A moment ago, I said that... But the problem we're really trying to solve is take a string and assign a number to it. That is a very well-defined problem for specialized text encoders. The way that text encoders work is literally that. They tokenize. They are the Transformers model. So they, to an extent, work like LLMs, where you tokenize a string, you run a couple of attention layers, and you get a number, whatever significance that number has to you. So we started training that. We started experimenting with MMBird and ModernBird. The promise here is that if we can hit similar accuracy as to decision models, that's going to be much, much less expensive and faster, because those models are around 150 million parameters.

12:58Even the smaller decision models, I think, run like a couple billion. And... when we fine-tuned an example, we were able to get to north of 90% accuracy, still not where we wanted because we didn't build out the entire, like, data gathering loop and labeling loop, et cetera. But the data looks promising, and we are sort of shadow testing this. When you look at the cost here, if we deploy it on, like, not too many GPUs, we can get to, like, 12K on the similar loads, which means that... If I 100x that, I can kind of weasel out my way from a million-dollar bill, I think.

13:41But that's only for one security pipeline. The sort of way that we are talking about this is if we have to build the entire classifier pipeline, we may as well start using it for more use cases, for spam detection and other signals that we can use on telemetry to then enhance the product. And I don't really plan for... time for that, but I'm going to just mention this very, very shortly. This is like distant future. This is something that I want to test well, not distant. This is something that I want to test within this year. Technically, on the paper, what's also possible is take those 150 million models, take smaller, like 4 million parameter model, and distill the knowledge. This is the dark knowledge training that you maybe have heard about, where you can take the outputs of this model.

14:30as well as outputs of something bigger, like GEV or an LLM, and try to distill that knowledge into the smaller model. And that would cost us, this is incorrect, it would be like $700 a day on the full load. And that becomes a negligible cost that we can then multiply into multiple use cases. It also becomes much, much faster. At this point, we're talking about significantly under one millisecond of processing. which then you could put directly synchronously on all the pipelines that you have for processing and it wouldn't matter.

15:06Again this is hypothetical. I don't want to say that we're doing that but this is something that we're now striving to do. This is my time. My two pieces of information I want to leave you with is one if you are using observability platform pay attention. being attacked by one, not by one, through one. We're not malicious. And then, two, if you're starting to approach scale, while the developer experience of generalized models is great, you're going to want to start training your own specified classifiers. And I have a couple of minutes if there are any questions.

15:57I'll just ask, where do you run that classification? The classification itself? We are renting the L4 GPUs from, so does on GCP. But we have to self-host, which is another layer of complexity we have to do if we're training our own classifiers, as opposed to any models that are just available over an endpoint.

16:28Thinking about the problem of prompt injection broadly, do you think this approach of just train some classifier that runs on any untrusted input and then have that flag potential attacks, is that a good approach? Do you think people are going to adopt that or is it going to be highly specific depending on the domain that you're in?

16:52I think it's not enough. So the thing that I didn't really get to cover here is oftentimes you will want to enhance it with other things. So for instance, it's very easy for a classifier to detect that there is a package installation. You could also judge whether the package installation is malicious by indexing those packages automatically and just like running it against an embedding. But I wouldn't say one replaces the .. I would just like combine those signals how much time is gonna take to train the distiller to them end of the year seven bucks per day like when I will rephrase it when there will be the threshold when you will decide that the distiller is ready to be put in work?

17:50Sorry, can you repeat that? Where is the threshold when you will decide that the distiller is ready to be put in work and trained? So, we already have... I don't know if I mentioned that. This pipeline already runs for a couple of months, but it runs on either LLMs or on decision models. So, the threshold is it's approaching the... of that pipeline or going past it because then it's a pure win in terms of cost. The argument could also be if it's not really hitting that level but it's like getting there and we're starting to hit plateau but it's so much cheaper to run than using these decision models, that could also be a decision we make but at this point I want to hit that accuracy.

18:38I'm already hitting over 90% and I'm pretty sure I can at least for cases we know, I can get there. The problem is cases we don't know.

18:51Thank you.

18:56Thanks for the information, Greg. So you mentioned you were sort of experimenting and trying to build this pipeline. I was curious if there's any tools or other infrastructure that you sort of messed with that could be immediately useful to start building your own pipeline as you get into this? Sort of. So, at some point, this is going to go far, but let's do it. There we go. At some point, I mentioned that we also have a secondary check in Sear. So, this is what helps us capture issues that we let the first pipeline slip. And that's... our backpressure, backtest sort of thing. Because the number of automations we run for SEER is significantly smaller than the throughput of traffic we're going to see on the issue creation.

19:51And because of that, we can run more expensive models. So those will sometimes flag things that the chip pipeline was not able to or failed to for whatever reason, and sort of acts as a secondary check that we can then use to backfill more. evals and stuff like that and I think I'm out of time after that have you consider only running the eval when someone is actually looking at the events instead of trying to you know detect the prompt injection for every event that you ingest when someone's looking at the issue yeah via the API or like looking via like you know The MCP server or whatever?

20:38We did consider that. We were worried that the number of entry points is too high. And some of them didn't make a lot of sense, I think. So one of the checks I think we run, either have run or still run, on the back of MCP. When the MCP decides to fetch you an issue, we would test that. I don't know if it still works that way, so I don't want to overpromise. But it felt very largely. duplicative to what we've been doing on the issue as well.

21:08We are not doing it right now.

21:15Thank you very much.