0:03Right, folks, I think we are good to start. I have been scheduled over lunch, so there may be some people walking in late who are a little bit fuller and have had some lunch. I won't pretend I'm envious of them. But hello, everyone. My name is Ryan Skidmore. I work here at Cloudflare, and I work on developer productivity and AI, which is a very broad remit, but it basically means... I work on all of the stuff internally to help make our engineers as productive as possible and also safe as possible when using things like AI. And, yeah, that's pretty much all I do every day is work with our engineers and, you know, help them be as good as possible at their jobs. And this talk is about building software instead of buying it.

0:51And that has gotten a lot realistic over the last few years. We have some audience interaction now, which is always fun. So can I get a quick show of hands? Who here, or your company, your employer, who pays for software that your team actively works around?

1:12It's quite a lot of people. Okay, so now put your hand up if you've ever thought that you could build a better version of that software over a weekend.

1:23Yes. a lot of people. And that used to be something that was a little bit more unrealistic, maybe a pipe dream, some would say. But actually, now the idea of sort of rebuilding a piece of software in a weekend is actually pretty feasible, and you can do it most of the time. So we're going to talk about what happens when you have that capability and what that means for the sort of the workplace and the enterprise and also. personal stuff as well. So if you have written software for more than one week, someone has probably told you that you shouldn't reinvent the wheel. And that might have been, you know, in sort of a design review, or it might have been, you know, in a sort of one-to-one or a conversation or something else.

2:08And the main, you know, advice behind it, which was really solid advice, was that, you know, the wheel already exists. So, you know, if someone else maintains it, your time is better. spent, you know, on pretty much anything else. Your time is better spent on building the things that only you and your team can do. And it was really good advice because the math was often sort of pretty lopsided and skewed. So, you know, building meant months of sort of engineering work, and you often didn't get any value out of it until those months of engineering work had been spent. You have to have a team to run it, and you're on call for it as well. And like all three of those things kind of suck. Um, and the benefits of buying it where you could buy it and you would have it live within like a week or three months depending on your vendor. Um, and someone else is on call for it, which is also great.

2:59No one gets woken up at 3 AM or something silly. Um, and buying also had its own costs, which, uh, you know, they're starting to see a lot of price. across the SaaS industry. This was, I wouldn't say a random stat, a carefully found stat. SaaS prices have increased 16.4% over the year, and a lot of SaaS licenses end up going unused. So like 65% of people have a license for something, and they just never use it because license management is hard and painful. And 79% of ITDs have been hit.

3:44with IT price rises at renewal. That all sounds like it kind of sucks, to be honest. So now we have some really great capabilities. In this time frame, we're talking from March 2023 to April 2025.

4:01The price of input tokens for LLMs has gone down by about 75 times. Now, this isn't like an apples to apples comparison, but this was... using a benchmark called MMLU, and both of the models here scored about the same. And as you can see in that two-year period, the price to essentially achieve that has gone down by about 75 times. So, you know, you get the same outcome, but 75 times cheaper. And the more interesting thing is that agents are now finishing longer tasks as well. So this is essentially, you know, what your money gets you. So there's something called METR, M -E -T -R, which measures the length of a task in human expert time that a model can finish on its own 50 % of the time, which is a very confusingly worded benchmark, but works pretty well for this.

4:57So GPT -4 managed to do this in about four minutes, and Claude Opus and some other models, you know, they've managed to, you know, really... high, and now, you know, you've got clawed Opus 4 .6, which can, you know, work on a problem for sort of 12 hours straight. And the benchmark actually hit its limit with models like Opus 5 and Opus 5 .5, where it actually can't measure it anymore, which is kind of crazy to see how in such a short period of time we've, you know, basically exponentially grown the tasks that AI is, you know, able to handle.

5:39handle. But we've talked about writing it. Writing it is kind of the easy part. You spend a weekend writing it, all looks good. But what we haven't really talked about is the owning it part. Because when you build something yourself, that means you also have to own it end to end. And that means running it, fixing it, and replacing it, and getting woken up at 3 a.m. when it subsequently breaks because your entire company is now. dependent on it so the answer is almost always yes you can build it but you should start to think about should we build it because one of the the trends that we started to see is we've now entered like a world of essentially personal software where everyone can build a piece of software that you know does exactly what they need and that's great that is amazing the problem is is when you have a company of 1,000 people each building personal software to do exactly what they need, that makes it a little bit more difficult because you have essentially 1,000 versions of slightly different but kind of the same software.

6:47So one example we're going to go through is something called AI Code Review. Now, some of you may know this example because we've blogged about it on our public blog before, but this is our internal AI Code Reviewer that we use. on merge requests and i was the one that built it so i am perfect to talk about it because i cannot stop yapping about it so code review is really great for catching bugs and sharing knowledge but it's also one of the most reliable ways to slow down an engineering team if you've ever you know created a merge request you know that sometimes you may be waiting 8 12 hours for to get a review and that can really slow you down no one really you know wants to be slowed down like that um Your merge request will sit in a queue, and then someone context switches to read it.

7:33They might leave a few nitpicks or small bugs. You go fix them, and then rinse and repeat, and things take days to be merged. And this is something we felt internally as well. Before we had this, the median wait time for a first review was often measured in hours. But now we found that it's actually measured in minutes, and we really love that. So we didn't start by. building something. We were very early into our sort of AI journey, I guess you could say. So we tried some off -the -shelf AI review tools, and some of them were really good. Some of them were not so good, but basically none of them were flexible enough for what we wanted. We sat in vendor calls, and we talked about, you know, all of our requirements and things like that, and, you know, we had sort of salespeople saying, yes, we can 100 % do this, but that doesn't guarantees, and we just found that that level of sort of, you know, lack of control was going to be really problematic.

8:34So we just sort of hacked something together to start with. We generated a git diff. We had a really quite dumb prompt, and we just told it, like, go find some problems in this code and then let me know. And honestly, it wasn't that effective, but it was better than nothing. And... Since then, we've really over-engineered our AI reviewer based on the OpenCode harness, and we've been using it for pretty much everything for the last six to nine months. And now it's a bit more than a git diff and a dumb prompt. It's something that is really a strong tool in our arsenal. So we built the first version in like an afternoon. I kind of looked a bit through how Claude Code was doing their review.

9:21tool, figured out that they were doing some, like, MCP magic and stuff like that, and I built our own version on GitLab using OpenCode, and we've since had, I think this number is actually a little bit higher now. It's probably about 500. We've had 500 merge requests to our internal AI reviewer, and that's not just from me and my team, but it's from people all over the company, and that's one of the great things about, you know, building your own internal software is if someone in the company sees a bug with something or sees a problem or thinks the behavior is... be different, they can just make the change. And with AI, that's incredibly easy. And what we've seen is there have been hundreds of people that have chipped in to fix bugs and add extra functionality and really sort of elevate the internal experience to be faster, more secure, et cetera.

10:07So we've published some stats previously in our blog post about what this is sort of the impact it's had on our engineering. And I think- probably for the last month, or last nine months, sorry, every merge request has had to be approved by the AI reviewer. And you can imagine how angry engineers get when they have to fight with an LLM. It turns out some of them actually get really quite angry. But we've only had to block about 16,000 mergers. And that's over the past four months, which isn't that much in the Cloudflare scale. And what we found is that when we do raise these issues, most of the time people do act on them. And if we run into issues, if things start failing, we're able to fail over quickly.

10:54And if everything caches on fire and an AI provider goes down, there are break glass mechanisms to make sure that things carry on working, because we still want to be able to ship code. So since March, we have run just over a million reviews through our internal AI reviewer. We don't know how that stacks up against public ones, but I think we're pretty proud of that number. That is a pretty big milestone for us, and I think that's really highlighted that, you know, people trust the tool. It's working. It's doing its job. And, you know, we've had success by building our own. And that's across about 350 ,000 merge requests and just under 7 ,000 repositories. And we actually published a blog.

11:41post yesterday about some cost savings we made to the AI reviewer. And now it costs us just 67 cents per review. If you're interested in checking out that blog post, it's on the Cloudflare blog. We managed to reduce our AI inference bill by 68% by just being absolutely ruthless with optimization. So go check that out as well because that's a super interesting case study.

12:08So One of the things that we ended up iterating on was that, in the very first versions, we flagged everything we could because we thought it was useful. And then we listened to a lot of feedback from people and found that, actually, we were raising a lot of issues which were kind of like non-issues, you know, nits, small things, things which, like, we probably shouldn't be doing. So we went back to the drawing board a bit, and we decided, OK, let's only flag things that people actually care about and that people are actually going to action. And when we did that, we found that actually people action those things more because they're more important and they're more valuable and they're a good use of people's time.

12:48So the thing that we can't do with the AI reviewer, and this is where, again, we had to take a step back and figure out, you know, is this the right tool for the job? Have we built the right tool for the job? Or do we need to extend the tool? The first one is that because of the size of some of our repos, we can only... really look at the git diff. We can't really look too closely at the design behind it. Sometimes there is no design. Sometimes it's a simple change. But other times, there may be a whole design dog behind it. But we just can't pull that in at runtime with everything else. It's difficult to check cost cross-system impact. So because we may have potentially up to 100 services that are in a path that this changes in, we certainly could check.

13:34that, but it would probably take about three hours, and the person making the MR would probably be really quite annoyed that we're delaying their MR by three hours because of it. And there's things like time-dependent races, where you have to run race checkers and things like that. It's really hard to spot those in a static diff. Sometimes you just need to run the race checker and things like that. And very large changes. Some of our MRs are really, really big, like millions. of lines, and it's very difficult to figure out a way to put that into the context of a model that only has a million-token context. So you have to do some crazy stuff to get that to work. But we found it doesn't replace human review. No matter how hard we try, humans will always have really good input.

14:22So where this took the conversation is it really gave us confidence that, in the right circumstances, it is a fantastic idea build your own software, because you can build it to do exactly what you want, how you want to do it. And when you have a request come in or when you want to support something new, you're not then dependent on a third party, a vendor, et cetera, to go build your thing. You can just build it yourself. But you should also ask, should we build it? Just because you can build it doesn't necessarily mean it's the right idea. And you should try and prototype it. That's one of the big things that we found for our software is it's difficult to commit lots of people and lots of time to a project that's a very big maybe.

15:10So spend an afternoon prototyping it. See what you can slop together. And then show it to people and say, hey, this is what I built. Is this something we want to invest further on? And then later, expect your tools to be replaced. So one of the big, I guess, anti-slop kind of things that I've heard is that you're just building tools that are full of tech debt and it's going to make everyone's life a hell. That's probably true. And the way that you shift your thinking about that is, actually, if it's really easy for me to build a tool like this, I can build the replacement in six months with better models in probably half the time with 100 % better features. So actually, I don't need to hang on to the idea that this software is going to be around forever. It might only be around for six months, but because we're solving the problem now, we're actually probably saving a lot of human capital and human time in solving the problem now.

16:04So when building makes sense, it almost always makes sense to, you know, build it if it's sort of critical and there's no product on the market that really does what you want. That's like a, you know, that's like a golden egg. If it's critical... but there's something on the market that really, really perfectly fits what you want to do, buy it, but make sure that there is an exit from the vendor. The last thing you want is to be vendor locked. And if there's an exit from the vendor, you can then move to your own tool when your requirements have changed, your business has grown, and you need to do different things. If something is not really that critical, sometimes it just makes sense to buy it. And actually, the conversation there is more, does it cost us more to buy it or to...

16:50just spend the tokens on building it. And unfortunately, tokens are getting really cheap. So that's kind of changing pretty rapidly. And the final one, build something small. If it's not critical, then you know what? Just try and build something and see where you end up. So some checks before you build.

17:12Try and actually figure out what's wrong with the tool you have. So if you're using a lot of legacy SaaS products that I don't think I'm allowed to name. You know, try and figure out what's wrong with them, and then figure out if that's a good reason to replace them. Build something small enough that you can just throw away if it doesn't work. Don't spend five weeks on it. Spend an afternoon. And make sure you are also considering, like, the security, the replacing, the maintenance, the on-call burden, all of that kind of stuff that comes with it, too. And reuse what exists. I reused a lot of stuff that existed for us. So open code, workers' KV, AI gateway, all of that stuff really existed so we could just reuse it. And make sure you measure it as well. Make sure you add telemetry, measurements, everything you can possibly think of because that will come in really useful when you're quantifying, okay, here's the impact of the thing that I've built.

18:06And then finally, keep a way out. Make sure you have an escape hatch to leave if things go really badly.

18:14Building got cheap, but owning it is still kind of a problem for now, but that is changing. So think about the why before you build it. And that is it. Thank you very much, everyone, for coming to my talk. This was my first talk, so I hope I did a great job. There's a... Thank you.

18:35There's a link to our original blog post there on our AI Code viewer, and then there's a recent post that we made about, you know... about the changes we made, go give it a read. It's super interesting, if I do say so myself. And then I will also be in the speaker connection language after this if you want to come talk to me about basically anything. And don't forget to give some feedback. And we have one minute left, so if anyone can get a question in one minute, you're more than welcome to. Right here at the front. If you've got this AI review box.

19:14Okay, so you've got an AI code review bot, and it is essentially the same model that the developer is probably using on their laptop, so wouldn't you consider any finding that happens in the PR to be already wasting the developer's time? Like, is there a reason why, I mean, yes, it should run there too, but shouldn't all of that same skills, prompts, and context just be part of the AI code review bot? of the developers pre -comit ideally yes and if ai was deterministic that would be fantastic but that's one of the problems is ai is almost you know never deterministic so we have to take kind of an adversarial approach where you have to say you know not just review the code but review it from like five different angles and that's when we build in a lot of extra tooling and prompts and you know extra stuff to essentially make sure that we're reviewing it from like all of the different kind of angles that we can we can find.

20:07And we've actually got a local version of that same AI reviewer that people can run locally. But again, because of the non-determinism, it makes it a little bit difficult to track everything all the time.

20:21So with regards to the skills and things that you set up for your developers in relation kind of to what he had just mentioned, how heavy is that footprint for your teams right now? When you say adherence to engineering standards, and things like that, how much of that is on the machines and being leveraged as part of the build-out versus then being checked? And is that something you guys track internally, where it's like, yeah, we have a library, a set of skills, a set of things that everyone should be leveraging in these development tools to keep our standards in place? Yeah, 100%. So the internal standards one is one of our big focuses at the moment is, how do we help AI model? generate code that is naturally adherent to those standards instead of doing it after the fact in review. So we have things like linters, for example.

21:08We have a bunch of RFCs, which include basically all of the different standards that they should adhere to. That's past agents' skills. They don't necessarily always get invoked when they should do because sometimes it's difficult for agents to figure out. But we really try to bring that back to the developer machine so that it's... you know when the code gets pushed up is it's as ready to go as possible but what we've also found as well with with recent models is they'll figure out that they need to wait for the ci to complete once they create a merge request they'll see the comments and then they'll just go fix them and push them back up and you know loop until everything's everything's good to go um so what i found is you know especially with models like opus 5 .5 for example i've not really needed to to guide it that much to get it into a good sort of production state it can just figure it out most

22:03I think we are done then, folks. Thank you very much.