0:02Hey, everybody. Thank you for attending this session. Today I'm going to talk to you about routing at the edge and some of the stuff that we're doing at Ramp that's pretty cool around the routing setup that we have and the complexity of the system that we've built, which in the end is a complex system but made simple. So...

0:25Okay, so that thing is not working.

0:33OK, thank you. So this is me. I'm leading production engineering at RAMP. I've been there for a bit more than a year and a half now. And some of the thing that we've been started doing since I joined RAMP is part of the coolest work that we have, which this project is part of. But... To get you a bit more sense of like what Ramp is and what Ramp is about, Ramp is a company with many customers, with a lot of different like currencies that are managed in the platform. But one of the things that it provides to companies is the way to save time and save money and do both at the same time. So basically we are aiming to be the financial operating system for companies and provide you with.

1:24all the things that you need in your systems to be able to optimize the way you are doing finance. At Ramp, we're probably one of the biggest customers of Ramp 2. We're dealing with a lot of contracts. We're dealing with a lot of expenses. And honestly, and I'm not saying that because I work at Ramp, but maybe I am in a bit, is like one of the system that's the easiest to deal with when you're managing people or dealing with big contracts or dealing with a bunch of systems. And overall, one of the statistics that's not on that slide but that we talk about is that on average the customers are saving about 7 .5 % of their money on the first year they're moving on ramp and on years after that. So one of the things is that when you have a system like this you obviously are attracting a bunch of customers.

2:12The company is growing crazily and as the company grows increasingly we are having an increasing number of scaling problems. that we have to deal with. And if you imagine infrastructure as basically city building,

2:29you end up having a city that you build, and you have initially one neighborhood. Imagine that as a small village where everything is located. And everything is cool until you get to the point where that space is way too cramped. And then you start having problems, either about putting more people in that city, or even about the roads. that are bringing people to that city. And so slowly, slowly, you start having traffic like this. That picture actually comes from a street in Montreal that's not too far from my place, which is known to be cramped like this pretty much every day around the same times, because everybody is going to work around the same area, and everybody is getting in those roads, and they get cramped, and there is more space. And so if you overcommit your ..

3:16infrastructure, you start having roads that are like this everywhere. And your systems are getting slower. There is more risk to your system. And so what you want to do generally is start thinking about, oh, do we kind of like evolve that system? At Ramp, we had the same problem. As I said, the company growing, the company bringing a lot to other companies was like bringing a lot more customers. And so we initially built Ramp in a way that made sense as you're building a new product and you're not. yet thinking about, oh, is it going to scale when we get to, like, thousands of customers? Initially, you build your product for, like, the first few customers, and then you start growing, and then you start hitting problems and start figuring out, like, how to get through those problems one by one.

4:00We obviously grew Ramp in a way that was making sense as we were growing, but a lot of the things that we learned, we learned along the way. The product kept evolving. We started adding new features. We started adding a bunch of other things, and as the company... growing, as the number of engineers that you are working is growing, not everybody necessarily has the same framework about, like, oh, are we going to build that new product, or are we going to scale it? And finally, we needed to keep up with that demand. So as we were working through that, we were like, what is the best way that we can think about this, and how do we get, basically, to the next step?

4:35So this is a very simplified schema of our architecture, where basically a customer was coming We had, like, one load balancer and the generally monolithic infrastructure where everything is. We had, like, a few services, like, mainly one huge database. And the target was getting into a system where we can have multiple targets, right? That kind of approach is basically called, like, a cellular approach. One of the things that it brings as an advantage is that you can keep scaling as much as you want as long as you keep adding, like, paths for reaching your system. Some of the things that we get as advantage from that were related to the targets that Ramp is having about the evolution of the future, right? One of them, as I was mentioning, is also like getting to scale.

5:20So if you add more parallel neighbors into your infrastructure, you end up having like roads on the new places that are not as used anymore, and you can start basically keeping scaling. You get new customers, you put them in the new cell, and then you have like... more scaling on that front. Another thing that we're getting from it is also fault isolation. When you make changes to your infrastructure, as your company grows, as it gets huge, and as you have systems that are basically at the limit of their performance all the time, doing any change gets very scary. You get to a situation where like, hey, I need to do database upgrade. Or do we do that in a safe way, and do we have the right process? that it doesn't affect our customer, or that even if you have a maintenance window, that maintenance window is as short as possible.

6:12If you have, like, a code that gets deployed and actually was wrong code that shouldn't have been deployed, then all of a sudden 100% of your customer are affected. But when you get to a cellular architecture, you get that isolation where if there is a problem... Sorry, in one side of your infrastructure, the others are unaffected because you can get to a point where if there is an issue, whether it's... because of the load of that specific system, whether it's because of a change that got deployed there that hasn't been deployed in the other places yet, you can have better observability of what's happening before affecting the rest of your platform.

6:45Finally, one of the things that we are trying to do also as we're growing and some of the things that we've announced this year is about extensions in other places. We are looking at extending in Europe, for instance. And some of the things in there require also being able to support data residency. So how do you support data residency if you have only one monolithic infrastructure where everything is happening? Well, generally you can't. And so you start rebuilding everything. But if your infrastructure is already supporting like a cell approach, you can just start deploying wherever you want. You can start deploying in different areas of the world and it doesn't like limit you about the geography of your deployments. The main problem with All of that, though, is as you're adding more ruts, you're basically getting more navigation concerns.

7:36Because if someone is getting to your platform, where do they go? And you generally don't want the customer to be the one that has to figure out where do I need to connect for my system. I do know that a bunch of companies are doing that. I know that there is a bunch of systems where generally, depending on what you're doing, you're going to have a different URL that you need to connect to. But I also... think that user experience is very important and customer experience is very important for a company and you generally want to do things in a way that's entirely transparent to customers one of the advantage also of making it transparent to customers is if in the future you need to repurpose or reorganize part of your infrastructure that becomes entirely invisible to your customers you can move them around to a different cell because that cell is all infrastructure one of the thing that I think probably a lot of you are gonna with that, like the holy grail of infrastructure and a lot of people working on deploying infrastructure is the idea of immutable infrastructure, where anytime you make a change, you build something new, you move your customers on the other place, and you destroy the old, because that way you never have the risk of big changes happening on that thing.

8:43And so if at some point you're like, hey, I built my architecture using VPCs or like a structure that way, but I decided that I don't want to use VPCs anymore, I'm going to use a gateway. You can do it, because that's not like a very complex change in one place. This is you build something new, you test it, you kind of start moving customers there, and once you finish moving your customers, you get rid of the old infra. And so if you want to be able to do that, but if your customers are targeting specific address, that makes for very awkward conversations of like, hey, by the way, starting Monday after 9 a.m., please use that URL, but because we're still going to be in migration, please use the old URL for like 10 more minutes, and then we'll tell you. when you can start using the new one. That gets very complicated. You're gonna start having support tickets all over the place. But if you make it in a way that's transparent to customers, that makes it way easier to communicate.

9:32And one of the things that's challenging in there is how do you handle the whole routing? And so this is, sorry, this is mostly what that talk is about, is thinking about the routing process and the work we've done.

9:49In terms of the requests that we're receiving at RAMP, we have many, many different systems that are reaching our platform. These are generally the most common ones. Of course, like webhooks and third-party services, which are things that we don't have control about the shape and form of what they're sending us. And we have to basically deal with, how do we handle these requests? Some of them are coming from bank processing. Some of them are coming from third-party providers that we're using for different aspects of the system. But we need to be able to understand where is that requests coming from, where is it supposed to be going, and how do we route that. Everything that's web UI and mobile apps, we have a better control over it. So we generally don't want to make too many changes at once, but we have a bit more control about what we're doing for it. So if we want to add any information in our request, we can do it.

10:37That makes it easier. Everything that the developer API requests is also something that's a bit in between, because we technically own the .. that we're giving to developers. But as you have developers already using your platform and already calling your APIs, you don't want to change that on them. You want to make sure that whatever they're using today keeps working, and we need to handle that. Everything that's credit card authorization, I put it separately, but technically, it's also part of third-party services. The main difference is that when you do credit card authorization, you have an extra constraint, which is how fast you need to answer. And so, that means that. We have optimized system to be able to answer those requests, but if you add a layer for routing those requests that is adding latency to that process, all the latency you're adding in the routing process, you cannot have to make your decision.

11:26And when you do credit card authorization, there is a bunch of systems that kicks in, including systems that are supposed to be dealing with, is that a fraud request? Is that something that we should be authorizing? Is there capacity for that business to make that expense? Is that user of that space? credit card having a fund that actually is authorized to make that specific transaction and all the decision needs to be made and even if we optimize that system that takes some time but you want to give that system as much time as possible so there is no risk of expiring like having a sorry like eating a time out of the request that is made by the credit card providers and finally some of the the things that that started appearing more and more in our systems this year is all the things that are agent workflows.

12:12So, or are agents interacting with our systems and the fact that there is more and more of agents interacting with systems, so more and more of requests. And everything that's also MCP tool traffic. So, basically when an MCP tool is trying to hit a system but might be hitting multiple systems at once or trying to kind of like do operations across multiple users or multiple different businesses, how do we handle the routing for that and how do we make it? So all of these are traffic that we need to kind of like handle with our system.

12:45Part of the constraints that we put ourselves in building this, which is aligned with this request, is that we want it to be having that system that's invisible, as I mentioned before, dynamic and fast. So we want to make sure that if something moves around or was routed to a given cell at one point but needs to be routed to the other cell, it needs to happen very quickly. It needs to be efficient enough. and we need to have control over it. Like, it needs to be something that can be reasoned about and can be understood when we're tracking our system. And finally, it needs to be compatible, because if you understand what I'm saying about all the different ways we're receiving traffic, even the ones that we have control over, the more change you need to make in the system to be compatible with whatever you're building, the more complex it's gonna be to roll out that new system you're building.

13:33And so if you want... it to be efficient, we needed to figure out how to route the existing requests in the path that we had. And this is where I'm introducing you to GoThere and FerrisWay, which are basically like our two routing services, our two routing systems. So GoThere is the original service, I'm going to talk to you about it a bit more in a moment, and FerrisWay is basically the very cute service that's kind of the subject of this talk today. Gother is generally the authoritative service that knows about routing and knows about where things need to go. And Ferris Way is its friend that makes everything quicker, faster, more efficient, and closer to the user.

14:18Ferris Way has been in production at RAMP since January 2026. So it's been a few months now. And as we rolled it out, it took us a few weeks only to move it to... handle 100% of our traffic. Ferrisway is a Cloudflare worker. It's built in Rust, compiled in WebAssembly. And it is currently dealing with about half a billion requests It is dealing with more than actually 5,000 requests per second on average. And it's dealing with about 10 terabytes of traffic per day. And it's doing that in a very efficient way.

15:00This is generally the architecture that we have for our system. So as I said, Ferrisway and Goddard are working hand-in-hand on the system. Goddard is a cloud service that we have deployed behind the scene. I'm also going to talk about the way it's working in a moment. But Ferrisway is basically the first point of receiving requests. But Ferrisway doesn't know everything. It tries to know as much as possible, and it tries to make decisions. So it's extractive. the information from requests. It's trying to resolve where that request is supposed to be going, and we have, like, multiple layers of cache that I'm gonna talk about, too. And when it finds a route for the specific request, it's gonna be skipping everything else and just sending the request to the place where it should be going. But when it doesn't find that, it's actually relying on Gother to be making the decision.

15:49The reason we have that infrastructure is because in every system, you don't want to have multiple... source of truth because having multiple source of truth means that sometimes you might have different information in two different places and that gets harder to kind of like figure out what needs to happen so what you generally want to do is having a single source of truth that's accessible from different system and so Ferris way takes care of routing the information that it has gother as the information that Ferris way doesn't so when Ferris way doesn't know what to do it relies on gother

16:23The way we're dealing with requests, which is good to understand exactly like the different layers that we are using here, is that depending on the request that we're getting, there is different source for the information that we have to grab to decide on where that information, like that request needs to go. Sometimes it's a request header, which is generally fairly simple. One of the things that's actually pretty nice at Ramp is that a lot of the requests that we add coming from the mobile application or the web application already add a router indicating which business those requests are for. And so that makes it easier because we can grab that information, figure out where that business is currently located, and route the request directly. But sometimes we have requests that have the business information as part of the request path or as part of the query string.

17:11So we also needed to be looking at this and be like, oh, if I don't have the information in the header, I need to look at this in a different way. different place, and so the request path or the query path sometimes can contain that information, and I can extract it, and then root based on that. But in some situations, I will be rooting based on a user email or user email domain, based on who is the person that's currently authenticated. And that information can be sometimes on a JOD token that would be, like, stored, provided as a header, too, but needs to be, like, unpacked. to be looked at and then figure out the information in it. And in some specific cases, which is generally probably for Webhooks, like mostly for Webhooks, you're going to have the information directly in the request body.

17:58So basically reading the body, finding the right information, and extracting that. And so when you look at all of these different paths, what is happening in the routing? So this is the way Gothar is working currently. We are looking at how do we... the routing key for the request. So, basically one of the methods I just talked about. Some of them are actually a bit more complicated than what is on the slide. But we have about, I think currently, about 100 different rules for routing requests based on path, based on different source. But as soon as we have a routing key, we're trying to resolve that routing key as quickly as possible. Remember what I was saying about trying to have the least latency possible on the request.

18:45So we're trying to use the local cache of the processes, which will have the information about what has been resolved recently. So we're using a tiny LRU algorithm for that, which basically tries and optimize for things that are currently being routed or requests that are currently being made, because most of the time, when customers are interacting with the platform or when something is happening, it's going to be the same keys coming over and over for a short period of time, and it's going to change to another. So we try to optimize for that pattern. If it's not in memory, we have a layer that can be deployed in Redis. Actually, we're not using that layer currently, but there is the capacity of adding that if we need to kind of add more caching between the different processes that are running. But one of the things, actually, is that Ferris Way kind of removes the need to have that, because a lot of requests are actually not coming to our router and dealt with directly at the edge.

19:39And finally, we're using DynamoDB as source of truth. So all the changes in routing, all the routing keys and mapping to cells are actually stored in DynamoDB, and this is our source of truth. So Gother manages that source of truth and owns that source of truth, and every time we're trying to do a resolution, if it gets to DynamoDB, it then gets cached into memory and using that tiny layer approach to kind of optimize the next request coming to that process. Once we have got the information, we then route to the different cells.

20:16To give the idea of anatomy of a request, this is what generally it looks like. So this is the different areas that we're looking at when we're extracting a routing As you can see, there is different place where that information is. So the more we're looking at the top. of the request, the faster we can make a routing decision, and the better it is for proxying routing. Because the way most of our routing works, and it's the same for both Goddard and Ferris way, we're trying to get the information of the routing key as soon as possible, resolve where it's targeting, and then the rest of the request gets streamed. So we don't have to cache that in memory, we don't have to keep reading, but until we know where to route the request, we don't know where to start sending it.

21:03So we need to buffer as much as possible so we get the information before routing. That's something that's going to be interesting when talking about Cloudflare workers too. As you can see, everything that's about the path of the request, everything that's about the query string of the request is generally toward the top, so fairly quick to kind of like route, even if it requires sometimes some of the parsing of the string. Everything that's headers is generally fairly easy to access. everything that's like Jot is also fairly easy to access, and then you get to load everything in the body once you finish loading the metadata of the request. That means that when you look at this, there is different cost to all of the extraction. So the small snippets here are actually accurate snippets in YAML that we use to configure the routing in both Goddard and Ferris way because they use the same routing rules.

21:57We have one file that is shared between And you can see that if I'm using a request header, it's generally fairly cheap. If I'm using the request path query, it's generally fairly cheap. JWT is medium cheap, because even if it's in the headers, I still need to do some processing to unpack the JOT and then find the right information inside of the JSON that has been unpacked. So that is extra steps I need to take, which means a bit more expensive in the process. But if I have to look at the body, that very quickly becomes expensive. And one of the things that's interesting is if I'm showing you those two different snippets, which technically are basically the same, there is, however, a very important difference that impacts the routing that we're doing.

22:47If you think about the way we have to do that, so I was mentioning it, we're doing like processing of the request. If my client is making a request, an answer for the server, the answer is generally going to be fairly cheap because I know which requests have been routed, but for everything that's proxy, there is the buffering step. So if I go back to my code snippets in there, that looks like something like this. Right? This is, the section that's highlighted is the section that I need to read to be able to make a routing decision.

23:18part of the things that we've tried to also optimize for is that on some requests that we are having the control of the payload, we're trying to see if everything that's contained in the request that we want to use for routing can be put toward the top, because the least of the requests I read, the most efficient I'm gonna be in routing. And so, now that we're thinking about the way this works, this should start to make sense about why having workers deployed at the edge and making these decisions closer to the user are actually making a lot of sense for us. because everything I need to buffer also means that I need to request to receive the information from the user before knowing where I'm sending that request. And so I need to keep that directly at, like, in my process.

24:04And if my user needs to send me a lot of bytes before I start sending that to the server, and imagine in a situation where, like, actually I have a cell that's actually closer to my user than the actual routing decision is made, I end up taking, like, a detour to be sent in the right place. So instead of deploying routers everywhere to be trying to serve the traffic, using Cloudflare and a Cloudflare worker gives us that presence everywhere. And so everything that's happening, whether it is transactions that are happening, whether it is web books that we're receiving, whether it is a user using the web application or the mobile application, we can make the decision as close to the user as possible and try and send that as quickly as possible to the right place.

24:47And that works pretty, sorry, that works pretty much the same way as Gothar is working, except that we're using a lot of the Cloudflare-provided technologies, some of them that I like to say that we're, like, overexploding a bit. We are doing the same extraction process. One of the big differences that we have between Gothar and Ferrisway is that Gothar is actually reading the YAML at runtime, parsing the request, and loading that in memory. For Ferrisway, we're actually compiling... those rules in native Rust right at build time, which makes the worker as small as possible and as efficient as possible in running through those requests. Part of the reason is because when you have a service that you deploy like Gother, most of the time it keeps running all the time. But when you have a service like FerrisWay, which is a Cloudflare worker, there is a lot more times where that service gets started, and a lot more times the service gets stopped.

25:38And so you want to optimize also for the start time of the service even more as everything else. If I need every time my worker starts, and it's a cold request, and I don't have a worker running, that the worker starts, load the LAML, parses it, and then make a decision on it. I'm adding a lot of latency to the request. Maybe not to the follow-up request, but that first one is gonna be impacted a lot, and that's not something that we want to accept. And so because of that, we compile all the rules directly in the code, and the extraction is generally going very, very quickly. One of the things I like to mention about it too is, as I mentioned earlier, if we have bodies that are way too heavy, one of the problems that we've hit as we had deployed FerrisWay and started using it for some of the heavier payloads is that sometimes it's trying to look for a routing key, and sometimes there is no routing key.

26:29Or sometimes the routing key is at the end of the payload. And it happens that some of our third-party vendors are sending pretty heavy payloads. And well, We had some things that were kind of like limiting the size of what we're caching. But at the time, I actually discovered, like, the way Cloudflare workers are being shared between requests, and the fact that there are specific times where you can have multiple requests handled by the same worker at the same time, which means that we started having out-of-memory exceptions in some of the workers because they were packing, like, webhooks and trying to find keys in there. So, we solved that by basically having, like, a very, very tight limit on how much we're buffering in FerrisWay. And one of the things that's cool about having a service like Gother that can have whatever capacity we want behind the scene is that any request that FerrisWay is like, I haven't been able to parse this, I haven't been able to find a routing key, I haven't been able to find what is needed, can just be then sent directly to the background process, which then has a lot more capacity to deal with it.

27:30And so that process is able to handle multiple megabytes of data without issue, even like 100. of megabytes if we wanted to, but FerrisWay is not. So we just use FerrisWay for what it's good at, and then rely on Gother for what it's not. In terms of the caching, we are also trying to optimize for how do we get answers as fast as And so as a Cloudflare worker is used for multiple requests, we have the same approach, where we are using a tiny LRE deployment inside of the worker. And so we are using the isolate memory, which, again, gets wiped every time the worker worker restarts, but it's totally fine. And that isolate memory gets built up based on requests that are passing through. So this is the part that I like to say we're a bit abusing the system, because we're really building a cache inside of the worker to make it very efficient.

28:19But we're talking about less than 10 nanoseconds for having answers when we're using the isolate memory, which is a huge optimization if you're receiving a few requests coming from the same place. Because if you understand a bit how Cloudflare workers are deployed and they are kind of like built, as I was saying, there is a lot of cold start because if a user in a specific region is trying to call a worker that's not yet started, there's gonna be one worker starting in the specific pub that they are reaching out to. But that worker, if there is not that many requests, is gonna keep being reused when it's needed until after a while there is expiration and a new one gets built. But if you think about it that way, if I have one user that's connecting from one specific region, then once I've done the resolution, for that specific business, then every other request that user is making in the span of time it's making requests, which if you load like the web UI or even the mobile app, there's a bunch of requests for assets and informations that are made at the same time.

29:10That thing is going to be super optimized because you get to less than 10 nanoseconds resolution for every request. Then, sorry, as a second step we're using the cache API. So same thing, we're slightly abusing the cache API. because we're using it for caching at the locality of the pup. So if you have your worker that got restarted because there was an expiration of its use, the next step is basically you start a new worker, but that new worker that starts will be able to resolve very quickly because it will be available in the cache API. So even if there is a cache miss in the memory, it's gonna check in the cache API, it's gonna have the answer, and it's gonna be able to kind of start doing that. And the cache API is...

29:56is in the range of like 10 milliseconds to be able to do resolution. So it's slower, but it's still fairly quick to be able to kind of do that resolution. And then when we don't have that information, we're looking at the more permanent cache that we have in Cloudflare, which is using Worker's KV. So the reason we're not using, sorry, we're not using Worker's KV for every request is because we're talking, when we're using Worker's KV, something in the range of. 100 milliseconds for requests. So it's still not very bad, but when you compare it to about 10 milliseconds for the cache API and to less than 10 nanoseconds for everything that is late memory, that can make a huge difference for many requests made by people. So what we're trying to do is optimize, really, the layers of cache.

30:43Now, that worker's KV, as I said earlier, is not a source of truth, right? It's just another layer of cache because we're trying to optimize for Cloudflare having information as soon as possible. And so what we're doing there is, if I don't have the information in the worker's KV, I'm actually just relying again to my authoritative source, which is Gother. So instead of routing directly to a cell, I'm going to be routing that information to Gother, and Gother is going to be making the decision. So when I'm doing my resolution, there's three outputs possible. Either I have a fresh hit, so that means that thing has been resolved recently, I know it's still valid, I'm going to be doing the routing. I can have a stale it, but it's still valid. So we have two layers in there. We have, like, a freshness, but we have also an expiration.

31:29If your thing is not fresh anymore but not expired, you can still root based on it, but you need to trigger a refresh, which is basically what Ferrisway is doing. So it's rooting the request right away, so we're not wasting any time, but it's triggering a refresh in the background, so then a future request will have a fresh information, and we know that if anything has changed, that will kind of... of, like, reconvene. And finally, we have either, like, a cache miss, so that information is not there yet, or an information that expired, which currently is very difficult, because I think we keep caching information for up to, like, we expire after 24 hours to improve routing. But if this is the case, then we basically it go there to get the information and trigger the refresh of all our cache layers.

32:16The way the . refresh is working is actually through the response that Gother is providing to Ferris Way. So we didn't want to add like a complex process that's running on top of whatever is happening in the worker, we just wanted to reuse the process that we have in part of the request. And so as part of the response that is going to be sent to the user, that data generally gets stripped, but Gother, when being it, is going to be making the routing decision sending the to the right cell, and when receiving the answer that needs to be provided back to Ferrisway to be sent back to the customer, the client request, there's going to be that header that's going to be added on the request. That header basically indicates here that, like for instance, business 1234 should be rooted to cell A, and that information is fresh until a given timestamp and expires at another timestamp.

33:10That information then gets put by various way when it reads that header as part of forwarding the response, and is used to be updating all of the caches. So we are updating the information in the worker's KV, we are updating the cache API, and we are also updating the isolate memory. So any information that goes back to us gets fresh everywhere. And so that means that any next request, whether it's coming from a worker in an entirely different region, could be using the KV. Or coming from another worker in the same POP using the cache API, or could be coming through the same worker, would be using the isolate memory. So this is what we're doing optimization there.

33:53The other things that we started looking at as part of building FerrisWay was, how do we unify the observability that we have of our platform? So we basically started using like a, this is like an example trace. for requests. In this case, you can see you went through FerrisWay. FerrisWay didn't have the information sent out to Goddard. Goddard ended up dealing with the routing and forwarding that to our application. And so we can see our core API in the end is being it. And the way we build that is actually relying on tailworkers. So for tailworkers, we are using, well, as I said, we have our main worker that is written in Rust. So we are basically generating logs that are being put into Cloudflare. And we set up a tail worker that's going to be grabbing all of these logs and collecting the logs that are in the specific formats that allow us to create spans.

34:45And from this information, we are pushing that to our tail collector to be able to kind of rely on traces. Of course, as we're grabbing the information, we're also reading whatever span ID that we get from the request. So we can connect that directly to customer requests. We can tail that directly to internal requests that might... have been routed again through an external path. And we have also everything that's, if the worker is not behaving correctly, if we haven't caught exceptions, if we have memory or CPU issues, we are still gonna be sending that to the tail worker, so we end up getting spans for it and we can trace that that request had a problem and is coming from this. The other thing that we wanted to deal with and that we had to improve on this year is also the deployment process.

35:37Some of the things that we were doing initially was just, oh, I'm pushing my new worker, is live, and then, oh, if there is an issue, I need to roll that back because there is a problem with it. Some of the issues that we were having sometimes was also related to cache configuration because sometimes Cloudflare worker are a bit weird with the way they are dealing with cache, even more when you're doing a proxy that makes requests because they kind of mix the server cache and the client caching, and it gets very difficult to wrap your head around. And we've had some issues related to that. And so the way we build this is actually taking advantage of the version deployment of Cloudflare workers, where we can actually deploy a canary with 0% of traffic. And setting up a special header on the request, you can specifically override the version that you want to eat with your traffic.

36:27So you can deploy a canary with 0% traffic. and still funnel the number of requests to that specific canary that you fully control. And so we can run live tests on our production deployment with all the production setup without any customer eating that production deployment yet because we have full control over it. And so the way we're tracking this is actually looking like having a worker return the version ID of its current deployment, which is like a binding you need to add on your worker. And when you do that, we basically deploy the work. Keep eating the worker with the specific header until we get the version of the current worker. And once we have that, we know that worker is deployed and ready to receive requests. And then we start running our tests. And if our tests are passing, we can then promote that new worker as the production deployment, which can save 100% of traffic.

37:16And if that fails, we just remove it and fail on the deployment. As simple as that. And it has been actually, like, helping us a lot, catching up, like, small issues. that we would have, where the worker would compile properly, all of our local tests would work, but then something in Cloudflare is not behaving as we expected to, but now we're able to catch it early.

37:38Finally on this, I think the takeaways to get from this session are that using Cloudflare workers, you can improve a lot on your routing and make things way faster than they would be otherwise. You can get closer to the customer, and you can still take advantage of the different Cloudflare services to optimize the way you're doing caching and not directly use a product like Cloudflare initially thought about using it. You still can have like an authoritative layer which allows you to have a unique source of truth and that's something that can work and that's working very well for us so far. You can still make everything very observable and linking to the rest of your system, which is what we're doing with observability. And you can still figure out ways to have this very safe... deploy and have all the testing done properly.

38:26So that's it for me. And we have about six minutes for questions. Yes. Thank you.

38:39Did you go straight to Russ Wassum, or did you start that in JavaScript and then Did you go straight to Russ Wassum, or did you start in? JavaScript and then port that over? No, so I actually didn't go straight to Rust, but I didn't go to JavaScript either. The initial ID is go there, as the name states. If people haven't caught it, it's actually go there. Like, it's actually a joke between the gopher, because it's a service written in Go, and the go there as a routing service. And so because that service was written in Go, and everything was like ready when I started I started looking at how do I deploy that in Cloudflare, I actually wanted to use the Go Wasm approach, but I realized very quickly that if you try doing Go Wasm currently, it gets very bloated, it's very heavy, like you get a worker that's super huge to deploy, which is also not very quick to deal with things, and that Rust was way better supported natively.

39:38I wanted something that was quick and efficient, that was having like a very, very quick start time, and that could be checked very easily. easily and preventing mistakes by compile-time verification, which Rust was lining up better than JavaScript at that time.

40:03Are you using Rust macros during build time to turn your YAML files into actual Rust? Yeah, I'm actually using a... build RS file that kind of like reads the YAML, which is also kind of like cool because that means you don't have any runtime dependency on YAML at all. Like it's basically just turning into like a native Rust. And it's basically generating like all the Rust artifacts as part of the build, which then can be tested. And one of the things that we're doing also as part of the testing is we have the same fixture tests that are running on both routers, either written in Go or in Rust, which is to verify that any change you're making is actually behaving the same on both sides.

40:53There's the person behind. I think that was before.

40:59Thank you. I'm noticing the read through cache being a way that you populate all of the optimal routes for things Can you comment on whether it's like optimal versus correctness? And then secondarily, if that were to, like, could you talk about some of the edge cases of like those read-through caches needed to be completely invalidated or were gone? Are you able to handle the onrush of like populating those caches? So could you talk about the correctness versus optimal and then the edge case? So in terms of correctness versus optimal, one of the thing that's . probably not always working for every company, but that's working very well for us, is that most of the time when we are doing routing, we are not gonna have routing changes that often.

41:51So it's like you're routing a business to a specific cell, you're routing a bunch of ideas that are related to that business to that same cell, it's gonna be fairly, fairly rare that there's gonna be a change in that. The way we're kind of navigating that right now is with that difference between freshness and expression. that I was talking about where you have that really, that optimization about like correctness versus like accuracy, versus, sorry, like speed. Where you end up having, we can route very quickly because that information was fresh enough that I can route it, but I'm gonna refresh to make sure that it doesn't change. And in cases where we would be moving things around, generally it's more under our control. And the five minute expiration that we are doing for freshness. right now is generally fairly sufficient.

42:36We have also control over that, because each routing key, as we're adding it in the source of truth, we can configure what is the maximum freshness and maximum expiration for it. We have a default, but we can do it per key, which gives us the flexibility that in time of migration, you could decide that the key is never fresh. And then it's never gonna get cached, and it's always gonna eat the source of truth, and then when you flip the switch, it's just gonna eat the right place right away. So far, we haven't had the need to use that. I don't know. necessarily see us having a huge need of using that, but it's something that we built in because we wanted to have some of the access. In terms of your second question regarding edge cases, I would say that a lot of the edge cases that we've hit actually have been related to how a ramp is built, and some of the ways, some of the requests we've done are, and some of the interactions that we have inside of our platform.

43:28A lot of the complexity that we are having in building this and getting into a similar architecture is because we have sometimes interaction between businesses where it's expected that all the information is in the right place and we need to figure out, oh, are we gonna route that across the place and in different cells? And so some of that sometimes gets complex because you end up having a request and be like, where do I route that request? Because I want that information to get there, but I also want it to get there, or do I structure this? And I think a lot of that is just because, as I was mentioning at the beginning, Ramp wasn't built thinking about, we're going to need that. It was built as, what do we need today, and how do we grow that, and now we're getting to the point where we need to solve that problem. And I think, yeah, a lot of the edge keys that we've hit has been related to that, not as much about the routing structure, which, to be honest, has been pretty good.

44:17We've had that since January. We've made some changes. There's been new routing keys added and new structure because we've been converting other part of the infrastructure to use it. But most of the core of the process hasn't changed at all.

44:32I think it's probably gonna be the last question. How big is your data set for all the YAML manifests that describe your routing today? And can you give us a sense of the volume and how many schema size and how complicated it all is? So you mean like the routing rules? I think right now We're probably around 100, 100 different routing rules. I might be off a bit, but I would say it's between 100 to 100. We have different routing rules, but a lot of them, as I said, one of the cool thing in all of that thing was a lot of the requests that we have for internal systems are actually using headers that are specifying which business they're trying to hit, because everything that we have under our control already was using that header.

45:20And that was actually a blessing. it was not depending on a bunch of things and expecting that. It was like actually our core application was hoping to get that header to make decisions and we're just hooking to that header to make our own decisions. The thing that have been growing the rules have been like other calls that we don't have control over. Either like the developer API because it was built to be like very easy for people to use but then we have to figure out what are you trying to eat with that or everything that's like webhooks and third parties. So that's why like Like, given the size and the number of features that Ramp has, it's not that many roles that we had to do because we have, like, one big role that catches, like, 90 % of the use cases. Do you expect that to grow as this has become, like, you're now adding more schemas and business units and, I don't know, API methods to your routing scope?

46:16And do you worry about change governance on... that core routing table? So yes, it's basically people in my teams that are mostly owning the routing changes there. So we are in the loop, and we're making sure things are working and that there is no weird rule that's going through that could be breaking. We also have a proper layer deployment where we're deploying in QA. We have an heavily used QA that developers are playing with all the time. So if there is an issue there and it crashes routing, people are going to realize very quickly. Four. the size and the growth, I think we've been doing a pretty decent job at trying to find what are the patterns that we can use that are going to be sustained in the future. We've been working with some of the product teams to be like, hey, your application currently you're using this.

47:04Can you actually add a business ID in there? And so for some actually third-party vendors, we were able to work with them to have as part of the response, like, whatever field we wanted in the payload. We were able to inject the business ID directly in the payload that we're now receiving as part of Webhooks, and we can root based on that. And so there is a lot of the work we've done has been also to kind of figure out, hey, we're trying to solve the problem, but we're trying to think about where it's going to go in the future, too, and how do we kind of have some control over that? So is it going to grow in a number of rules? Probably, as we have new features and new things, as we have new third-party vendors. As we get to other countries and expand all across, there's going to be vendors that are that have their own way of doing things that we need to figure out, but is it going to be going out of control? I don't think so, at least not at this time.

47:52Okay, thank you very much. I think we're at the end, but I'm happy to chat with you after. Okay, thank you everyone.