Since the MS team is gearing up to announce Throttling feature coming up in April we wanted to already touch base and share what the feature is about and how it will help OData and Custom Services -based integrations to avoid timeouts and other failures due to system busy conditions.
The Challenge
The idea of service creation is claimed to be produced through the analysis of users’ certain patterns with data manipulation:
- Users have a large workload time what eventually creates struggles to get into the system. There are cases where they start to consume all resources available causing some level of starvation for other requests users need to process in the system.
- Eventually, when some parts of the workload exercising a lot of resources, this leads to different conditions whereas the system is being slowed down and is not able to service any request reliably due to high latency, low throughput and etc.
- This, in turn, will make users hit retry, which happens to be on a constant basis. Unfortunately, hitting twenty times retry just to see that whether the request got through will eventually cause further damage to the system, leading to an exponential increase in loads, raised latency, and timeouts.
Introducing Throttling
The best way to describe throttling would be through highway system example – when you defined a certain level of infrastructure, there are always constraints built into those infrastructures which are very similar to freeways.
When you have this kind of ramps allowing vehicles to merge into the freeway, there are sometimes, depending on traffic conditions and other circumstances, bounds to have certain congestions that take place in these systems during the day. Oftentimes, if gone unchecked/unmitigated, it could lead to a lot of issues including long waits where no cars are able to move to get on or off the ramp.
With a new feature, we would like to introduce throttling manager. Essentially, it’s a mechanism to monitor what’s happening within the infrastructure, identifying patterns, gridlock, and congestions to auto-adjust or mitigate the traffic. That could be a pause for a certain amount of time before another vehicle can enter unloading system infrastructure giving time to break up the gridlocks or create some space for additional vehicles to enter.
The interesting part here is that the “traffic light” does not move between the colors in a constant manner. They are taking real-time information based on a certain segment of that freeway auto-adjusting to how long it needs to halt a particular vehicle entering the freeway.
As the result, throttling manager, once implemented into the F&O platform, will essentially be doing exactly the same.it will be looking for various signals that it’s able to collect and gather, inferring information to recognize congestion. Then, throttling manager will provide users with a signal, which is basically an HTTP 429 response.