Angry 3rd Party Login Cycle
Problem
API performance degradation under unexpected load.
My Role
Performance investigation and remediation.
Stack
.NET, nginx, Grafana.
Summary
The client is a trading platform. They have a backend API used by their own frontend apps (webtrader, CRM), but also available for 3rd parties willing to use their own frontend tools and consume just the API. The client approached me when one of their instances became slow under heavy load that seemed to come out of nowhere.
A quick look at Grafana showed a likely offender: one of the brokers that was integrated directly with the API was sending an unusually high number of requests. Further investigation showed 2 concrete problems:
- The broker apparently was using a simple script running in a loop that tried to poll some API endpoints as fast as possible.
- Their requests had a very specific pattern: login -> business logic request -> login again -> business request -> login yet again -> ...
My client's API used JWT authorization, with a token lifetime of 24 hours. But the broker wasn't aware of that or didn't care, and, instead of authenticating once a day, was trying to get a new token after every request.
My dialogue with the client:
Me: They're misusing your API. Can you contact them and explain what they are doing wrong?
Client: No. We have a prior story and they are already angry with us. If we push them, they might terminate the contract.
Me: I guess we can't even rate limit them?
Client: Absolutely no. They are not happy even with the current performance. If we make it worse, they'll just leave.
Alright, rolling with the punches. The broker is out of reach, but we can still work with our side. I'm checking out the code and diving in.
The first target is the login endpoint. Turns out, the login flow is quite complex: in addition to the usual business of password verification, the endpoint needs to check permissions against an intricate claim-based system, perform complex validations and register login in an activity log. Every action requires one or more trips to the database.
One way to reduce the number of database queries without touching too many parts of the code is caching. We had a discussion with the Ops team and I suggested simple HTTP caching on the webserver side, but they were sour about it, so I decided to go with application-level caching instead. The API is built on an up-to-date version of .NET, so I just went with IMemoryCache provided by Microsoft.
The caching logic was simple: a "cold" login request arrives and performs all the necessary DB queries. Then we put the queried data into the cache with a TTL of 60 seconds. The API allows logging in either by email or by account number. Either of them can be used both as a cache key. A very straightforward change with minimal blast radius.
Unfortunately, we couldn't just deploy the fix as is. As mentioned before, the broker was using a naive script in an infinite loop. If we'd deploy the fix, it would just push the API harder. So we're deployed the fix and introduced Nginx-based rate limiting with a generous limit. As a result, the API was able to process more requests per second than before while consuming fewer resources. All in all, the average login endpoint response time went from 2s to ~150ms, 13x faster than before.
The client was happy about that; the 3rd party was (presumably) happy, because from their point of view, the performance improved. At this point the emergency part of the engagement ended, but we kept working with the client to further improve the performance of business-related endpoints.
Emergency resolution - 1 day.
Further performance work - several weeks ond and off, as needed.
