Stray Bug Counter

Problem

The client is a QA marketplace that connects manual testers to clients who want to test their websites or mobile apps on specific devices. They came to me with a weird issue: every time a manager opened the backoffice web app, the whole platform would grind to halt for 15-20 minutes. Since the marketplace targets a specific region, the managers knew when their clients were online, so they got away with working after-hours to avoid disruptions. But this was, obviously, not sustainable.

Solution

The initial conversation revealed that the problem had emerged gradually: the marketplace had worked smoothly at launch, but had become increasingly sluggish over time. Still, since the slowdown was just a minor nuisance at first, it was never prioritized until the nuisance grew into a serious problem.

The platform had an Ops team, but no in-house developers. The original integrator was not around anymore, so there was nobody to ask about the code; I was on my own.

A quick sanity check showed no anomalies, the server appeared healthy. My next suspicion was database transaction locks, but there was nothing there either. When I opened the backoffice app, htop showed CPU usage spike - the system was actually doing something, not just waiting idly. Time to check out and inspect the platform code.

The app was built on the legacy .NET framework 4.5 and ASP.NET MVC. My first move was to check the login endpoint, but there was nothing suspicious there. The login process was lightweight. Inspecting the frontend app, I noticed that, immediately after login, the app fetches /dashboard endpoint. In the DashboardService of the backend app I found an absolutely fantastic picture.

The marketplace has a concept of "bugs": issues that testers report as a result of their work. There is a corresponding Bugs SQL table, at that time with ~60,000 entries. Not a particularly large data set by itself, shouldn't be a problem. The actual problem: the dashboard in the backoffice app showed the total number of bugs created since the service launch. They were sorted by status - open or closed. The database was MSSQL, the application used Entity Framework, and the records are queried using LINQ, an ubiquitous .NET mechanism. Now, MSSQL + Entity Framework + LINQ is a very tricky mix, because it requires strong discipline and strong mental model of what happens in the DB and what happens on the .NET side, in application memory. Apparently, the previous integrator didn't have such mental model.

The code was fetching all Bugs records into RAM, with one stray .ToList() call. Just to count them. This is disastrous on its own, but it got worse: sorting by status (open/closed) also happened on the .NET side instead of DB side. And this happened on every dashboard load, stalling the whole platform for 15-20 minutes.

At this point, the issue was clear. After some local experiments that looked promising, I, with the client's approval, commented out the offending code and deploy. The client and I then had an interesting dialog:

Client: I have no idea what you just did, but the system is blazingly fast now!

Me: That was a badly written code for counting bugs. I will go ahead and rewrite it now, so it won't stall the whole platform.

Client: Nah, we're good. We never cared about this number anyway.

Me: Then why it's there?

Client: The previous integrator added it on a whim. Guess they needed to show something on the dashboard. We decided it wouldn't hurt back then, but we never really needed this statistic. Just remove it from the backoffice app completely.

...A very typical story with custom dashboard that are way too often padded with some useless number just to show fancy widgets. The engagement thus ended without follow-ups.

All in all, it took less than one day.