Recently I did some major updating to a web application I wrote a few years ago. The web app was the usual LAMP stack and ran on 10-20 public kiosks.
The web app allowed an admin to run two kinds of reports on the data. The first report took a very long time to run, but the second report was much simpler and was usually quicker. After all the updates were done suddenly both reports took forever to run.
I had updated MySQL to the latest version and the database architecture had changed to support new features so I spent a lot of time re-examining how the db was set up and checked the indexes on the tables.
I then looked at the queries to see if any of them could be optimized. I added timers around queries that I thought might be slow, but everything seemed normal. Yet still the request times were abysmal.
I tried using xdebug to see if there were any processes that were being called that I was missing when reviewing the code, but everything was as expected there.
In the end I finally realized that the problem was with session. session_write_close(); was not being called in the reports scripts and so each request was locking the other out until it was done generating the report. This was a very simple thing that I overlooked.
When debugging its always best to start with the simple and move towards the complex. Checking that session was properly handled should have been one of the first things I checked. Long load times on concurrent request should’ve been a red flag, but so much had changed that I didn’t start at the beginning and instead dove into more complex areas that only muddied the issue. Its the sort of mistake that you tell yourself you’ll never do again, but somehow you end up repeating anyway.