It Is The Code, Stupid

Published September 18, 2026

Those were the Days

I worked for the same company for 24 years. I was at the same place but the company changed around me: there is a distinct dividing line, a before and after, if you will. The company I started with had a boss, I worked for him, mostly. It was a time of innovation, trying new things, failing and trying again. It was a character trait to walk out to the edges like you do in a video game: new land is revealed on the map. The adventure was there in both of us. On the other side of the dividing line, the multinational came in and bought its own service provider. I became a part of a very very large family. This time the change was subtle: The company was bought because we did things much cheaper and more efficiently, we were not bound by the red tape of the giant. The idea, at the very top where fate was decided, was to use us as a template and launch into new markets, cheaply. As you can imagine, it did not happen. Conservatives, standards-happy people won, slowly. IT stood its ground: We had our own system room, our own servers, our own battalion of applications, our own developers. They worked. No one denied that. At that time, shutting us down and bringing in “the standards of the company” cost more. We stayed. I stayed and my SQL Server expertise became somehow known, among experts.

SQL Server, My Happy Place

I first touched version 6.0 in 1998. I deployed 6.5 very briefly and quickly upgraded to 7.0 when I joined the company. I retired and most of the SQL Servers were, if I remember correctly, SQL Server 2017, with newer versions being tested. I did everything that could be done with SQL Server and then some. I am not talking about just building queries, or writing T-SQL stored procedures. I designed schemas, implemented backup and archiving policies, I ran a system room with 22 SQL Server instances, connected to (at its height) 250+ local SQL Server instances by native replication. Been there, done most.

After the company that had been was acquired by the multinational, people came to Turkey from all over Europe, to help us get “oriented”. I made friends with many wonderful people. They helped a lot. Most of them understood the dilemma, even suggested workarounds. I am eternally grateful to them. Seeing what we had done, what I had done, also brought an understanding of what we were capable of. So, it was not a surprise when I was asked to “have a look” at a SQL Server problem in Russia. I didn’t go there, it was all remote. It was 2010, the international climate was not like today.

Servers, Service Packs and Security

All below is from my notes in a notebook.

I will be as brief as I can:

The Russia brokerage had a Windows 2000 Advanced Server SP4, running SQL Server 2000 Std SP3a (in 2010, it happens: don’t fix if it is not broken, until it is). An important report is pulled every day, many times by the users, used in customs declarations. It completed in 35 seconds on this server.

Microsoft released SP4 for SQL Server 2000. The Security department said: Go and install it, it is required for Security. This is a corporate setting, all kinds of tests to run before anyone touches a critical system. They did their homework. They sourced a new server, kept the original for the users, so customs could continue to work while the SP4 was tested.

The new server was, well, new. It had 1 CPU with two cores, the old one was 2 CPUs, 1 core, but with a slower system clock. Old had 1.5GB memory, new 6GB. Everyone was thinking more memory would be so good for SQL Server, so while the goal was SP4, this was two birds, one stone. All numbers below are on the new server from now on. Your anchor is 35 seconds on old HW, don’t forget, and the three numbers in parentheses: installed memory, what the OS can see, what SQL Server is allowed to take.

Windows 2003 Std SP2, still SQL Server 2000 Std SP3 (6GB/4GB/2GB), the report finished in 102 seconds. No one was alarmed, yet. Establishing the baseline for the new server. Just in case, they pulled Enterprise Licenses from the company pool, both for OS and SQL Server. Requests, approvals, the whole IT management chain was involved. (They did not change the SQL Server memory allocation yet)

Windows 2003 Ent, still SQL Server 2000 Std SP3 (6GB/6GB/2GB), the report finished in 15 seconds. Modern Ent OS, better support for memory handed down to SQL.

The moment of truth: Windows 2003 Ent, SQL Server 2000 Std SP4 (6GB/6GB/2GB), the report finished in 3673 seconds.

Teams of experts scrambled, as you can imagine. An expert thought of changing MAXDOP. Maximum Degree of Parallelism. In short, the number of concurrent threads the DB engine is allowed to use when serving the query, it was 2, they changed it to 1.

Windows 2003 Ent, SQL Server 2000 Std SP4 (6GB/6GB/2GB), MAXDOP=1, the report finished in 1669 seconds. More experts, more combinations. Windows 2003 Ent, SQL Server 2000 Ent SP4 (6GB/6GB/4.5GB), the report finished in 7071 seconds. Windows 2003 Ent, SQL Server 2000 Ent SP4 (6GB/6GB/4.5GB), MAXDOP=1, the report finished in 1674 seconds.

By that time, Russia was pushing for an “Exception”: A special permission from Security to stay at SP3. Security was not taking the chance (they would be liable) and pushing back. Country management was in open revolt. Somewhere in that same week, a bigger server was being priced, something like three and a half thousand dollars, to be shipped straight to site.

It is the Code, stupid.

I was engaged, began asking about scripts of the offending queries that were executed for the report, table and index schema, number of users, table and DB sizes. At one point they granted me remote access, not common practice, out of desperation, probably. I was in contact with Russia, Europe IT (the people that helped Turkey were now looking at me for help and I was trying to return the favor), even people from the US, who were responsible for ALL the SQL Servers in the company, worldwide. They were meeting me for the first time, my only reference from Europe IT, “this is a good guy, he is running a country IT on his own, knows SQL in and out, trust him but obviously, watch over his shoulder.” I was very careful, trying not to offend anyone by saying “it is the code, stupid” but slowly gaining ground towards this idea, nudging them.

Because I already said in one of my emails to the person I was talking to in the USA:

“In my experience, anything hardware related will effect performance maybe up to an order of magnitude, when we are talking about rack-mounted, mid-range servers. Even then I am putting a lot of slack when I say an order of magnitude, 50-200% is a more probable range. As a developer and DBA, it was always my fault or my design short-comings that was the deciding factor on performance. Let’s not forget: If we were dealing with app/query performance on one server, then I would focus on code itself. But here we have a server with perfectly acceptable perf., and one with ridiculous figures, both with (I hope) same code on them so I am trying to keep focus on difference, both hardware and conf.”

I was hinting at the code, was trying to be political saying I was also exploring hardware and configuration.

Meanwhile I worked the code. I found out the query was calling a function twice, dead code that was not adding anything to the result, flattened the function and some views to tables and inline code in the stored procedure. All the tricks I had in my bag to reach streamlined, simple code. Drawbacks: The person I was emailing with in Russia, with a “meh” command of English, told me it was the code of a former developer, long gone. In fact there were moments he asked me what the code was doing. He was maintaining a whole DB and that part, he simply was not familiar with. No documentation, all DB artifacts named in Russian. Deciphering tables became a dictionary-assisted job.

But… I did what I did best (fire stations and the only fun part) and dropped the bomb on them. These figures below were my version of the stored procedure (they later called it “Enh”, enhanced version.) The output was byte-identical, by the way.

Windows 2003 Std SP2, SQL Server 2000 Std SP4, 27 seconds.
Windows 2003 Ent SP2, SQL Server 2000 Std SP3, 30 seconds.
Windows 2003 Std SP2, SQL Server 2000 Std SP4, MAXDOP=1, 32 seconds.
Windows 2003 Std SP2, SQL Server 2000 Ent SP4 (6GB/6GB/4.5GB), 32 seconds.
Windows 2003 Std SP2, SQL Server 2000 Ent SP4 (6GB/6GB/4.5GB), MAXDOP=1, 32 seconds.

I was a real genius when naming my own code: _burak4

But also note: Their best on SP3 was 15 seconds, my best with my enhanced stored procedure on SP3 was 30 seconds. I made it worse, if they were able to stay on SP3, under exception. Clearly, what I did was something very specific to how the DB engine changed from SP3 to SP4. Can I pinpoint what that was, no, SQL Server change logs lacked that detail. Did it work, it certainly did.

It WAS the code, I think.

They had been about to bring Microsoft in. Instead the man who answered for every one of those servers in the company wrote to ask what steps I would like to follow. Please let me know, he said, and we will get moving.

They never bought the server. They never called Microsoft either.