Microsoft, Who Let the Bugs Out?
October 23, 2001
Indeed, Culp made some valid points -- e.g., many attacks succeed because existing patches have not been applied -- but he sidestepped the central question of how the vulnerabilities got there in the first place.
Culp merely states, "While the industry can and should deliver more secure products, it's unrealistic to expect that we will ever achieve perfection. All non-trivial software contains bugs."
With that in mind, it's time to back up a bit, to the release of Windows 2000, almost two years ago.
News at the time discussed an internal Microsoft e-mail describing 63,000 known defects in Windows 2000. The e-mail was intended as a wake-up call, from one of the architects of Windows 2000 to the other developers. The 63,000 consisted of three types of problems: postponed bugs, BUGBUG comments, and Prefix bugs.
Postponed bugs are actually the least worrisome. Bugs in the database resolved as "postponed" are automatically re-opened once a product ships, so people use the bug database as an electronic Post-It note, opening and then immediately postponing a bug for a future-desired feature.
Real bugs that are postponed are done so only in agreement with the tester who found the bug. Bugs with demonstrated potential for data loss or operating system instability are extremely difficult to postpone.
BUGBUG comments -- comments containing the exact string "BUGBUG" that could be grepped for later in the source code – were placeholders left by developers to mark something to be revisited later. A BUGBUG comment might have mentioned a potential improvement in an algorithm, or it might have said, "need to check the return code here" or "should handle the message being too long."
These usually weren't blatant oversights -- such as not checking that a memory allocation succeeded, as Windows is stress-tested in low memory conditions, so those bugs are found -- but instead, are more subtle situations that never occurred in normal usage.
Prefix is a sophisticated internal Microsoft tool that can simulate code execution across function calls and knows how standard Windows APIs can fail. It flags potential NULL pointer accesses, buffer overflows, and other unusual situations that the code does not handle correctly.
Because Windows 2000 was subjected to intense stress-testing in Microsoft's labs, the belief was that any code bug that hadn't manifested itself as an actual error was not important enough to fix.
With hackers actively attempting to find weaknesses in Microsoft products, this assumption is no longer valid. The e-mail recognized that problem, and was an admirable attempt to clean up the code.
That 63,000 is a large number, but only a small percentage are really bugs. As a developer on Windows 2000 when the e-mail went out, I was assigned my share, about 30 in all. On inspection, my postponed bugs had all been validly postponed, and almost all the BUGBUGs were obsolete (the change had already been made, or wasn't necessary).
A few of the Prefix bugs were real bugs, but most were the Prefix tool getting confused. Furthermore, only certain bugs will actually result in an exploitable security hole. In any event, the code was cleaned up. But the results won't be available to businesses until the server version of Windows XP ships next year.
More important is what these bugs represent: a combination of developer laziness and arrogance. Laziness to not write the code more robustly the first time, and arrogance that it wouldn't matter.
Scott Culp is not a developer, but his position reflects some of the same arrogance. The relationship between Microsoft developers and testers is a microcosm of the relationship between Microsoft and its users. Sometimes I felt myself feeling the same way Scott Culp is feeling: "Would these annoying people just stop finding bugs already?"
The problems enforcing fixes for this kind of bug are due to the inequality between the testing and development groups at Microsoft. Testers were not allowed to touch the source code when I worked there.
During the Windows 2000 development cycle, it was decided that a particular feature that had already been written would be removed from the product, because the test team did not have the resources to test it. This is fine, but the developer in question must have felt that this was an intrusion on his turf, because somewhere in the Windows source code there is a bit of code commented out, preceded by a comment that begins, "Now that testers are designing software…"
This neatly sums up the "us vs. them" attitude between developers and testers.
If a developer claims that a BUGBUG or Prefix bug is not really a bug, there is not much a tester can do, short of designing an exploit to create an actual crash. Unfortunately, testers are judged more often by the number of bugs they file than by their severity.
Microsoft is now talking more and more about security and reliability. I never heard about anyone getting a bad performance review for leaving too many BUGBUGs in their code, but maybe now things have changed and XP Server will be more robust.
But until then, every time you hear about a new exploit directed against Windows 2000 (or Internet Information Server, or any other Microsoft code that faces the Internet), you can ask yourself, "Hmmm, was there a BUGBUG comment about that one?"
Adam Barr worked at Microsoft for over 10 years before leaving in April 2000. His book about his time there, "Proudly Serving My Corporate Masters," was published in December 2000. He lives in Redmond, Washington. Adam can be reached at: firstname.lastname@example.org.