In IR reporting, precise language matters

Earlier I wrote about the Affinity Gaming v. Trustwave lawsuit over improper Incident Response (IR).  In this second installment of the lessons learned from the lawsuit filing, I’ll focus on using precise language in IR.  Precision in your choice of language matters. It matters a lot.  When you are sure about something, you need to say you are sure.  When you are not sure, say so too.

Choice of language is so important in infosec reporting that I spend a good deal of time talking about it in the SANS FOR578 Threat Intelligence course.  We focus on the proper use of estimative language to differentiate between what is known and what is believed. We also focus on the use proper estimative language.  If you’re not sure, how confident are you in a particular assessment?  We then use that knowledge to critique vendor threat intelligence reports for the good, the bad, and the downright ugly.

Putting it to work
I’m going to put some of that knowledge to use here to evaluate some of the language in the complaint.  Let’s examine this excerpt:

Trustwave also stated that it “believe[d] that the attacker became aware of the security upgrades that were taking place and took several steps to remove both the malware and evidence of the attack itself. Almost all components of the malware were deactivated and/or removed from the systems on October 16, 2013. This activity ended the breach.”

There are a few things seriously wrong with this.  First, there’s no evidence backing the assertion that the attackers “became aware of security upgrades.”  I’ll let that go for a moment since this quote was probably taken from the executive summary and the evidence could be elsewhere in the report.  On a more pragmatic note, even if attackers had become aware of security upgrades, in more than a decade of IR work I’ve never seen attackers run away from a target.  They may adapt their tactics, but I've never seen them run away from a good payday.

A second obvious problem with this excerpt is that the report says “Almost all components of the malware were deactivated and/or removed” (emphasis added).   Without needing to know any of the details, this statement is inconsistent with the statement “This activity ended the breach” which follows immediately after.  What does "almost all" even mean in this context?

Horseshoes and hand grenades
Completing a successful remediation is not like horseshoes or hand grenades – close (e.g. “almost all”) doesn’t count.  Miss even one machine, and attackers will use that foothold to reinfect other machines in the network.

Close counts with these - not IR reporting.

It probably sounds now like I’m second guessing Trustwave’s actions here.  In my earlier post, I said I wouldn’t do that, and I won’t.  This is purely a critical review of the language that was used in the portions of the report that are available as part of the legal filing.  This is all about language, not about the IR actions.

Either framepkg.exe is malware or it isn't - you can't have it both ways...
Other language inconsistencies are highlighted by Affinity’s counsel in the following excerpt.

Trustwave’s report contains inconsistencies regarding the ongoing existence of malware on Affinity Gaming’s systems. As noted above, in its “Incident Dashboard” Trustwave asserted that the malware had been removed. However, elsewhere in the Report, Trustwave states that one particular piece of malware, Framepkg.exe, was still “running on the system at the time of acquisition” which occurred on November 1 and 2, 2013.

If Framepkg.exe is indeed malware (that seems likely) and Trustwave knew this (as claimed in the lawsuit filing), then the declaration that the malware was removed is clearly not accurate and is again a symptom of imprecise use of language.

Might the failed remediation be IT’s fault?
The Framepkg.exe inconsistency might be sloppy work.  But let me offer another possible explanation.  It is doubtful that the Trustwave consultants would have been responsible for actually cleaning the FramePkg.exe malware from the impacted server.  Confucius says “cleaning malware is a dangerous game, better to rebuild infected servers than try to clean.”

Once again, Confucius never really weighed in on IR best practices.

*In fact, nobody should have “cleaned” the malware.  Once a server is compromised, rebuilding the server is the only real option.  I’ve been pretty clear on my opinions about this,  see my Shmoocon presentation with Mark Baggett titled Wipe the Drive.

Okay, so we know that one of Affinity’s system administrators should have rebuilt the server as part of the remediation.  But did they?  I’ve worked several IR cases with Rendition Infosec where recommendations were passed to IT, where they were ignored or only partially completed.  In many cases, IT reported back to management that all recommendations had been implemented completely.  Based on my experience, I submit that it is at least possible that Trustwave notified Affinity IT of the need to rebuild the server and they simply failed to do so.  If this is the case, it would change things in the lawsuit substantially.

Trust but verify
This leads to another key recommendation for IR remediation - someone independent of IT should audit to ensure all recommendations were actually completed. We recommend this every time, but some clients have misplaced confidence in their own IT and choose instead to skip this step.  This is probably rooted in the desire to complete the IR as quickly as possible, but we all know that speed and quality are often at odds with one another.

What precisely is an "inert" backdoor?!
One final note about imprecise language comes from this excerpt explaining that a “backdoor component appears to exist within the code base, but appears to be inert.”  A backdoor either exists in the code or it doesn’t.  The use of the word “appears” here is not appropriate.  The statement that the backdoor “appears to be inert” is also very misleading.  Perhaps what they really meant to say is that there is no evidence that the backdoor was used. 

Closing thoughts
Using precise language is always important when writing reports in IR (or any infosec discipline for that matter). Choosing your words carefully can make all the difference when defending your investigation.