How to fail at Incident Response

I'm a firm believer in having a sound incident response plan (and policies to go with it).  One big piece of this is having a plan with regards to how the IR team should communicate.  How should you communicate?  Well, that's going to depend on your situation.  But let me first answer the easier question: how you should not communicate.

Whatever you do, don't count on your main method of corporate communication during an incident response
There are many reasons for this, but a big one is that the main method of communication may have been compromised by the very attackers you are trying to repel.  If you continue to communicate over compromised channels, you're violating some pretty fundamental tenets of OPSEC and giving your playbook to your attackers.  Where I come from, that's a huge fail.

Why do I bring this up now?
Well, I bring this up consistently when I teach SEC504 (Incident Handling and Hacker Techniques), because I've seen organizations fail at it repeatedly.  I've also been on some "red team" exercises and benefited from these failures. One high profile failure happened over the weekend though and that's what inspired me to write this blog post.  The Syrian Electronic Army (SEA) compromised the UK sites of PayPal and Ebay.  Obviously these organizations were quick to respond.  Unfortunately, they coordinated communication (presumably a conference call) over their standard email system.  But the SEA had access to the email of one of the incident responders.  How do we know?  Well, they were daring enough to tweet about it after one of the employees commented what a bad idea it was to use their standard email to coordinate the response.

Most attackers are not this brazen and many IR teams give their playbook to the intruders every step of the way.  Unfortunately, coordinating out of band comms during an incident is not easy.  This is something that you plan before the incident occurs. 

Most corporate phone systems are VoIP now, so even trusting these phones is sketchy at best.  I recommend some out of band email through a provider like outlook.com and some burner phones to be safe.  Obviously common sense applies here.  If you deal with regulated data, understand the implications of communicating that data through a third party provider (i.e. not your internal email system).

Out of band comms - good for more than just security
Did I hear someone say DRP?  Yeah, out of band comms are good for that too.  I'm a firm believer in having a good disaster recovery plan.  How will you communicate if the email system is down because a fire took out the power panel that feeds that server rack?  Not everyone needs a secondary mailbox, but at a minimum every key member of the IR team should have one.  It sure would be good if the people coordinating the response had access to communications that were per-configured.  

And what about the VoIP system?  When that goes down, communicating to fix it can be hard.  Sure everyone has cell phones, but having a few dedicated cell numbers ready to go that are tied to a position (rather than a person) is a real win.  I've used this very successfully during large responses.  We have a few cheap burner phones with prepaid plans.  Then we distribute the numbers and say "if you have something to report about X, call this number... Y, call this number... etc."  Then you can staff a 24x7 response simply by handing off the phone.  You can set up call forwarding to private lines too, if appropriate, but there's no guessing who to call.  The key points are:

1. You aren't dependent on your internal phone system for IR.  This may not be present in a disaster or may not be trusted in a compromise.
2. Staff knows what number to call.  No more disturbing off duty personnel to see who's responsible for an item right now.  This is a real problem during active IR and I can say that adding these burner phones has increased morale every place I've worked.
3. The price is so reasonable it's within the reach of any IR team.  Worst case cost: burner phone $50, monthly prepaid service, $600/yr.  If that's not in the budget to at least get one phone for the IR lead, you need to seriously consider IR priorities (i.e. find a new job).