Benefits of Using IRC or Group Chat & Video Conference During Incident Management

When a team of engineers is dealing with a real-time incident, such as a system outage, troubleshooting a problem or dealing with a malicious hacking attack, having excellent communications is critically important. The appropriate communications tool can make a world of difference in dealing with the issue and learning from it afterwards. As important as the engineering work itself is, lack of good communications is what often gets tech teams in trouble.

You should enable real-time communication in certain collaborative tasks. This will reduce unnecessary email traffic and clutter, enable people to to focus better on their tasks,and minimize time wasted in bringing each other up to speed When multiple people are working together in real-time on a near term collaborative task, such as:

• Crisis Management
• Troubleshooting
• Dealing with hacking attacks
• Build and deployment
• Web application migration
• Upgrade or maintenance
• QA testing

Many companies use a phone conference and/or email to assist in real-time while the collaborative activity above is ongoing. Since Email is not instantaneous and real-time the way a group chat application is, and since email is not a suitable medium for quick questions, and quick one-line responses, smart teams use a real-time group chat tool like IRC (Internet Relay Chat) to enable and facilitate real-time conversation. Benefits of using IRC or a real-time textual group chat tool instead of email are:

• Tech managers, project managers, crisis managers and new tech people joining the effort can quickly catch up with what has been going on (in any level of detail they want) by reading the IRC history transcript so far. This is a much faster and efficient way than using email or pulling someone away to talk in person asking what has been going on. (If email were to be used instead of IRC, a new person joining in would have missed the previous emails on the topic.)
• When an engineer working on such a collaborative task steps away for a while and comes back, they can quickly catch up on what transpired while they were away by reading the IRC history transcript.
• Email is not cluttered by short back and forth messages with lots of text to read and filter
• The IRC transcript can be used for the post incident retrospective and report (“post-mortem”).
• Unlike a phone-only conference, the IRC transcript can be read and analyzed to learn lessons from this incident. For example:
  • Analyze what problems the team ran into
  • Analyze what worked and what didn’t
  • Analyze how well people collaborated and communicated
  • Timelines of events

I can personally attest to the above benefits. Over the past 15+ years, my development and operations teams in different companies have regularly used IRC to great advantage. Tools like Wikis and blogs are great for collaboration, documentation and sharing information on projects. An group chat like IRC is an indispensable tool for real-time collaboration.

2013 August Update:

With multi-participant video conferencing becoming commonplace thanks to Google Hangouts, I have updated this post to include video conferencing combined with group text chat.

The rest of this update has moved to its own blog entry titled ‘What I Learned During the Hacking Attacks of August 28, 2013.’

2014 July Update:

I now recommend organizations to consider using Slack. See my 2014 July comment below.