The surprising truth about enterprise ChatOps

There’s this rumor going around that ChatOps was the invention of start-ups. I don’t know about you, but I can think of several enterprise IT/Ops teams who started using chat to coordinate and troubleshoot over a decade ago. (Ok, back then we called it “instant messenger”. But still.) So I hope I don’t spoil the rest of this post by letting you in on a little secret: if you’re in enterprise IT, you’re probably doing some ChatOps already.

Still with me? Good. Because chances are, there are even more elements of ChatOps you can incorporate into your existing practices, making for a less stressful workday. After all, leaving the office happy is the biggest win ever.

As my teammate, Sean Regan, put it in his post introducing ChatOps to the Atlassian community:

“ChatOps is a collaboration model that connects people, tools, process, and automation into a transparent workflow. This flow connects the work needed, the work happening, and the work done in a persistent location staffed by the people, bots, and related tools. The transparency tightens the feedback loop, improves information sharing, and enhances team collaboration.”

People, tools, transparency, automation… sounds a lot like DevOps, right? That’s no accident. Adopting ChatOps practices is a great way to support DevOps at scale. Small changes such as piping notifications from multiple tools and systems into a single chat app for all to see are a super-effective way to build muscle around culture, automation, and shared responsibility. I’ll point out a few more ways ChatOps strengthens your DevOps-at-scale game as we go along.

So without further ado, the Hipchat Data Center team and I bring you five surprising ways your enterprise team can win big with ChatOps. (Drumroll please!)

1. Kill your phone bridge and use ChatOps for incident management

If you’re using phone bridges to troubleshoot incidents and roll out a fix, listen up. And if those phone bridges involve lots of people distributed across lots of different locations, listen good.

A simple way to try a bit of ChatOps is to put down your phones and create a dedicated chat room next time an outage strikes. You’ll round up the same team of fixers, but this time you’ll have a rolling, written record of the conversation to refer back to as you’re troubleshooting, plus an easier way to share files, runbooks, and tickets with each other. The more robust enterprise chat tools (*cough* Hipchat Data Center *cough*) let you fire up voice and video calls right from the app as well, so you can talk live for a few minutes when you need to.

That chat history is clutch, by the way. When you’re in the middle of a prolonged outage, and it’s midnight, and you’re passing things off to the team across an ocean (because your eyes are bleary and your brain is mush), you don’t have to worry about writing up a hand-off summary or forgetting to mention key details.

Along with short-lived per-incident rooms, you can dedicate a permanent room that pulls notifications from various monitoring systems into one place for your site reliability engineering (SRE) team. During an incident, it’s common to fine-tune the settings to filter out notifications that aren’t related. The right data to the right people at the right time. Improved signal-to-noise ratio. (Insert your favorite buzzword phrase here – just make sure to use it when pitching the idea to your boss. (wink) )

2. Deliver amazing service beyond your ticketing system

If you’re on a team that provides service to internal customers (Workplace Support, etc.), there are at least 3 ChatOps techniques you can adopt at scale.

Open a chat room for micro-requests – How many of your service requests could be resolved simply by pointing someone to a piece of documentation, or providing a 5-second answer? Quite a few, I’d bet. A chat room open to all employees is an easy way to service these requests faster than you can say “SLA”. When Atlassian rolled out a new VPN client, our IT team created a room for people to ask questions and get help. That room deflected a massive number of tickets and helped the team complete the rollout ahead of schedule.

Pipe request notifications into chat – Any respectable enterprise chat tool will integrate with your ticketing system. So have that system ping your team’s room when a new request comes in. Optionally, you can tune the notifications so only the most urgent requests are pulled in. And because you’re already in chat, you can discuss who will field it and share relevant info with your teammates right then and there.

Use per-ticket rooms – A favorite ChatOps practice among service teams at Atlassian is to open a temporary room for medium- and heavyweight requests. In cases where the requestor needs to provide more info, a chat room facilitates this much faster and more intuitively than lobbing comments at each other on a ticket all day long. Not to mention being able to fire up a video call right from the chat room and use screen sharing to diagnose problems, even if you’re not in the same building.

3. Make change management manageable with ChatOps

Companies of all sizes need consistent change management practices to plan for, control, and reduce risk around changes to production systems. If you have to comply with governance standards (PCI, HIPAA, etc) the stakes are higher, and the process is extra-heavy.

I’m amazed by how many teams track changes via email threads, and keep documentation on some little-known network drive that few people know how to access. Information silos make collaboration even harder, and the struggle is real enough as it is. But two ChatOps practices can really take the edge off.

First, invite all the key players to join a room (or rooms) dedicated to change management. You’ll be able to interact quickly during the approval process, as well as link to relevant tickets, share supporting docs, etc. This typically eliminates the need for daily meetings (woot!) and speeds the whole process up. Plus, your chat room holds a clear record of all the discussions, which comes in handy for troubleshooting, root cause analysis, and any compliance-flavored record-keeping.

Next, sprinkle in some automation. Use your chat’s API to pull in notifications from your build and deploy system. And if you really want to be on top of your change management game, hook into your ticketing system so pending items are surfaced in the room before the change is imminent.

Btw, if your current change management practices involve a lot of “throw it over the wall”, bringing everyone together via chat is a good way to build DevOps muscle.

ChatOps makes change management more... manageable.

4. Let the robots make you look good

In case the message hasn’t come through already… automation, automation, automation. Making the robots do repetitive tasks so the humans can focus on higher-value work is essential for operating effectively at scale – and a major component of ChatOps practices. (It’s also the “A” in DevOps’ famous C.A.M.S. framework. Just sayin’.)

Chat-dwelling robots, “bots”, are a step beyond integrating your chat tool with ticketing and monitoring systems. Bots can actually carry out operations on your behalf. They save you time, eliminate human error, and leave a tidy little paper trail.

For example the team over at SumoLogic built a bot that pipes information about Jira issues to their change management chat room whenever someone in the room links to a ticket – similar to how the Jira integration for Hipchat Data Center works. The bot also acts as a sort of personal assistant, reminding them of pending changes they’ll need to take action on soon.

Bots can help you translate incoming support requests into your local language, greet people who join the room after hours and let them know when they can expect a personal response, or do just about anything else that can be written up as a script.

5. Pair ChatOps with common-sense security

Like change management, security and privacy are important no matter how big your company is, or what business you’re in. Any enterprise-grade chat tool will have strong user management. Most (including Hipchat Data Center) integrate with 3rd-party user directories and password management systems, as well. So you’re covered in the external-facing direction. But what about internal-facing?

ChatOps doesn't mean sacrificing securityThat’s where privacy settings come in. You can make a room private and invite only a select group to access it – perfect for rooms that integrate with systems that contain sensitive information like deployment or monitoring tools. Also, consider setting permissions around who can set up integrations and install bots. When sensitive data may be involved, you might want to make that an admin-only capability.

Of course, most chat rooms can be open to anyone in the company. And if you’re adopting the ChatOps practice of dedicated rooms for support and every-day Dev+Ops matters, open rooms are essential.

Choose a path to victory

The real surprise about enterprise ChatOps is that, in many ways, it’s not all that different from what you’re doing now. I bet you recognized at least one of your current practices on this list. Or something really close to it. Build on that! Steady evolution is the name of the game, and (let’s face it) just the reality of working at enterprise-scale. A year from now, you’ll look back and see that ChatOps has quietly revolutionized the way your team works.

.  .  .

Need an on-premise chat tool that’s built for massive teams? Check out Hipchat Data Center and start winning with ChatOps.

Take a tour

Hipchat Data Center presents: 5 surprising ways en...