The problem

Jira has a very useful mail integration system that allows issues to be created and commented on via email. While this works well for the simple case there is a lot more information in the mail that we receive from both automated support requests and direct customer conversations that we could feed into Jira issues. Rather than hard-code the extraction of this information into Jira itself my project was to produce general configurable, extensible system for processing incoming mail, allowing us to classify messages based on key phrases and tags in the mail text.
Read on for the solution …

The technology

While a number tools for processing mail exist (most famously Procmail which implements its own DSL) there is already a standard language for implementing mail filters called Sieve. As a language Sieve is (deliberately) limited; by being purely declarative and having no loops or variables it prevents rogue scripts from tying up a mail server. However it is also extensible, which makes it interesting for this project.
There is a Java implementation of Sieve as part of Apache James project, however it appears to be only intermittently maintained and doesn’t support any of the useful proposed extensions to the language, such as the Body extension. But an initial spike test application did show that it was cleanly architected and should be extensible enough for what I wanted to do.

Making JSieve useful to Jira

While extracting information from headers such as subject would be useful the real power of a mail processing system would be to classify messages based on key phrases and tags in the mail text. The ultimate target is to be able to specify a script that classifies mail for a project, such as the following:

As mentioned JSieve doesn’t support searching the body of messages, so my first task was to implement a basic subset of the proposed ‘body’ match test. This is simply a case of subclassing JSieve’s AbstractTest and implementing the parameter validation and the actual test. The matching is currently done via a simple substring match on the mail body.

Integrating with Jira

What happens to incoming mails after they are picked up by Jira’s mail-service is defined by a mail-handler, which is configured by the administrator; current ones include the ability to create or comment on issues. Consequently I created a new mail handler ‘SieveHandler’. This takes the normal parameters of a handler plus a script (‘sievefile’).

As we want to have a sane fallback in the case of an invalid or missing user script the SieveFile class inherits from the ‘Create or Comment’ handler, which is the most common case, and falls back to that behaviour if there are any problems.

Taking action

The Sieve specification defines a number of basic actions such as ‘reject’ and ‘fileinto’. However for Jira we need to define our own as the target system is much richer than a mail filesystem. For the purposes of ShipIt I’ve implemented two; ‘setcomponent’ and ‘setpriority’.
Implementing a custom action consists of subclassing JSieve’s AbstractCommand an creating the validation and execution code. What execution does is push a corresponding ‘custom Action’ class onto the stack for that mail. Once the sieve script has completed these actions are executed in turn. In our case the actions take the parameter they were give during the mail processing (priorities and components) and set call the appropriate actions within Jira to set the values for the issue.

Future work

Obviously the next step is to add more actions to those available to allow script writer a larger number of parameters to modify from mail messages. Additionally there are some proposed extensions to sieve that would give scripts much more power. One option is to complete the body extension to allow matching on attachment types and even content. The other is a regular-expression extension that would allow much more complex match clauses, plus the contents of matches can be substituted into the paramaters to actions. Another useful addition would be to store the sieve scripts in the Jira database and allow editing through the admin interface.

ShipIt V – Sieve mail processing for Jira...