Currently the preferred way to enable SSL or port 80 for a Java application is to install Apache as a reverse proxy. But using mod_proxy or mod_jk you can use standard SSL certificate formats and use it to securely open port 80 and map to the application.

However using Apache is slightly overkill when we’re using it as a glorified port-redirector. Apache is not just a HTTP server but an application server, and its design reflects this. One problem we’ve been having recently is that in periods of high load Apache responds faster than the application it is fronting, and a large number of worker processes will be created, loading the CPUs and memory even more. Once Apache hits its maximum process limit all applications will become unresponsive, regardless of which one is loaded. While it is possible to adjust the timeouts and number of processes I’ve been looking at the possibility of using other HTTP servers; read on for one option …

Recently I’ve been looking into a more lightweight server that has the functionality we need without the additional overhead. One that has been getting some traction recently is lighttpd, and it is being used to run some very highly loaded bittorrent sites. It was built to address the C10K problem so should be far more scalable. It supports the few features we need in a front-end (proxying, SSL, simple redirects), and runs in a single-threaded mode, which should make better use of processor affinity and give the applications a bit of breathing room on any additional processor caches. To give it some testing I’ve installed it on one of our internal-facing applications.

We are now running most of our applications in an SSL-only mode, with non-SSL requests being redirected to the SSL equivalent. The steps to get this configured are:

  1. Install lighttpd, ensuring SSL is enabled. As it is included in most recent Linux distributions this is simple enough.
  2. By default only a few core modules are enabled. Enable ‘mod_proxy’ and ‘mod_redirect’ by uncommenting them in the config.
  3. For older (pre-1.4) versions you will need to set the default port (‘server.port’) to something other than 80.
  4. Setup the default port 80 virtual host. We use IP addresses for this as name-based virtual hosting doesn’t work with SSL. All this virtual host does is redirect requests to the SSL host:
  5. $SERVER["socket"] == "63.246.22.51:80" {
    url.redirect = (
    "^/(.*)" => "https://jira.bamboo.atlassian.com/$1"
    )
    }
    
  6. Next we setup our SSL host. You need a PEM-formatted certificate; if you have a .crt and .key file you can just join them together. You may also need the CA certificate. Our application server is running on 127.0.0.100:8080, so we use proxying to just forward the requests:
  7. $SERVER["socket"] == "63.246.22.51:443" {
    ssl.engine = "enable"
    ssl.pemfile = "/var/www/domains/bamboo.atlassian.com/jira/ssl/star.atlassian.com.pem"
    ssl.ca-file = "/var/www/domains/bamboo.atlassian.com/jira/ssl/DigiCertCA.crt"
    proxy.server  = ( "" => ( ( "host" => "127.0.0.100", "port" => 8080 ) ) )
    }
    

Normally that’s all there is to it. However, during restarts of the application server lighttpd will show a default 503 (service unavailble) error page. As this is a normal maintenance event we want something a bit more informative. This is simple enought; just create a directory to hold the custom error page (I put it in /etc/lighttpd/error-html) and add a page called ‘503.html’ with the custom message. Then add the line:

server.errorfile-prefix = "/etc/lighttpd/error-html/"

to the configuration file and the server will pick it up.

To give an idea of the difference, here’s the processes of a normal, low loaded Apache front-end:

USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root      1665  0.0  0.2 18476 10356 ?       Ss   02:14   0:00 /usr/sbin/httpd
apache   16578  0.0  0.1 18744 7028 ?        S    04:02   0:02 /usr/sbin/httpd
apache   16579  0.0  0.1 18752 7024 ?        S    04:02   0:02 /usr/sbin/httpd
apache   16580  0.0  0.1 18752 7024 ?        S    04:02   0:02 /usr/sbin/httpd
apache   16581  0.0  0.1 18752 7024 ?        S    04:02   0:02 /usr/sbin/httpd
apache   16582  0.0  0.1 18752 7024 ?        S    04:02   0:02 /usr/sbin/httpd
apache   16583  0.0  0.1 18752 7024 ?        S    04:02   0:02 /usr/sbin/httpd
apache   16584  0.0  0.1 18752 7024 ?        S    04:02   0:02 /usr/sbin/httpd
apache   16585  0.0  0.1 18752 7024 ?        S    04:02   0:03 /usr/sbin/httpd
apache   17758  0.0  0.1 18752 7024 ?        S    05:26   0:02 /usr/sbin/httpd
apache   19103  0.0  0.1 18752 7024 ?        S    06:59   0:02 /usr/sbin/httpd
apache   21160  0.0  0.1 18752 7008 ?        S    19:07   0:00 /usr/sbin/httpd
apache   21161  0.0  0.1 18752 7016 ?        S    19:07   0:00 /usr/sbin/httpd
apache   21162  0.0  0.1 18752 7008 ?        S    19:07   0:00 /usr/sbin/httpd
apache   21163  0.0  0.1 18740 6984 ?        S    19:07   0:00 /usr/sbin/httpd

At peak load the number of these processes can reach over 500.

Here’s the corresponding lighttpd process:

USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
lighttpd 29038  0.0  0.0  5576 2032 ?        S    May01   0:00 /usr/sbin/lighttpd

I’m going to continue running it for a while, and possibly move it onto our main Jira and Confluence hosts once we’re satisfied there are no issues.

One place this may cause issue is with Bamboo and triggers. For security reasons triggers may only originate from the IP of the host that the source repository is on, but when it’s behind a proxy the IP always appears to be the local host. One option to fix this is to point the trigger at the application-server’s HTTP port (normally 8080). However, if you’re feeling brave you can try downloading and building the beta release of lighttpd. This contains support for AJP as a proxy backend (as used by Apache’s mod-jk), which preserves information about the incoming request that is lost in the proxy requests.

Fresh ideas, announcements, and inspiration for your team, delivered weekly.

Subscribe now