Going Large with Apache 2.2.4 and Tomcat 5.0.30


This time I have been playing with Apache 2.2.4 and Tomcat 5.0.30. This is a powerful setup, which can give you a fairly enterpri$e web solution :P. Technical platforms I have been working on is Ubuntu Dapper (using Heartbeat2) and CentOS 5 (with Redhat cluster-manager, to make it even more enterpri$e :)).

The main goal is to have a redundant installation with high availability, capable of scaling, so that your java powered website will be able to meet tomorrows demand of serving.

First of all, I like to grab the source of tomcat and apache, to get the latest and greatest?!? Head over to Apache to for fill your needs. When I worked with this, tomcat 5.0 was in release branche 5.0.30-beta and httpd was in 2.2.4. This article will probably be relevant to other versions of tomcat and httpd.

My article will be based on the following setup in mind:
You have a java powered website, using a database to store data and a file system to store multimedia files etc.

I will assume that you have two backends, for redundancy of course , to serve NFS/GFS like file system and a database of your choice (Probably connected to a SAN and maybe you use DRBD?).

I will concentrate on the frontends.

Depending on your funds, I will recommend two machines in front for running the Apache httpd service, two machines in the middle running tomcat, and two machines running as backends. For even more ways to put more money into it, add two redundant cache servers up front šŸ™‚ (running varnish of course). That should make you ready for traffic!

The more simple setup, which goes along way to, is to have apache tomcat and httpd running on the same hardware. I will have this in mind when I’m writing this, and for advanced users, this should be a no-brainer to scale up.

I assume I have a NFS like file system mounted under /shared/ with webapplications under /shared/webapps/ and my bogus webapp under /shared/webapps/gamelinux/. I will not write about Redhat Cluster-Manager, nor about heartbeat, as I belive that the web has plenty of stuff about it, and google is your friend here šŸ™‚

So you have downloaded apache httpd 2.2.4 (or newer)
I compile apache like this:

./configure --prefix=/usr/local/apache-2.2.4 --enable-disk-cache --enable-mem-cache --enable-cache --enable-proxy-balancer --enable-proxy --enable-proxy-connect --enable-proxy-http --enable-rewrite --enable-expires --enable-headers --enable-http --enable-so --enable-proxy-ajp --enable-info --with-mpm=worker

I enable caching in apache, dough it is very simple, and I dont use it daily, it can give you some nice benefits and it can be a nice swiss-army-knife! (“cache everything for ever – run for an hour or wget most of you site – take down tomcat, mysql etc.” or maybe you are getting slashdoted?) For my setup I use mpm=worker, which works best (memory and crash issues) for me. The mod_proxy_balancer with ajp support is the coolest thing here. We want httpd and tomcat to talk over ajp, since it is a more efficient way of communicating than using the http protocol.

(For a more state-of-the-art, high-performance HTTP accelerator, head over to the Varnish website)

The important part of the apache config, that makes it talk nice to tomcat, is something like this:

ProxyTimeout 30
ProxyPass / balancer://prodcluster/ stickysession=JSESSIONID nofailover=On
ProxyPassReverseCookieDomain balancer://prodcluster/
ProxyPassReverseCookiePath / /

<Proxy balancer://prodcluster/>
BalancerMember ajp:// route=tomcathost1
BalancerMember ajp:// route=tomcathost2


Other things that could be nice to have in your httpd config:

ProxyPass /balancer-manager !

<Location /balancer-manager>
SetHandler balancer-manager
Order Deny,Allow
Deny from all
Allow from
DocumentRoot /shared/webapps/gamelinux

<Directory "/shared/webapps/gamelinux">
Options Indexes FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all

# Serve static content from document root with httpd
RewriteCond %{REQUEST_URI} !^(.*.jsp)$
RewriteRule ^/(.*) %{DOCUMENT_ROOT}/$1 [L]

There are alot more of details that I could give you about my setup, but for simplicity I have presented the essentials. In a large setup, you should always stress test and read documentation to find and enhance your weak points.

For tomcat, I install it also under /usr/local/tomcat-5.0.30/.
I usually write my own custom startup scripts, but I don’t want to go into details here… The important part, is the server.xml in the tomcat config directory. For my basic setup, you need the ajp-connector, jvmroute(which should be different for each tomcat server) and in-memory session replication.
I give you a brief summary of the parts here:

<Connector className="org.apache.coyote.tomcat5.CoyoteConnector"
port="8009" minProcessors="100" maxProcessors="2000"
enableLookups="false" acceptCount="1000" debug="0"
connectionTimeout="90000" useURIValidationHack="false"

<Engine name="Standalone" defaultHost="localhost" debug="0" jvmRoute="tomcathost1">

<Cluster className="org.apache.catalina.cluster.tcp.SimpleTcpCluster" managerClassName="org.apache.catalina.cluster.session.DeltaManager" expireSessionsOnShutdown="false" useDirtyFlag="true" name="gamelinux">

<Membership className="org.apache.catalina.cluster.mcast.McastService" mcastAddr="" mcastPort="45564" mcastFrequency="500" mcastDropTime="3000"/>

<Receiver className="org.apache.catalina.cluster.tcp.ReplicationListener" tcpListenAddress="auto" tcpListenPort="4001" tcpSelectorTimeout="100" tcpThreadCount="6"/>

<Sender className="org.apache.catalina.cluster.tcp.ReplicationTransmitter"replicationMode="pooled"/>

<Valve className="org.apache.catalina.cluster.tcp.ReplicationValve" filter=".*.png;.*.swf;.*.jpeg;.*.bmp;.*.gif;.*.flv;.*.js;.*.jpg;.*.htm;.*.html;.*.txt;"/>

You also need to add <distributable /> to your web.xml in your /shared/webapps/gamelinux/WEB-INF/web.xml file.

When you got tomcat and apache up and running, you should check out to see that both tomcats are in state OK. You should then check to see if JSESSIONS survives takeing down and up one tomcat server at the time. A simple code snippet to check that could be:
In /shared/webapps/gamelinux/index.jsp

<%@ page language="java" %>
<h1><font color="blue">Test page served by Tomcat5</font></h1>
<table align="centre" border="1">
<td>Session ID</td>
<td><%= session.getId() %></td>
<td>Created on</td>
<td><%= session.getCreationTime() %></td>

Hopefully you now have a state-of-the-art apache httpd and tomcat setup. You should configure RHCM or heartbeat to fail over an IP address along with httpd. I also recommend to setup database pooling in tomcat, which would make tomcat more failsafe if the backend has problems.

Here is a simple apache cache config as a bonus. You should play around with this abit, and you might get to like it…:

CacheEnable mem /
MCacheSize 4096
CacheEnable disk /
CacheRoot /var/cache/apache224/gamelinux
CacheDirLevels 5
CacheDirLength 3
CacheDefaultExpire 300
CacheIgnoreNoLastMod On
ProxyVia Full


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s