Expectations

A colleague was running some load tests on an application this weekend (application server based, using JBoss if you must know) and after doing about 24 hours of tests, he'd restarted JBoss for some reason. When catching up with things today I noticed that the application had been restarted yesterday afternoon and wondered why.

In the end, it boiled down to disbelief.

We typically deploy this application on a 280R (2G, 2x UltraSPARC-III 900MHz), making primary database connections to an equivalent system with more memory (8G) running Oracle (with a fiber connection to a RAID device). For development testing, we normally use an older 220R (2G, 2x UltraSPARC-II 450MHz, it was available a couple years back when we did early application testing) connected to a database which is pretty close to what we use in production. Load testing is normally done running the applications on external systems.

This round of testing is all being done on an Xserve (1G, 2x G4 1.33 Hz); the application under test, an oracle database instance, one additional application in the container, a mysql database instance (storage and queue management for JMS) and the testing client applications along with whatever adhoc tests and scripts are being run. When the results came in, they seemed unlikely (quite a bit higher than expected), resulting in a restart of the application container to make sure things were working as correctly.

When reality confounds expectations, confusion ensues.