IsoveraDL Performance Testing - Summary and Recommendations

Problem

Isovera conducted testing to determine the minimum server hardware required to run several IsoveraDL instances concurrently. We also desired to gather general data on IsoveraDL performance to guide future performance optimization effort and to provide BEN Collections with expectations of performance and system load.

Solution

Implement a test hosting platform to conduct user performance testing of the common IsoveraDL functionalities and determine an acceptable recommendation using the reported data.

Recommendation

The IsoveraDL SaaS implementation will require a hardware platform roughly equivalent to the large instance (2 dual core 1.1 GHz processors, 7.5 GB of memory), with processor capacity a more important factor than memory. This platform will be sufficient to run up to 10 IsoveraDL instances of varying size (a few hundred to several thousand records).

Procedure

  1. Identify a test platform: Amazon Elastic Compute Cloud (EC2) is an emerging technology platform allowing cost effective on-the-fly hosting solutions using virtual machine technology. We set up two hosting instances; one equal to a typical entry-level web server and another equivalent to a more high power machine.
    1. Small Instance: equivalent to 1.1 GHz 2007 Opteron or 2007 Xeon processor, 1.7 GB of memory, 160 GB of instance storage, 32-bit
    2. Large Instance: equivalent to 2 dual core 1.1 GHz 2007 Opteron or 2007 Xeon processors, 7.5 GB of memory, 850 GB of instance storage, 64-bit
  2. Set up test environment: We installed 10 instances of IsoveraDL version 2.1 using sample databases of metadata records derived from existing BEN Collections and ranging in size from large to small on both the low powered and high powered hosting machines.
  3. Develop test plan: We wrote seven user test cases covering the most common IsoveraDL functionalities users and administrators will execute. We decided on key criteria for the tests:
    1. Page load time would be the primary metric for our tests.
    2. A minimum acceptable threshold for page load time for each test was established
    3. Multiple concurrent users (up to 5) would conduct tests simultaneously to simulate real-world expected system loads. Conducting the tests with 5 simultaneous users equates to 2,880 page hits a day.
    4. System resources (CPU & memory) would be monitored and logged during the test
  4. Conduct test cases: 5 users gathered in a conference room. Users conducted each test case simultaneously and were instructed to record the amount of time each page took to load in seconds. Of the 5 users, two used Internet Explorer and three used Mozilla Firefox, and the users accessed a mix of small and large IsoveraDL databases. Page load time values were recorded for each test case, and notes were made of any system failures or errors. The tests were repeated for both lower power and high power hosting machines.

Detailed Results and Recommendations

Results showed that the high powered system instance was able to meet or at least approach the minimum acceptable threshold for page load times for most tests (see graph below). A few tests, particularly the test for editing controlled vocabularies, had very high load times. However, this is related to the design of the system functionality rather than a hardware limitation and will be optimized in a future IsoveraDL release. Since these tests were for less commonly used administrative functions their performance, while important, is not as critical as publicly accessible functions such as search, browse, and submit.

Results for the low-powered instance were not as encouraging. Several tests resulted in page load times of 10 seconds or greater and 100% CPU usage, indicating that the machine was overloaded. However, when the tests were repeated on the low-powered instance with 1-2 simultaneous users, page load times were quite snappy and within the minimum acceptable level. The low-powered configuration (1.1 GHz, 1.7 GB of memory) therefore may be adequate for individual BEN collections who install IsoveraDL locally and do not have high web traffic.

The IsoveraDL SaaS implementation will require a hardware platform roughly equivalent to the large instance (2 dual core 1.1 GHz processors, 7.5 GB of memory), with processor capacity a more important factor than memory. This platform will be sufficient to run up to 10 IsoveraDL instances of varying size (a few hundred to several thousand records).

A few other results were noted during testing. A small number of tests resulted in browser errors or page not found responses. However we suspect these were due to browser or network issues, and in most cases the system even when slow to respond proved quite stable. On average users using the Firefox browser reported comparable response times to Internet Explorer users, though for certain tests Firefox was slightly faster.

Future Performance Optimizations

Based on results of the testing, the IsoveraDL developers have identified a few key configuration and/or coding changes that will be applied to IsoveraDL instances over the coming months and are expected to result in performance gains:

  • Increase maximum allowed database connections
  • Increase use of caching for commonly accessed pages
  • Optimize SQL query structure in IsoveraDL codebase
  • Revise and optimize code for managing controlled vocabularies