Optimization Under Constant Load Increase

I’m sure there’s been a time when you used software (website or application) which made you want to smash your computer. You can easily remember how annoying it became waiting for query results or for a response to your actions. At best, you just left the site or closed the application and forgot about it. At worst, you had to endure this and go on using it, while berating its developers.

The question of system performance worries users as well as the person responsible for business software. Clients don’t want to tolerate the poor performance of your software.

The two most significant reasons for decreasing performance are internal software issues and increasing system loads. Now I’d like to focus on the latter. Increasing system loads may be caused by a rising number of users and performed operations. This fact can be both good and bad. It’s good because it shows that you have more customers, your business is expanding and is successful. However, this good circumstance can cause bad consequences. If you don’t solve the performance problem, your customers may appear unsatisfied with your software and your business accordingly. Such reputation risks can cause you to lose customers and direct and indirect financial losses. It is the most likely and most negative effect of a system’s poor performance.

So, what can we do to avoid this?

First, prevention is better than cure. While creating a software system, we always provide the possibility for vertical and horizontal scaling. A service-oriented architecture (SOA) is very useful for these cases. Vertical scalability means adding resources (CPU power, memory capacity or disk space). The system can be developed so that when more hardware resources are needed, some configuration settings can be changed without buying new equipment. Horizontal scalability means adding more nodes, which increases performance and makes the system highly available. It is useful to have a reserve instance which can be run in case of a sudden rise of system loads or any other problem that can be solved this way.

When developing the system, in addition to scalability, we added instrumentation to enable us to observe changes in loads and take the required steps before the problem arises.

Still, if an application has already been developed without considering the possible load increase or the loads exceeded the expected ranges, then we take another route: we optimize the application. The method of optimization depends on the level of performance loss. For instance, if the application continues working slower, it may be enough to change some settings (i.e. DB connection pool size, thread pool size for HTTP connector, heap size etc.) and the problem will be solved. However, this step is definitely not sufficient if performance loss is more critical, like as follows:

1. The application works so slowly that users can’t receive valid data. Still, every user is served.
2. The application can’t serve all requests, though most users are still served albeit slowly.
3. All system resources are used, the system hangs, no client can be served. This situation is called denial of service.

In all these cases system optimization may include one or more of the following actions:

• Reengineering the system: gather metrics, identify bottlenecks, update some components or the whole system to remove the bottlenecks. It may be database optimization, caching, replacing technologies, revising system architecture, etc. I wrote about this in detail in this article, but here, I want to mention that reengineering is expensive, however, its results may improve the current situation and also prevent such situations in the future.
  
• Adding hardware resources or replacing them with new, better features (vertical scaling). New resources allow the system to work quicker. As a rule, such optimization does not require changes in software code.
• Clustering the system, adding instances (horizontal scaling). Obtained aggregate power can significantly increase the system performance. Generally, this way requires fewer expenses and shows better results, though it demands that changes be made in the software system.

There is no definite decision on the problem of increasing system loads and no unique advice about what is the best way to optimize the system. Everything depends on the particular system, the severity of the problem and the expected results.