Production/POC/Benchmark check list

For whoever running benchmarks or POC or about to move into production with GigaSpaces:
Here are set of tuning activities to be done prior moving into production or when running your POC or benchmarks. These are listed according their priority.
Most of the following are critical for environments with large amount of spaces/clients.

See below also additional info regarding Solaris OS you might find interesting.

Memory planning
HW capacity planning should be done in advance to provide servers with sufficient free memory to avoid swapping at any point in time. With no sufficient free RAM virtual memory starts swapping memory pages to the disk decreasing overall performance 20 to 50 times. It should be strictly avoided.

Command to check free memory in Linux:
$ free

Used swap should be zero, while free memory should be sufficient to host all GigaSpaces and third party components planned for deployment. With all SW components loaded after reboot, free memory should be at least 300MB. Having free memory below 50MB is critically low with the danger of OS or application crash.

Ulimit and file descriptors tuning
This is critical when running multiple GSC on the same machine or when having multi threaded clients.

Set in /etc/security/limits.conf (root permissions required):

  • soft nofile 16384
  • hard nofile 65536
  • soft nproc 2047
  • hard nproc 32768

Reboot the machine after changing this file to allow the changes to take affect.
Nofile is relevant for amount of connection/sockets and nproc is relevant for amount of threads.
In general each application thread opens a connection/sockets to the space. Every sync replication using a dedicated connection/socket.

GSC JVM Tuning
Since the applications using GigaSpaces leveraging in most cases very fast CPUs, the amount of temp objects created is relatively large for the GC to handle with its default settings. This means careful tuning of the JVM is very important. In extreme cases we would even suggest to use Java real time that is slower than the regular one but much more deterministic with its GC activity.

See below example of settings we will recommend for applications that have lot of temp objects created and cannot afford long pauses - these are good for cases where the business logic and the space are collocated (aka embedded space):

-Xms2g -Xmx2g -XX:UseConcMarkSweepGC -XX:CMSIncrementalMode -XX:CMSIncrementalPacing -XX:CMSIncrementalDutyCycleMin=10 -XX:CMSIncrementalDutyCycle=50 -XX:ParallelGCThreads=8 -XX:UseParNewGC -Xmn150m -XX:MaxGCPauseMillis=2000 -XX:GCTimeRatio=10 -XX:+DisableExplicitGC

See more info here:
http://www.gigaspaces.com/wiki/display/OLH/MemoryManagement-SpaceSchema

Please note that -XX:+UseConcMarkSweepGC has the heaviest impact on performance - decrease of 40%.

The following set of parameters shows 20% better performance than with -XX:+UseConcMarkSweepGC while the pause size still is below 100msec in embedded test with payload 10KB and 100 threads:
-Xms2g -Xmx2g -Xmn150m -XX:GCTimeRatio=2 -XX:ParallelGCThreads=8 -XX:UseParNewGC -XX:MaxGCPauseMillis=2000 -XX:DisableExplicitGC

GSM/LUS JVM Tuning
The GSM/LUS are essential components within the GigaSpaces. The LUS is GigaSpaces directory service. The GSM is the deployment and service management service. Each of these consume some resources and have some periodic events going on that are essentially some kind of heart bit mechanism that must happen correctly to identify the health of the system.
The correct way to deploy these would be to separate these in 2 processes and run these on dedicated machines with plenty of memory (1G min) and careful tuning of the JVM GC to reduce the young generation size that will minimize GC pauses and avoid stop the world events that could lead to split brain scenario, no ability to access the spaces and general system instability.

When there is no dedicated machine to run the LUS / GSM , we recommend having the LUS/GSM on machines with less GSCs in order not to saturate the machines CPU that will lead to pause with the LUS activity that will make the system unstable.
With Solaris machines (see below) you can bind a process to a dedicated CPU or even allocate min amount of CPU percentage to make sure it will get the resources it needs.

Multiple LUS will provide better HA , but will consume more resources from the spaces and in fact some more resources from the clients processes. The spaces will need to register themselves into multiple LUSs and the clients and spaces will get events to update their local lookup cache from all running LUSs. Please note we removed the LUS-client chat with 6.5 which makes the LUS to work less. This makes it more robust and reduce the amount of JVM temp objects it creates that may lead to long GC activity.

Due-to the above the recommendation is to have 2 LUS/ GSM running. No more.

With 6.5 we have improved large portions of the LUS and improved the LRMI communication mechanism that makes the LUS much more robust. This allows the LUS to support very large clusters with very large amount of clients. We have users running 6.5 with clusters larger than 100 spaces and more than 1000 clients.

6.0 will support up to 20 partitions out of the box but will need some tuning to its LUS JVM to be on the safe side. For larger clusters make sure you use 6.5 or above.
In extreme cases with gigantic systems we can scale the LUS activity by running multiple LUSs , each responsible for subset of the clients/spaces.

Here is an example for settings for the GSM:

-Xms1g -Xmx1g -XX:UseConcMarkSweepGC -XX:CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycleMin=10 -XX:CMSIncrementalDutyCycle=50 -XX:ParallelGCThreads=8
-XX:UseParNewGC -Xmn50m -XX:MaxGCPauseMillis=1000 -XX:GCTimeRatio=10 -XX:DisableExplicitGC

CPU resource management for GSM/LUS - Specifically for Solaris
To make sure the GSM/LUS will have CPU resources and continue to serve clients even under high CPU usage by other processes running on the machine you can use Solaris OS pbind and priocntl commands.

pbind -b ... (more)

asked 2008-08-18 03:42:36 -0500

shay hassidim gravatar image

updated 2013-08-08 09:52:00 -0500

jaissefsfex gravatar image
edit retag flag offensive close merge delete