Welcome to the new Gigaspaces XAP forum. To recover your account, please follow these instructions.

Ask Your Question
0

java process hogging the CPU

Hi,

I am running GigaSpaces XAP 6.5 GA, on Windows with jdk1.6.0_05.

The development team here run it on XP with a single space local. What we are seeing is that the CPU usage for the java exe gradually increases until it ends up at 80% when the space is idle. No one is writing to the space or reading from it and the process takes 80% until we close the space (it moves up and down by a few percent but never reduces to a reasonable level).

On our testing server we had set up 3 spaces, 2 of them had not been used at all other than viewing them through the GS-UI application. The 3 instances of java were taking 30+% processor each (the one with objects wasn't any worse than the empty ones) and the server was unusable.

I read in a post by Shay Hassidim ([here|//question/6517/high-cpu-usage/&tstart=180])that the CPU usage should be approximately 0% when the space is idle and we never see it running at less than 30% no matter how high a spec machine it is on.

There are no other java applications running on the machines so when we stop GigaSpaces the java exe closes down.

Has anyone seen this? Any idea what might be causing it?

Vin h4. Attachments

[HeapDumper.java|/upfiles/13759716203465634.java]

[MonitorWorker.java|/upfiles/13759716207573734.java]

{quote}This thread was imported from the previous forum. For your reference, the original is [available here|http://forum.openspaces.org/thread.jspa?threadID=2546]{quote}

asked 2008-08-20 12:59:59 -0500

vfinncm gravatar image

updated 2013-08-08 09:52:00 -0500

jaissefsfex gravatar image
edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted
0

Can you close the GS-UI running and see if this reduce the amount of CPU activity?
Shay

answered 2008-08-20 13:03:53 -0500

shay hassidim gravatar image
edit flag offensive delete link more

Comments

Shay,

Is 0% CPU still expected? I have observed (on 6.6.0 m3) 30 to 40% CPU utilization at idle with 8 partitions (7GB heap per partition). Profiling shows LeaseRenewalManager and PersistentGC threads as the top cpu hogs. This is with no UI attached.

Regards,
Larry

larrychu gravatar imagelarrychu ( 2008-08-20 18:11:00 -0500 )edit

Hi,

The GS-UI was closed. I only opened it for a few minutes to check if the space was alive and had data. On at least one of the developers machines where we saw 80% usage the GS-UI had never been run.

Vin

vfinncm gravatar imagevfinncm ( 2008-08-21 04:35:11 -0500 )edit
0

We saw this high cpu usage issue at system idle and took the following steps to alleviate. Ensure that :

  1. GS configuration (security policy etc) is correct. Any misconfiguration will cause gs to start but have high cpu usage.
  2. When GS starts, there should be no exceptions thrown
  3. Cache GS proxies instead of doing space lookup each time. eg: private static IJSpace space = (IJSpace)SpaceFinder.find(gsUrl); public IJSpace getSpace(){return space;}
  4. Ensure that all gs instances have unique space names in the gs startup urls: eg:

Instance 1: "/./mySpace1?group=<hostname here>+group" Instance 2: "/./mySpace2?group=<hostname here>+group" etc....

h4. Attachments

[HeapDumper.java|/upfiles/13759716214938521.java]

[MonitorWorker.java|/upfiles/13759716219755021.java]

answered 2008-08-21 14:13:30 -0500

wasefmasood gravatar image
edit flag offensive delete link more

Comments

The PersistentGC thread doing some internal cleanup every 10 seconds. This can be configured using the space-config.lease_manager.expiration_time_interval property.

The LeaseRenewalManager used at the client side and at the space side.

At the client side it is used with: - notify registration (DataEvertSession) - Map API local cache - Local View - Client side lookup cache

At the Space/GSM/GSC/PU side it is used when registering these into the LUS and renewing the lease.

The Space/GSM/GSC side LeaseRenewalManager activity can be controlled by the com.gs.jini.config.maxLeaseDuration (4 sec by default) and the com.gs.jini.config.roundTripTime (8 sec by default) properties. It is not advisable to modify these.

see more here: http://www.gigaspaces.com/wiki/displa...

My bet would be on the JVM that is causing this behavior. We found 1.6 causing problems. Can you try with 1.5?

Shay h4. Attachments

[HeapDumper.java|/upfiles/13759716213383243.java]

[MonitorWorker.java|/upfiles/13759716219400943.java]

shay hassidim gravatar imageshay hassidim ( 2008-08-21 16:09:52 -0500 )edit

Hi,

I do 2 and 3.

As for 1 there are no failures but thje log does show Info messages. +12-Aug-2008 08:05:49 INFO [net.jini.discovery.LookupDiscovery]: exception occurred during unicast discovery to 192.168.100.107:4162 with constraints InvocationConstraints[reqs: {}, prefs: {}]+ +Caused by: java.net.ConnectException: Connection timed out: connect+ +12-Aug-2008 08:18:40 INFO [net.jini.lookup.JoinManager]: JoinManager - failure+ +Caused by: java.rmi.ConnectException: An existing connection was forcibly closed by the remote host; nested exception is: java.io.IOException: An existing connection was forcibly closed by the remote host+

These are only Info not warnings or errors and they occur a few minutes apart so I woundn't expect them to cause continuous CPU usage but maybe they'll mean something to someone here?

Vin

vfinncm gravatar imagevfinncm ( 2008-08-25 07:39:48 -0500 )edit

This message means a client trying to look for a lookup service on 192.168.100.107:4162 and can't find it. This should not cause a vast CPU utilization. In any case , if such lookup service does not exists the client should not search for it. Shay

shay hassidim gravatar imageshay hassidim ( 2008-08-25 08:14:16 -0500 )edit

Hi,

I tried 1.5 (jdk1.5.0_16) and still have the problem.

It isn't quite as bad but is still fairly bad. I tried it on one of the machines where java was taking 80% and it dropped to 60%. On the machines where it was taking 30% it dropped to 20%.

So a noticeable improvement but still not good.

Vin h4. Attachments

[HeapDumper.java|/upfiles/13759716219940964.java]

[MonitorWorker.java|/upfiles/13759716211927064.java]

vfinncm gravatar imagevfinncm ( 2008-08-26 09:44:08 -0500 )edit

Getting a memory and thread dump will help here very much.

See attached utilities that will help you to get a thread dump and a memory dump from a running JVM.

You can also use JConsole to get these.

You might have a thread leak or something that is running in the background that is causing this behavior. We should be able to find this once we will have the memory and thread dump.

Shay h4. Attachments

[HeapDumper.java|/upfiles/13759716219384386.java]

[MonitorWorker.java|/upfiles/13759716218329386.java]

shay hassidim gravatar imageshay hassidim ( 2008-08-26 14:17:19 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2008-08-20 12:59:59 -0500

Seen: 121 times

Last updated: Aug 21 '08