Welcome to the new Gigaspaces XAP forum. To recover your account, please follow these instructions.

Ask Your Question
0

PermGen error when HotDeploy

Hi, we are working on a Hot Deploy solution for our plattform and we are experiencing out of memory errors after a few PU restarts. The restart process is controlled by using the GigaSpaces Admin API (We have our own agent, very similiar to this one: http://docs.gigaspaces.com/sbp/xap-ho... ).

Some runtime details:

  • GigaSpaces 9.6
  • GSA, 1 GSM, 2 GSC
  • All the PUs: Cluster schema as partitioned-sync2backup with 2 primaries, 1 backup each.

A workaround used is restarting the GSC after the hot deploy (pu restarts) or when the used PermGen is higher than 80%. The problem with this solution is that after restarting the GSC all the primaries are located in the same GSC and the backups in the other GSC. Is there a way to force that each GSC always have 1 primary and 1 backup?

Do you have any recomendation o reduce the PermGem space usage or do some cleaning? (We are using the plattform/lib/ext and optional/pu-common folders to store third party libraries).

Thanks,

Manuel Vieda

asked 2014-11-24 18:08:25 -0600

edit retag flag offensive close merge delete

3 Answers

Sort by ยป oldest newest most voted
0

I suggest you start with this: http://docs.gigaspaces.com/sbp/xap-ho...

Shay

answered 2014-11-24 19:20:51 -0600

shay hassidim gravatar image
edit flag offensive delete link more

Comments

Hi Shay,

We have reviewed the implementation of the xap-hot-deploy project and configured to work with our platform but we still get the same error with the PermGen memory.

This image shows the PermGen memory after the first and second hot-deploy attempt:

image description

In the second attempt, a java.lang.OutOfMemoryError is thrown an all GSC are restarted (in production environment this take much more time to recover):

2014-11-25 12:20:13,188 GSA WARNING [com.gigaspaces.grid.gsa] - [gsc][2/21752]: Reported Error [Restart Regex [.*java\.lang\.OutOfMemoryError.*] matching [java.lang.OutOfMemoryError: PermGen space]], restarting. restart-on-exit=ALWAYS
2014-11-25 12:20:13,188 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][2/21752]: Stopping process
2014-11-25 12:20:33,401 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][2/21752]: Process destroyed without shutdown hook
2014-11-25 12:20:33,401 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][2/]: Starting with command [D:\Desarrollo\Servers\gigaspaces-xap\/bin/gsc.bat]
2014-11-25 12:20:34,300 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][2/17164]: Assigned process id
2014-11-25 12:20:48,058 GSA WARNING [com.gigaspaces.grid.gsa] - [gsc][1/2352]: Reported Error [Restart Regex [.*java\.lang\.OutOfMemoryError.*] matching [java.lang.OutOfMemoryError: PermGen space]], restarting. restart-on-exit=ALWAYS
2014-11-25 12:20:48,059 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][1/2352]: Stopping process
2014-11-25 12:21:08,261 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][1/2352]: Process destroyed without shutdown hook
2014-11-25 12:21:08,261 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][1/]: Starting with command [D:\Desarrollo\Servers\gigaspaces-xap\/bin/gsc.bat]
2014-11-25 12:21:08,780 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][1/8840]: Assigned process id

Do you have any workaround to prevent this downtime? Or some technique/configuration throw GS to reduce the usage of the PermGen memory?

Thanks

Manuel Vieda

Manuel Vieda gravatar imageManuel Vieda ( 2014-11-25 11:42:04 -0600 )edit

Do you have any thread local usage with your code?

You will have to restart the GSC instead of restarting the PU instance to resolve this.

There might be several reasons for this. None of them related directly to XAP. It could be Java issue or some hanging threads (thread local mostly) that prevents the classloader to be dropped completely when restrting the PU instance. We have seen this problem. Its not new.

shay hassidim gravatar imageshay hassidim ( 2014-11-25 12:07:41 -0600 )edit
0

Manuel, did you try to use http://docs.gigaspaces.com/sbp/xap-ho... tool? Or what custom behavior do you need?

answered 2014-11-25 11:10:21 -0600

edit flag offensive delete link more

Comments

We have developed our own hot-deploy-agent, customized for our platform. Anyway, we tried this tool and get the same result. After running the process a few times (2 or 3) we get the same java.lang.OutOfMemoryError: PermGen space following by a GSC restart (downtime).

Manuel Vieda gravatar imageManuel Vieda ( 2014-11-26 15:30:17 -0600 )edit

I suggest you restart the GSC instead the PU instance.

That's the simplest solution.

shay hassidim gravatar imageshay hassidim ( 2014-11-26 15:55:58 -0600 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2014-11-24 18:08:25 -0600

Seen: 110 times

Last updated: Nov 24 '14