PermGen error when HotDeploy

Hi, we are working on a Hot Deploy solution for our plattform and we are experiencing out of memory errors after a few PU restarts. The restart process is controlled by using the GigaSpaces Admin API (We have our own agent, very similiar to this one: http://docs.gigaspaces.com/sbp/xap-ho... ).

Some runtime details:

GigaSpaces 9.6
GSA, 1 GSM, 2 GSC
All the PUs: Cluster schema as partitioned-sync2backup with 2 primaries, 1 backup each.

A workaround used is restarting the GSC after the hot deploy (pu restarts) or when the used PermGen is higher than 80%. The problem with this solution is that after restarting the GSC all the primaries are located in the same GSC and the backups in the other GSC. Is there a way to force that each GSC always have 1 primary and 1 backup?

Do you have any recomendation o reduce the PermGem space usage or do some cleaning? (We are using the plattform/lib/ext and optional/pu-common folders to store third party libraries).

Thanks,

Manuel Vieda

asked 2014-11-24 18:08:25 -0600

Manuel Vieda
3 ●9 ●10 ●12 http://manuelvieda.com/

edit retag flag offensive close merge delete

add a comment

3 Answers

Sort by » oldest newest most voted

I suggest you start with this: http://docs.gigaspaces.com/sbp/xap-ho...

Shay

answered 2014-11-24 19:20:51 -0600

shay hassidim

17818 ●7 ●11 ●15

edit flag offensive delete link

Comments

Hi Shay,

We have reviewed the implementation of the xap-hot-deploy project and configured to work with our platform but we still get the same error with the PermGen memory.

This image shows the PermGen memory after the first and second hot-deploy attempt:

image description

In the second attempt, a java.lang.OutOfMemoryError is thrown an all GSC are restarted (in production environment this take much more time to recover):

2014-11-25 12:20:13,188 GSA WARNING [com.gigaspaces.grid.gsa] - [gsc][2/21752]: Reported Error [Restart Regex [.*java\.lang\.OutOfMemoryError.*] matching [java.lang.OutOfMemoryError: PermGen space]], restarting. restart-on-exit=ALWAYS
2014-11-25 12:20:13,188 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][2/21752]: Stopping process
2014-11-25 12:20:33,401 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][2/21752]: Process destroyed without shutdown hook
2014-11-25 12:20:33,401 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][2/]: Starting with command [D:\Desarrollo\Servers\gigaspaces-xap\/bin/gsc.bat]
2014-11-25 12:20:34,300 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][2/17164]: Assigned process id
2014-11-25 12:20:48,058 GSA WARNING [com.gigaspaces.grid.gsa] - [gsc][1/2352]: Reported Error [Restart Regex [.*java\.lang\.OutOfMemoryError.*] matching [java.lang.OutOfMemoryError: PermGen space]], restarting. restart-on-exit=ALWAYS
2014-11-25 12:20:48,059 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][1/2352]: Stopping process
2014-11-25 12:21:08,261 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][1/2352]: Process destroyed without shutdown hook
2014-11-25 12:21:08,261 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][1/]: Starting with command [D:\Desarrollo\Servers\gigaspaces-xap\/bin/gsc.bat]
2014-11-25 12:21:08,780 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][1/8840]: Assigned process id

Do you have any workaround to prevent this downtime? Or some technique/configuration throw GS to reduce the usage of the PermGen memory?

Thanks

Manuel Vieda

Manuel Vieda ( 2014-11-25 11:42:04 -0600 )edit

Do you have any thread local usage with your code?

You will have to restart the GSC instead of restarting the PU instance to resolve this.

There might be several reasons for this. None of them related directly to XAP. It could be Java issue or some hanging threads (thread local mostly) that prevents the classloader to be dropped completely when restrting the PU instance. We have seen this problem. Its not new.

shay hassidim ( 2014-11-25 12:07:41 -0600 )edit

add a comment

Manuel, did you try to use http://docs.gigaspaces.com/sbp/xap-ho... tool? Or what custom behavior do you need?

answered 2014-11-25 11:10:21 -0600

Pavlo Romanenko
1

edit flag offensive delete link

Comments

We have developed our own hot-deploy-agent, customized for our platform. Anyway, we tried this tool and get the same result. After running the process a few times (2 or 3) we get the same java.lang.OutOfMemoryError: PermGen space following by a GSC restart (downtime).

Manuel Vieda ( 2014-11-26 15:30:17 -0600 )edit

I suggest you restart the GSC instead the PU instance.

That's the simplest solution.

shay hassidim ( 2014-11-26 15:55:58 -0600 )edit

add a comment

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

PermGen error when HotDeploy

3 Answers

Comments

Comments

Your Answer

Question Tools

Stats

Related questions

PermGen error when HotDeploy edit

3 Answers

Comments

Comments

Your Answer

Question Tools

Stats

Related questions

PermGen error when HotDeploy