PermGen error when HotDeploy

edit

Hi Shay,

We have reviewed the implementation of the xap-hot-deploy project and configured to work with our platform but we still get the same error with the PermGen memory.

This image shows the PermGen memory after the first and second hot-deploy attempt:

In the second attempt, a java.lang.OutOfMemoryError is thrown an all GSC are restarted (in production environment this take much more time to recover):

2014-11-25 12:20:13,188 GSA WARNING [com.gigaspaces.grid.gsa] - [gsc][2/21752]: Reported Error [Restart Regex [.*java\.lang\.OutOfMemoryError.*] matching [java.lang.OutOfMemoryError: PermGen space]], restarting. restart-on-exit=ALWAYS
2014-11-25 12:20:13,188 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][2/21752]: Stopping process
2014-11-25 12:20:33,401 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][2/21752]: Process destroyed without shutdown hook
2014-11-25 12:20:33,401 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][2/]: Starting with command [D:\Desarrollo\Servers\gigaspaces-xap\/bin/gsc.bat]
2014-11-25 12:20:34,300 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][2/17164]: Assigned process id
2014-11-25 12:20:48,058 GSA WARNING [com.gigaspaces.grid.gsa] - [gsc][1/2352]: Reported Error [Restart Regex [.*java\.lang\.OutOfMemoryError.*] matching [java.lang.OutOfMemoryError: PermGen space]], restarting. restart-on-exit=ALWAYS
2014-11-25 12:20:48,059 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][1/2352]: Stopping process
2014-11-25 12:21:08,261 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][1/2352]: Process destroyed without shutdown hook
2014-11-25 12:21:08,261 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][1/]: Starting with command [D:\Desarrollo\Servers\gigaspaces-xap\/bin/gsc.bat]
2014-11-25 12:21:08,780 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][1/8840]: Assigned process id

Do you have any workaround to prevent this downtime? Or some technique/configuration throw GS to reduce the usage of the PermGen memory?

Thanks

Manuel Vieda

Do you have any thread local usage with your code?

You will have to restart the GSC instead of restarting the PU instance to resolve this.

There might be several reasons for this. None of them related directly to XAP. It could be Java issue or some hanging threads (thread local mostly) that prevents the classloader to be dropped completely when restrting the PU instance. We have seen this problem. Its not new.

We have developed our own hot-deploy-agent, customized for our platform. Anyway, we tried this tool and get the same result. After running the process a few times (2 or 3) we get the same java.lang.OutOfMemoryError: PermGen space following by a GSC restart (downtime).

I suggest you restart the GSC instead the PU instance.

That's the simplest solution.