Welcome to the new Gigaspaces XAP forum. To recover your account, please follow these instructions.

Ask Your Question
0

GSA Restarted GSC

Hi All,

In our production environment, The GSA restarted two GSC components:

1)At 2016-12-29 15:30:

2016-12-29 15:29:40,294 GSA WARNING [com.gigaspaces.grid.gsa] - [gsc][4/11780]: Reported Error [Restart Regex [.*java\.lang\.OutOfMemoryError.*] matching [java.lang.OutOfMemoryError: Java heap space]], restarting. restart-on-exit=ALWAYS
2016-12-29 15:29:40,295 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][4/11780]: Stopping process
2016-12-29 15:30:00,463 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][4/11780]: Killing process with sigar
2016-12-29 15:30:00,513 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][4/11780]: Process destroyed without shutdown hook
2016-12-29 15:30:00,514 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][4/]: Starting with command [/usr/local/payu/payu_v4/gigaspaces-xap-premium-10.0.1-ga//bin/gs.sh, start, "GSC"]
2016-12-29 15:30:06,453 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][4/20975]: Assigned process id

2) At 2016-12-29 15:50

2016-12-29 15:46:41,725 GSA WARNING [com.gigaspaces.grid.gsa] - [gsc][3/11702]: Reported Error [Restart Regex [.*java\.lang\.OutOfMemoryError.*] matching [java.lang.OutOfMemoryError: Java heap space]], restarting. restart-on-exit=ALWAYS
2016-12-29 15:46:41,725 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][3/11702]: Stopping process
2016-12-29 15:47:01,893 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][3/11702]: Killing process with sigar
2016-12-29 15:47:01,944 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][3/11702]: Process destroyed without shutdown hook
2016-12-29 15:47:01,944 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][3/]: Starting with command [/usr/local/payu/payu_v4/gigaspaces-xap-premium-10.0.1-ga//bin/gs.sh, start, "GSC"]
2016-12-29 15:47:06,799 GSA INFO [com.gigaspaces.grid.gsa] - [gsc][3/397]: Assigned process id

In all the logs file there are not OutOfMemoryException or SpaceMemoryShortageException.

Please can you help us to identify why the GSA restarted the GSC´s components?

Log files: C:\fakepath\nodo2.zip

Kind Regards.

asked 2017-01-09 15:07:02 -0500

guedim gravatar image

updated 2017-01-09 15:14:24 -0500

edit retag flag offensive close merge delete

1 answer

Sort by » oldest newest most voted
0

Hi,

You had long GC that took more than 2 minutes. This cause the GSC to become available and restarted by the GSA.

2016-12-29 15:50:06,030 WARNING [com.gigaspaces.start] - Long GC collection occurred, took [126922ms], breached threshold [60000]

Regards,

Yuval

answered 2017-01-10 04:14:32 -0500

Yuval gravatar image
edit flag offensive delete link more

Comments

Also the OOO actually occurred on the GSC but the log message didn't flush to the log file. The GSA which monitor the GSC memory was aware of it.

Yuval gravatar imageYuval ( 2017-01-10 05:19:47 -0500 )edit

Thank you.

We found the .hprof files (java_pid11702.hprof - java_pid11780.hprof) with the memory issues. Also the garbage files with Consecutive Full GC and Long Pauses.

Thanks.

guedim gravatar imageguedim ( 2017-01-10 11:57:54 -0500 )edit

did you solve the problem? Or find a way to solve it? I also have this issue with 10.0.2 version, it seems like this version has a lot of memory issues

Ivan Y gravatar imageIvan Y ( 2017-01-11 15:08:05 -0500 )edit

No, Right now i have to add more memory to the GSC.

But, also my question is, Why there are not SpaceMemoryShortageException logs? I try to replicate the error, but I always have SpaceMemoryShortageException before OutOfMemoryException.

|14:29:07.930|http-nio-172.18.16.10-8085-exec-107|ERROR|com.payu.houston.babylon.api.helpers.ApiErrorResponseHelper | error processing payment tx ! :(
com.j_spaces.core.**MemoryShortageException**: Memory shortage at: host: 172.18.29.15, container: pppPpsPuSpace_container2, space pppPpsPuSpace, total memory: 972 mb, used memory: 833 mb
        at com.j_spaces.core.MemoryManager.shortageCheck(MemoryManager.java:388) ~[gs-runtime-10.0.1-11800-RELEASE.jar:na]
        at com.j_spaces.core.MemoryManager.monitorMemoryUsageWithNoEviction_Impl(MemoryManager.java:313) ~[gs-runtime-10.0.1-11800-RELEASE.jar:na]
        at com.j_spaces.core.MemoryManager.monitorMemoryUsage(MemoryManager.java:257) ~[gs-runtime-10.0.1-11800-RELEASE.jar:na]
        at com.gigaspaces.internal.server.space.SpaceImpl.executeTask(SpaceImpl.java:2850) ~[gs-runtime-10.0.1-11800-RELEASE.jar:na]
        at com.gigaspaces.internal.server.space.operations.ExecuteTaskSpaceOperation.execute(ExecuteTaskSpaceOperation.java:24) ~[gs-runtime-10.0.1-11800-RELEASE.jar:na]

Kind regards.

guedim gravatar imageguedim ( 2017-01-11 21:41:20 -0500 )edit

Ivan,

What kind of memory issues do you have with GS 10.0.2. I have the same version.

Kind Regards

guedim gravatar imageguedim ( 2017-01-11 21:42:42 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

Stats

Asked: 2017-01-09 15:07:02 -0500

Seen: 58 times

Last updated: Jan 10