Welcome to the new Gigaspaces XAP forum. To recover your account, please follow these instructions.

Ask Your Question
0

Gigaspaces LRMI Java Heap Space OutOfMemoryError

Hi,

I am getting a java.lang.OutOfMemoryError: Java heap space on light remote method invocation protocol (and watchdog). You can see the log below. Does anyone know why this would happen and how to solve it?

What I tried is changing the maximum heap size for the agent, the container, the lookup service and manager:

export GSA_JAVA_OPTIONS=-Xmx256m  
export GSC_JAVA_OPTIONS=-Xmx8192m  
export GSM_JAVA_OPTIONS=-Xmx4096m  
export LUS_JAVA_OPTIONS=-Xmx4096m

This didn't help either. It makes sense though - the heap is failing because of lrmi related issue, but I don't see a setting that can help me solve it.

2013-10-05 21:01:53,124 GSC SEVERE [com.gigaspaces.lrmi] - exception in main selection loop; Caused by: java.lang.OutOfMemoryError: Java heap space  
    at java.util.HashMap.newKeyIterator(HashMap.java:886)

    at java.util.HashMap$KeySet.iterator(HashMap.java:920)

    at java.util.HashSet.iterator(HashSet.java:170)

    at sun.nio.ch.Util$2.iterator(Util.java:303)

    at com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.doSelect(AbstractSelectorThread.java:67)

    at com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.run(AbstractSelectorThread.java:51)

    at java.lang.Thread.run(Thread.java:722) 


2013-10-05 21:02:20,232 GSC SEVERE [com.gigaspaces.lrmi] - exception in main selection loop; Caused by: java.lang.OutOfMemoryError: Java heap space  

2013-10-05 21:02:40,582 GSC SEVERE [com.gigaspaces.lrmi] - exception in main selection loop; Caused by: java.lang.OutOfMemoryError: Java heap space  

2013-10-05 21:04:09,455 gs-space-0.0.2.2 [1] INFO [com.sun.jini.reggie] - unmarshalling failure; Caused by: java.lang.OutOfMemoryError: Java heap space   2013-10-05 21:04:45,675 GSC SEVERE [com.gigaspaces.lrmi] - exception in main selection loop; Caused by: java.lang.OutOfMemoryError: Java heap space  

2013-10-05 21:04:49,640 GSC SEVERE [com.gigaspaces.lrmi.watchdog] - Unexpected exception in watchdog thread.; Caused by: java.lang.OutOfMemoryError: Java heap space  

2013-10-05 21:05:05,617 gs-space-0.0.2.2 [1] WARNING [com.sun.jini.thread.TaskManager] - GS-JoinManager Task ran [net.jini.lookup.JoinManager$ProxyRegTask@69f9aca2] and caught an exception.; Caused by: java.lang.OutOfMemoryError: Java heap space   2013-10-05 21:04:47,283 GSC SEVERE [com.gigaspaces.lrmi] - exception in main selection loop; Caused by: java.lang.OutOfMemoryError: Java heap space  

2013-10-05 21:04:40,335 GSC SEVERE [com.gigaspaces.lrmi] - exception in main selection loop; Caused by: java.lang.OutOfMemoryError: Java heap space  

2013-10-05 21:04:51,065  WARNING [com.sun.jini.thread.TaskManager] - GS-LookupDiscovery Task ran [net.jini.discovery.LookupDiscovery$DecodeAnnouncementTask@5164ce69] and caught an exception.; Caused by: java.lang.OutOfMemoryError: Java heap space   2013-10-05 21:05:21,328 gs-space-0.0.2.2 [1] WARNING [com.sun.jini.thread.TaskManager] - GS-JoinManager Task ran [net.jini.lookup.JoinManager$ProxyRegTask@4a8f3a1d] and caught an exception.; Caused by: java.lang.OutOfMemoryError: Java heap space   2013-10-05 21:05:14,649 gs-space-0.0.2.2 [1] WARNING [com.sun.jini.thread.TaskManager] - GS-JoinManager Task ran [net.jini.lookup.JoinManager$ProxyRegTask@3787ca39] and caught an exception.; Caused by: java.lang.OutOfMemoryError: Java heap space   2013-10-05 21:04:48,512 GSC WARNING [com.sun.jini.thread.TaskManager] - GS-LeaseRenewalManager Task ran [net.jini.lease.LeaseRenewalManager$RenewTask@4e833ee2] and caught an exception.; Caused by: java.lang.OutOfMemoryError: Java heap space   2013-10-05 21:05:51,389 gs-space-0.0.2.2 [1] SEVERE [com.sun.jini.thread.TaskManager] - Ran [net.jini.lookup.JoinManager$ProxyRegTask@7c0469d7] but caught exception:   java.lang.OutOfMemoryError: Java heap space   Attempt to log caused: ; Caused by: java.lang.OutOfMemoryError: Java heap space

I have a 8 server configuration with 4 GSM's 4 LUS's 16 GSC's (or 8 GSC's) 8 primary partitions and 8 backups (or just 8 primaries). I am getting the error when I try to insert/write objects in the space one by one. When I reach around 200000 (the first partition that reaches this number, fails with the exception above. And I can see that the RAM is not yet field - there is still space in there (8GB - 3.5GB used - around 4GB free).

I also see some additional information in the GSC console (which I couldn't find in the log) about FastConcurrentSkipListMap's iterator causing the issue: image description

If anyone likes to visit the question on stackoverflow, its here.

EDIT:

What's the XAP release you use?

Gigaspaces:
gigaspaces-xap-premium-9.6.1-ga-b9700
Java:
java version "1.7.0_15"
Java(TM) SE Runtime Environment (build 1.7.0_15-b03)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)

Can you please post the code that perform this remote activity?

Its nothing special (just a write), and I am doing this for 5000000 times for example:

        LeaseContext<Profile> profileLeaseCotnext = space.write(profile);
        Profile profile = profileLeaseCotnext.getObject();
        LOGGER.debug("Wrote " + profile + "with UID:" + profileLeaseCotnext.getUID() + ".");

Do you have very large objects written into the space?

I can't see the exact size, but the profile is an object that contains integers, strings (couple of words), dates and objects (2 or 3 of them) containing the same things. Most of them are empty. I can see the stuff in the ui. Its really not that big. Nothing in there worth for a BLOB.

asked 2013-10-06 09:19:38 -0500

Despot gravatar image

updated 2013-10-07 12:10:48 -0500

edit retag flag offensive close merge delete

3 Answers

Sort by ยป oldest newest most voted
0

Which log did you provide? Forget about "available RAM" on the box(es), all you care about is JVM heap. When this error occurs, can you verify that every JVM in the cluster has plenty of headroom? Use the UI and/or JConsole.

answered 2013-10-08 11:17:54 -0500

DeWayne gravatar image
edit flag offensive delete link more

Comments

Its not coming nearly to the available RAM space. So that's not the issue here.

Despot gravatar imageDespot ( 2013-10-08 15:00:29 -0500 )edit

Yes I know. The question is "are any of the many jvms in the cluster running out of memory"?

DeWayne gravatar imageDeWayne ( 2013-10-08 15:52:33 -0500 )edit

Ah ok. Since I already provided, in my original post, a quote regarding making the jvm memory bigger (see GSA_JAVA_OPTIONS, GSC_JAVA_OPTIONS, GSM_JAVA_OPTIONS, LUS_JAVA_OPTIONS), this " Forget about "available RAM" on the box(es)" sounded to me like a question.

Despot gravatar imageDespot ( 2013-10-10 03:17:49 -0500 )edit

You were spot on. Although I placed the jvm options I forgot to change the "export" (linux command) to "set" (windows command), so they were never made bigger. I checked that with the UI (hosts tab -> click on GSC and check the jvm available space and used space) with the help of Shay Hassidim ( http://ask.gigaspaces.org/users/32/sh... ). Although Shay helped me first figure this out, I'll pick your answer as correct, since somehow I believe that he is not doing this for the points :) and on the other hand you are missing a boost ;)

Despot gravatar imageDespot ( 2013-10-10 03:25:15 -0500 )edit
1

Can you please post the code that perform this remote activity?

Do you have very large objects written into the space?

What's the XAP release you use?

Increasing the GSM , LUS and GSA heap would not help.

Why you run 4 GSM's 4 LUS's ? You need 2 from each max.

answered 2013-10-06 09:26:38 -0500

shay hassidim gravatar image

updated 2013-10-06 09:26:48 -0500

edit flag offensive delete link more

Comments

"Why you run 4 GSM's 4 LUS's ? You need 2 from each max." This doesn't affect the issue. I tried with 2 of them and got the same result.

Despot gravatar imageDespot ( 2013-10-07 11:28:17 -0500 )edit

@shay, I answered your questions in my original post (see the EDIT section).

Despot gravatar imageDespot ( 2013-10-07 12:11:51 -0500 )edit

You should use the default mode where nothing is returned back after a write. It will speed up the write and lower the amount of LRMI activity when updating data.

I need your heap dump and some additional info. Can you please contact me directly? I'm not sure LRMI is the problem. It might be something else leaking.

Shay at gigaspaces.com

Tnx

shay hassidim gravatar imageshay hassidim ( 2013-10-07 16:37:32 -0500 )edit

Thanks for your help Shay!!!!!

Despot gravatar imageDespot ( 2013-10-10 03:29:15 -0500 )edit
0

If we could able to see the demo then it will be clearly understood by everyone. javascript charts

answered 2013-10-08 02:22:32 -0500

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2013-10-06 09:19:38 -0500

Seen: 2,144 times

Last updated: Oct 08 '13