Welcome to the new Gigaspaces XAP forum. To recover your account, please follow these instructions.

Ask Your Question
0

One-way firewall and too many open files

In our test environment we have a one-way firewall so we can connect to any server, but they can't connect back. So when we're deploying a data grid from the Management Center the ServiceProvisionListener notification fails. Not a big problem as we can just refresh the GUI and see what happened. But during our subsequent load tests the GSM fails with "too many open files" (the GSC has no problems). Could this be related (i.e. the repeated event notification failures accumulating open sockets, although the UI is long since closed) and is there a way around it (somehow deregistering the listener) or how do we launch our GSM and GSCs with a pre-defined grid (is a configured grip exportable)? h4. Attachments

[nohup.out|/upfiles/13759711308445987.txt]

{quote}This thread was imported from the previous forum. For your reference, the original is [available here|http://forum.openspaces.org/thread.jspa?threadID=2932]{quote}

asked 2009-03-05 09:12:22 -0500

mirv gravatar image

updated 2013-08-08 09:52:00 -0500

jaissefsfex gravatar image
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

Jonas,
I don't think there is a connection between these.
The default linux OS settings must be changed before moving into
production.
Fixing the ulmit settings (ulimit -n 32767) will resolve the "too many
open files" problem.

See the following for some tuning tips:
//question/5173/productionpocbenchmark-check-list/
http://www.gigaspaces.com/wiki/display/XAP66/TuningInfrastructure

I suggest you running the UI on the same network as the production
machines and use some remote desktop tool such as VNC or NX to access
it. It will resolve the auto refresh problems of the UI. Refreshing the
GUI manually can be annoying...

Shay

Attachments

  1. nohup.out

answered 2009-03-05 09:23:00 -0500

shay hassidim gravatar image
edit flag offensive delete link more

Comments

_Note to self: Don't post forum questions at the end of the day, they tend to be a bit rash_

I'm attaching the gs-all.sh nohup log where the issue can be seen (the connected remote GSC had no open file issues):

1:09:25 PM Grid deployment started 1.09:46 PM Grid deployment complete 1:11:35 PM First failed event notification (to the UI) every 2 minutes Repeated event notification failures 2:23:01 PM /proc/stat fails with Too many open files (ulimit is 1024) 2:23:34 PM Failed to create non-configured Exporter, defaulting (we have no exported configured) 2:24:45 PM Client socket accept starts to fail with Too many open files

Unfortunately I'm not a Linux admin enough to know of a command to list the open file descriptors of a running process so I just shut it down. h4. Attachments

[nohup.out|/upfiles/1375971131378653.txt]

mirv gravatar imagemirv ( 2009-03-06 00:40:04 -0500 )edit

Can u call this before running the gsc::
ulimit -n 32767
And see if this resolves the open files error?
To check the max open file run:
ulimit -a
Shay

shay hassidim gravatar imageshay hassidim ( 2009-03-06 06:59:40 -0500 )edit

As I said, ulimit is 1024 and by doing the following we're no longer getting it:

  1. Start GSCs and GSM inside firewall
  2. Start Management Center outside firewall and connect to GSM
  3. Deploy data grid (GSM will start polling MC as ServiceProvisionListener)
  4. Exit Management Center
  5. Stop GSM
  6. Start GSM (polling is gone)

This seems to indicate that the polling is in fact responsible for the too many open files (although I agree that's not what I would expect), but as we've avoided the situation we'll not being digging into it further at the moment.

mirv gravatar imagemirv ( 2009-03-09 06:41:05 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2009-03-05 09:12:22 -0500

Seen: 28 times

Last updated: Mar 05 '09