Welcome to the new Gigaspaces XAP forum. To recover your account, please follow these instructions.

Ask Your Question
0

Rapid Deterioration in Event Processing Rate WIth Polling Container

Hello,

Four instances of a client program write a total of 100,000 small objects, all instances of the same class, into a single space as quickly as they can. The space has a polling container that invokes an event handler for each new instance. The event handler does a trivial amount of work, then writes the object back with a changed value that prevents the object from being picked up again by the polling container.

Event processor performance starts out slow (50 - 60 per second) during the period when the objects are still being written to the space. It then climbs very quickly to around 2,000 per second, stays there for 10 - 20 seconds, then tail off steadily. By the time all 100,000 have been processed, more than 10 minutes have passed and the processing rate is in the 30s per second.

The cpu on the space host is pegged for the entire run, and the gsc reports this:

FINE [org.jini.rio.cybernode]: Threshold=CPU, Status=breached, Value=1, Low=0, High=1

from time to time.

What causes this dramatic deterioration in the processing rate, and how can I prevent it?

I have attached a few of the relevant source and xml files.

Thank you.

-Dan h4. Attachments

[PrimitiveIndexMain.java|/upfiles/13759710016341632.java]

[PrimitiveIndexPojo.gs.xml|/upfiles/13759710018586033.xml]

[PrimitiveIndexPojo.java|/upfiles/13759710012629633.java]

[PollingPUBean.java|/upfiles/13759710016808033.java]

[pu.xml|/upfiles/13759710018429533.xml]

[Incrementer.gs.xml|/upfiles/13759710019295033.xml]

[Incrementer.java|/upfiles/13759710013016433.java]

{quote}This thread was imported from the previous forum. For your reference, the original is [available here|http://forum.openspaces.org/thread.jspa?threadID=3018]{quote}

asked 2009-04-19 14:33:30 -0500

subuta gravatar image

updated 2013-08-08 09:52:00 -0500

jaissefsfex gravatar image
edit retag flag offensive close merge delete

1 Answer

Sort by » oldest newest most voted
0

Dan,
Can you explain why you need to update the Counter object as part of the doTran method?
Why you need to update this object with each transaction? I’ve got a feeling this is a major performance bottleneck. Do you have the Counter Key indexed? Or is it the Space ID field?

If you want to boost the processing you should use MultiTakeReceiveOperationHandler. See:
http://blog.gigaspaces.com/2008/10/03/how-to-implement-my-processor-the-polling-container-benchmark/

Why the Incrementer processed field is not indexed? It seems set as part of the template which means the matching is done based on this field. This field seems to be missing from the Incrementer.gs.xml.

Shay

Attachments

  1. PrimitiveIndexMain.java
  2. PrimitiveIndexPojo.gs.xml
  3. PrimitiveIndexPojo.java

answered 2009-04-19 23:13:30 -0500

shay hassidim gravatar image
edit flag offensive delete link more

Comments

Hi Shay,

The problem is standardized for a study I am writing. I can't optimize the hot spot away as I am comparing results achieved with the hot spot in place.

There are only 500 Counters, and the number doesn't change during the run. I commented out all of the Counter activity and saw the same deteriorating performance, so I knew that the deterioration was not a function of manipulating the counter. Sorry I didn't include this information in my post.

Indexing the 'processed' field did the trick. I guess the polling container was taking longer and longer to find unprocessed objects as the ratio of processed:unprocessed shifted over the course of the run. Indexing smoothed out the performance. It also improved performance.

The blog post looks very helpful. I will experiment with those techniques.

-Dan h4. Attachments

[PrimitiveIndexMain.java|/upfiles/13759710013204377.java]

[PrimitiveIndexPojo.gs.xml|/upfiles/13759710017738577.xml]

[PrimitiveIndexPojo.java|/upfiles/13759710019882377.java]

subuta gravatar imagesubuta ( 2009-04-20 01:57:22 -0500 )edit

Thanks. Very good. We are planning an option allowing the space to identify query on non-indexed field to allow users to fix such issues easily. Shay h4. Attachments

[PrimitiveIndexMain.java|/upfiles/13759710019585697.java]

[PrimitiveIndexPojo.gs.xml|/upfiles/13759710018238197.xml]

[PrimitiveIndexPojo.java|/upfiles/13759710016345997.java]

shay hassidim gravatar imageshay hassidim ( 2009-04-20 06:51:17 -0500 )edit

Cool. Please let me know what you discover about the problem I experienced with the index on the primitive field.

-Dan h4. Attachments

[PrimitiveIndexMain.java|/upfiles/13759710026825818.java]

[PrimitiveIndexPojo.gs.xml|/upfiles/13759710022525918.xml]

[PrimitiveIndexPojo.java|/upfiles/13759710028944418.java]

subuta gravatar imagesubuta ( 2009-04-20 07:10:03 -0500 )edit

There should not be any problem with such. Can you provide a simpler test case with a space class that include only an int indexed field data type which manifest the problem?

Shay h4. Attachments

[PrimitiveIndexMain.java|/upfiles/13759710022059238.java]

[PrimitiveIndexPojo.gs.xml|/upfiles/13759710029628238.xml]

[PrimitiveIndexPojo.java|/upfiles/13759710024537338.java]

shay hassidim gravatar imageshay hassidim ( 2009-04-20 07:43:00 -0500 )edit

Hi Shay,

Here you go.

I think you'll find that if you reverse the order of the two properties in the gs.xml file or switch to annotations then the read() works.

As a reminder, in January I sent you an email on the topic of the order of elements in gs.xml. Here is an excerpt:

+I am reasonably certain that null-value entries have to precede the routing entry in the class section of a gs.xml file. If they don't, it seems that a rule for the composition of the xml file is violated . . .

+I note that the documentation for gs.xml files mentions some mandatory sequencing, but it doesn't seem to describe this requirement. Perhaps there is more to be documented on the topic of the required order of items in the xml files.+

-Dan h4. Attachments

[PrimitiveIndexMain.java|/upfiles/1375971002753676.java]

[PrimitiveIndexPojo.gs.xml|/upfiles/1375971002361896.xml]

[PrimitiveIndexPojo.java|/upfiles/1375971002177816.java]

subuta gravatar imagesubuta ( 2009-04-20 08:53:52 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2009-04-19 14:33:30 -0500

Seen: 85 times

Last updated: Apr 19 '09