Welcome to the new Gigaspaces XAP forum. To recover your account, please follow these instructions.

Ask Your Question
0

GSIterator - Scratcher :(

Hi I have a cluster with 6 million objects of about 10 Gig of data on space. I need to take backups at EOD. I use GSIterator to get all objects in space and persist to a file. I donno I am setting iterator buffer size as 100 but some how client is pulling as much data as it can from space and client becomes slow consumer and eventually dies of outOfMemory. Questions If buffer size is set to 100. Does that mean client pulls the data in batches of 100 from the space and not worried about how much data client has consumed? Whats the good way to do this??

Following is sample code I use.

// Create a query for "New" and "Insecure" orderEvent objects. SQLQuery<orderevent> queryInsecureOrders = new SQLQuery<orderevent>(OrderEvent.class,"");
// Create and configure an iteratorBuilder to build iterators for the space with the specified templates. IteratorBuilder iteratorBuilder = new IteratorBuilder(gigaSpace) .addTemplate(queryInsecureOrders) .bufferSize(100) .withHistory(); GSIterator gsIterator = iteratorBuilder.iterate(); while (gsIterator.hasNext()){ get object write to file. }

{quote}This thread was imported from the previous forum. For your reference, the original is [available here|http://forum.openspaces.org/thread.jspa?threadID=3288]{quote}

asked 2009-10-22 16:07:28 -0500

venkatg gravatar image

updated 2013-08-08 09:52:00 -0500

jaissefsfex gravatar image
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

It will make more sense to perform the cold "backup" from within the space. This will avoid the need to ship the objects over the network. You can implement this via a task executor. This will speed up the process very much. Other option involves using the spacecopy API. This will copy the data into a persistent space.

Please note that the current GSIterator is not optimized for high performance with a remote partitioned space. This has been improved with 7.0.2. As a workaround you should connect to each partition separately and use its direct proxy with the GSIterator. This will be much faster.

Having a batch size of 1000 makes more sense. 100 sound too small.

In any case , a client should not get into out of memory. You should report this to Support.

Shay

answered 2009-10-22 17:47:36 -0500

shay hassidim gravatar image
edit flag offensive delete link more

Comments

Thanks ... direct proxy solved my issue.

venkatg gravatar imagevenkatg ( 2009-10-27 08:13:18 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2009-10-22 16:07:28 -0500

Seen: 79 times

Last updated: Oct 22 '09