Welcome to the new Gigaspaces XAP forum. To recover your account, please follow these instructions.

Ask Your Question
0

Where to start?

How would you use Gigaspaces as the processing engine for a messaging system like the following: - Some number of messaging clients (say 100) connecting to several front end machines. - Each incoming message would be inserted into a space for processing. - For each message inserted: - Validate message. Send reply to sender if error - Write message to data storage of some sort (database or grid) - Write message back into a space (or queue) for a listener on the front end system to read and send to the recipient.

This is pretty much a standard Master/Worker pattern except for the following: - The order of messages must be preserved for a given sender/target pair. So although a sender may send 10 messages, if all of them are to the same recipient, they have to processed one by one and delivered to the recipients in the order in which they were sent. But if they are all to different recipients they can be processed in parallel.

{quote}This thread was imported from the previous forum. For your reference, the original is [available here|http://forum.openspaces.org/thread.jspa?threadID=2136]{quote}

asked 2007-05-27 16:42:02 -0600

updated 2013-08-08 09:52:00 -0600

jaissefsfex gravatar image
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

William, See below answers embedded within your original text:

+How would you use GigaSpaces as the processing engine for a messaging system like the following: - Some number of messaging clients (say 100) connecting to several front end machines. - Each incoming message would be inserted into a space for processing. + This should be a simple write operation. You can consider using batch write to speed up this operation.

+- For each message inserted: - Validate message. + This is a simple take operation. The template should be a simple template to match all inserted messages.

+Send reply to sender if error + The validate functionality should write back into the space status object. A worker will consume error status objects via take operation and act upon.

+- Write message to data storage of some sort (database or grid) + Do you need to do that in sync or async mode? Does this need to be part of a space operation or separate transaction? In any case , this can be done via a worker that is taking status object with success value and perform the database activity. Please note this might be the bottleneck of the system , so here you should consider to perform the database activity in async manner or have multiple workers dealing with database operations.

+- Write message back into a space (or queue) for a listener on the front end system to read and send to the recipient. + Once the worker dealing with writing the data into the data storage completed , it will write relevant object to the space. This will be taken by relevant worker that will send ack to the recipient.

+This is pretty much a standard Master/Worker pattern except for the following: - The order of messages must be preserved for a given sender/target pair. So although a sender may send 10 messages, if all of them are to the same recipient, they have to processed one by one and delivered to the recipients in the order in which they were sent. But if they are all to different recipients they can be processed in parallel.+ To preserve FIFO per recipients while allowing different recipients to be processed in parallel , you should assign each recipient an ID. The inserted Validate messages that start the whole process should be written in FIFO mode. The workers should perform take operation in FIFO mode.

To be able to scale and process data in parallel you should partition the space, and have the workers running IWorkers implementation having the routing field as the recipient ID. With the above architecture each space will act as a queue for different set of recipients , effectively constructing parallel queue. This will allow every recipient to have its data processed in FIFO manner, but in the same time process the difference recipients in parallel.

See the following as a starting point for your application: http://www.gigaspaces.com/wiki/displa...

See more here: http://www.gigaspaces.com/wiki/displa...

With GigaSpaces 6.0 the above would be implemented using processing units via Spring based wiring of the space operations: http://www.gigaspaces.com/wiki/displa...

Shay

answered 2007-05-27 21:11:02 -0600

shay hassidim gravatar image
edit flag offensive delete link more

Comments

I actually work with Bill.

We do plan on doing the dbms update async however we can't lose a message. We need to add messages to the space and have them persist hopefully in a high performance file store which is shared using a clustered files system on a solid state disk device.

Then we need to have an event handler deal with the sql dbms insertion async.

I am thinking of using regions.

1 region is replicated then have a local region per member in the grid.

Trying to figure out the best topology for our proof of concept.

Any ideas would be much appreciated.

I kind of have something working but i am not sure its actually what i want.

tsporthd gravatar image tsporthd  ( 2007-06-19 15:44:01 -0600 )edit

In general you should use the mirror running/accessing embedded persistent space. Send me an email to shay@gigaspaces.com to get more details. Shay

shay hassidim gravatar image shay hassidim  ( 2007-06-19 18:20:02 -0600 )edit

Ok since we are going to use the Grid as a DBMS replacement in a sense the idea is as follows:

Message come into the grid from our fix servers.

We need to make sure that each message is logged so in the event of a total grid failure we can re-start the grid and not lose anything. Hence when we write them to the grid we want them to be stored on disk in a transaction. Based on experience with other vendors it seems that using a backing store which is file based is much faster and of course when using SSD its really fast. So the write to the grid would write to the file. Also we would use an event to write part of the message to a Relational DBMS which can be used for reporting. We would handle the write to the relational dbms in an async manner since our recovery given a total grid outage would be done via the files.

In my proof of concept I have 30% of the message in a replicated cluster in a Service Grid then remaining 70% are in partitioned cache both use persistence and i do see the files being created in the GenericPersistProperties directory under GigaSpacesEE5.2.

tsporthd gravatar image tsporthd  ( 2007-07-06 09:59:00 -0600 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2007-05-27 16:42:02 -0600

Seen: 65 times

Last updated: May 27 '07