Welcome to the new Gigaspaces XAP forum. To recover your account, please follow these instructions.

Ask Your Question
0

Need help with design

I'm a GS newbie, and am not sure how to best implement the following feature:

1) A provider (for ex. JMS queue) generates messages, which have some key data elements, for ex. movement_nr and movement_date 2) For any key value, only one message may be processed at a time (the others with the same key value must wait until the active one has been sent on to its destination) 3) Other messages may be processed in parallel.

Example: consider 5 messages with keys

1) AAA111-03 2) BBB222-03 3) AAA111-03 4) CCC333-12 5) DDD444-23

Messages 1, 2, 4 and 5 may be processed in parallel, but message 3 has to wait until message 1 has been processed.

I do not know in advance what the set of key values will be, or how many different key values there will be (several hundreds or even thousands)

Any advice?

Thanks,

Isabelle

{quote}This thread was imported from the previous forum. For your reference, the original is [available here|http://forum.openspaces.org/thread.jspa?threadID=2064]{quote}

asked 2007-10-18 08:31:03 -0600

updated 2013-08-08 09:52:00 -0600

jaissefsfex gravatar image
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

Here is a very basic description how the problem should be addressed:
- The space will be running in partitioned topology running collocated worker (PU).
- Each PU will be acting as a ?queue? ? hence constructing a parallel distributed queue
- The JMS will be routing the incoming messages into the relevant PU in FIFO mode. This will be done via our JMS API that transfers JMS messages to Space POJO?s.
- The worker within the PU will be consuming the incoming messages in FIFO mode ensuring the consumption of the messages in the correct order.

How this sounds?

Shay

answered 2007-10-18 11:04:01 -0600

shay hassidim gravatar image
edit flag offensive delete link more

Comments

Hi Shay,

Well, I sort of had been thinking along the same lines, but : I do not know how many PU's I would need, and I do not know the values of the keys either so I cannot create the queues beforehand either.

Also: I do not know when to clean up a PU (i.e. once I have seen a specific key, I do not know how many more of them I will get - maybe none, maybe many).

By the way, my sources are not all JMS, could be files in a directory or whatever. So using the JMS approach is not a must.

Ignoring Gigaspaces for a minute, one possible logical layout is as follows:

The input comes in on a queue, a single Gatekeeper component is listening to the queue.There is also a pool of workers, let us assume the pool size is 3.

1) The Gatekeeper extracts the key out the message and determines whether the message has to wait (i.e. whether a worker is currently working on the same key). 2) If the message has to wait, it goes to a FIFO parking lot for its key. 3) If it does not have to wait and a worker is available, it is assigned to the worker 4) If it does not have to wait and no worker is available, it goes to a FIFO waiting room

Once a worker is done with a message, it will first try to retrieve its next task from the waiting room for its current key. If that is empty, it tries to take the first message from the waiting room. If that is also empty, it sits around waiting for work.

The waiting room could be a JMS queue. But the parking lots have to be created/removed on-demand.

Do you think that would work? What would be the best way to implement the on-demand parking lots?

I prepared a PDF file with a drawing of the situation at the start, and after the first messages have been processed, but I have a problem uploading it (your Tomcat server gives a NullPointerException). Thanks for your help,

isabelle gravatar imageisabelle ( 2007-10-19 02:01:03 -0600 )edit

Hi Isabelle, if I am not mistaken we have discussed this project in the past.

The GigaSpaces master-worker pattern addresses very nicely your requirements.

See http://www.gigaspaces.com/wiki/display/GS6/ComputeServer-Master-WorkerPattern-6.0

Our Open Space programming model will allow you to develop and deploy your master-worker Spring based application in a distributed environment.

See http://www.gigaspaces.com/wiki/display/GS6/OpenSpaces

You can send my the diagram to guys at gigaspaces dot com.

Regards,
Guy

guy sayar gravatar imageguy sayar ( 2007-10-19 02:09:01 -0600 )edit

Hi Guy,

Indeed we did. We are now looking at GS again, because

1) our requirements have changed (much higher throughput required than previously estimated) 2) we have serious 2nd thoughts about the more traditional architectures (don't say "I told you so" :wink: )

I am going to send you the diagrams by mail in a minute. One thing I do not like about my diagram is that the solution is intrusive: the architecture would change significantly if I wanted to not do the sequencing. There are currently discussions about this, if the impact on throughput is too big we may need to drop the sequencing.

Best regards,

Isabelle

isabelle gravatar imageisabelle ( 2007-10-19 02:15:04 -0600 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2007-10-18 08:31:03 -0600

Seen: 20 times

Last updated: Oct 18 '07