Welcome to the new Gigaspaces XAP forum. To recover your account, please follow these instructions.

Ask Your Question
0

Loading Large Files In XAP

I have follow up questions on the approach mentioned here for handling large objects.

SpaceFile spaceFile = new SpaceFile(myfile.getName(),fileToBytes(myfileName));
gigaspace.write(spaceFile);
  1. Where does the above code run - on the client or the server node?
  2. What if the client/server node does not have enough memory to load the complete file in memory at once to convert in bytes? Assume the file is as big as 10 GB. The XAP cluster has 3 nodes with each having 5 GB memory.
  3. Can the loading of large files be done in parallel? I understand there is distributed task api recommended for rapid data loads that provides the ability to perform data loads faster but how does the above approach blend with that?

Kindly let me know your thoughts.

Thanks!

asked 2017-01-12 00:06:33 -0500

shahamit gravatar image
edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted
0
  1. The example given in your link is client code, but it could be changed to run on the server.
  2. I am not quite sure I understand your question, can you please describe your use case? You could design a data type that contains fragments of the file and store it across multiple partitions.
  3. Distributed task is designed to run a piece of code across all partitions and implements a scatter/gather pattern. https://docs.gigaspaces.com/xap121/ta...

answered 2017-05-05 09:29:31 -0500

Dixson Huie gravatar image
edit flag offensive delete link more
0

Few suggestions: - Break the file into multiple space objects and assemble these when reading. This will avoid loading the entire file into memory when writing / reading. - Leave the file on file system , but store its location (URL) within the space object and use a web server (running as a PU) to provide access to the file.

Shay

answered 2017-05-05 10:37:54 -0500

shay hassidim gravatar image
edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2017-01-12 00:06:33 -0500

Seen: 26 times

Last updated: May 05