Archived

This forum has been archived. Please start a new discussion on GitHub.

Question about sending large data.

Dear Sir,

These days, we are suffering with sending large data by ICE.
Haha...., maybe we are runing to the wrong direction, therefore, we call for help.

Here is problem: we define sendBytes(...) to send byte sequence.

We want to send 50MBytes one shot. Ice report LimitedMemory Exception. We check that default MessageSize is 1MB in ICE, and set Ice.MessageSizeMax = 60000, it works. And we check the memory usage.

It shows that 105MB was used. Why?

My analysis is that our own data stucture uses 50MB, and Ice protocol buffer uses 50MB, am I right?

More importantly, Ice can only send fixed message size(restricted by Ice.MessageSizeMax property). Even we could dynamically set this property to fullfill the actual sending data size. But it's quite odd.
As our initial expectation, ICE would deal with this matter by stream oriented method. However, it's not. That means we need slice our own large data if we set a fix message size.

My question is: the following code:

for(i=0; i< n_slices; ++i)
{
getSlice(seqByte& s);
sendBytes(s);
}

We assure that the Ice invoke would be in order. But Ice uses thread pooling, we wonder whether the datas reached Server side are in order.

Good Luck!

Comments

  • benoit
    benoit Rennes, France
    JaneShang wrote:
    Dear Sir,
    It shows that 105MB was used. Why?

    My analysis is that our own data stucture uses 50MB, and Ice protocol buffer uses 50MB, am I right?

    Yes, that's right. Ice will copy your data structure in its internal buffers.
    More importantly, Ice can only send fixed message size(restricted by Ice.MessageSizeMax property). Even we could dynamically set this property to fullfill the actual sending data size. But it's quite odd.

    You can set this property to a very high value if you don't want in practice to have a limit. However, it's not recommended to transfer very large amounts of data in one call, see [thread=946]this thread[/thread] for more information.
    As our initial expectation, ICE would deal with this matter by stream oriented method. However, it's not. That means we need slice our own large data if we set a fix message size.

    My question is: the following code:

    for(i=0; i< n_slices; ++i)
    {
    getSlice(seqByte& s);
    sendBytes(s);
    }

    We assure that the Ice invoke would be in order. But Ice uses thread pooling, we wonder whether the datas reached Server side are in order.

    Good Luck!

    The data will be in order as long as you're using twoway invocations (the sendBytes call in your client will only return once the server has dispatched the invocation).

    Benoit.
  • One more thought: if you want to transfer large amounts of data, IcePatch2 may be just what you want. Check out the manual for more info.

    Cheers,

    Michi.
  • michi wrote:
    One more thought: if you want to transfer large amounts of data, IcePatch2 may be just what you want. Check out the manual for more info.

    Cheers,

    Michi.

    i find IcePatch2 very great easy to use, but it has very much limitation: it's great in patching a lot of files automatically, but not very good in transmitting certain specified files, i think maybe an ICEPatch API is needed!