Archived

This forum has been archived. Please start a new discussion on GitHub.

Unexpectedly large message

Hi,

I started having problems with Ice after a recent upgrade to 3.4.1. I am using Dynamic Ice, in C++, to transfer sequences of long strings. I am getting MemoryLimitException, but the size it reports seems much too large.

Here's what happens:
the client invokes an operation with 3 strings. The lengths are 4697, 25 and 500 characters, respectively. On the other side, when the server is trying to create an input stream with
Ice::InputStreamPtr in = Ice::createInputStream(ic, params);

It then throws this exception:

Ice runtime exception: ../../include/Ice/BasicStream.h:77: Ice::MemoryLimitException:
protocol error: memory limit exceeded:
requested 4237453388 bytes, maximum allowed is 1048576 bytes (see Ice.MessageSizeMax)


I would increase the message size limit, except that the requested size looks huge to me, and does not match the size of the message actually being transferred. This makes me very suspicions of what is actually happening. How could I debug this?

Myrosia

Comments

  • mes
    mes California
    Hi,

    The most likely reason for the MemoryLimitException is a mismatch between the format encoded by the sender and the format expected by the receiver. In this situation it's very easy to cause a MemoryLimitException because the input stream is attempting to translate some arbitrary bytes into a size value.

    However, without seeing the code, it's very difficult for us to provide any more help. If you're still having trouble, please also provide details about the versions of your operating system and compiler.

    Regards,
    Mark
  • Thanks, this actually helped. Should have been obvious to me from the start! I have changed some settings as part of upgrading Ice installation, one process wasn't picking up the new libraries. Once I figured that this could be a problem, I sorted that out and the problem appears to have gone away.

    Myrosia
  • Oops - I thought that the problem was solved, but it isn't :( This time the server kept throwing exceptions about receiving too large messages, but after I re-started it a couple of times, settled back to normal.

    Intermittent errors are a pain to fix, obviously. But this does not look to me like a compilation or version mismatch error now, given that re-starting solved it. What else should I be looking at?

    Both the client and the server are written in C/C++, compiled with gcc 4.5, and I am running OpenSuse 11.3 with all the latest patches.

    Any suggestions as to where to start looking would be very helpful.
  • mes
    mes California
    If you can create a small, self-contained example that demonstrates the problem, we'd be happy to take a look at it.

    Regards,
    Mark
  • Oops - accidentally sent my last response by email instead of forum. Here's the bit of information that went through email.

    Hi,

    My first step would probably be to add some code to the client, immediately after calling out->finished(), that creates an InputStream from the newly-generated blob and attempts to unmarshal the sequence that you just marshaled, using the same code as in the server.

    I'm assuming that your code also writes a count of the number of strings in the blob, even though your snippet didn't show it. That is:

    out->writeSize(inc);

    for (int i = 0; i < inc; i++) {
    out->write(inc);
    }

    Note that we only provide email support to customers that have a support contract. Please post any replies on our forum.

    Mark


    > On Wed, Dec 15, 2010 at 10:33 AM, Myrosia Dzikovska
    > Well, that's the problem - if I could make up a small self-contained
    > example that demonstrates the problem, I would have debugged it by
    > now.. But this is an intermittent error: as I said, it kept happening,
    > then I thought it was fixed with a recompile, and then it went away
    > again after a restart. I know that it is very difficult to give any
    > kind of tech support in this situation, too, so I wish I could be more helpful.
    >
    > At the moment, at a loss of how to even replicate the set of events (other > than, I starte the system and 1 times out of 10 it breaks). If I had some
    > idea about what the likely failure points are and what to
    > watch for, I might be able to make a better guess based on my code of
    > what the issue is. If one side is indeed encoding in a different
    > format than the other is receiving, is there a way to confirm this? If
    > I knew what sort of debug messages to put in, I would perhaps have more
    > useful info the next time the problem happens. So far, I tried to print out
    > the parameters just before they are written into the stream with "write"
    > operation, and those look OK.
    >
    > The code is:
    > std::vector<Ice::Byte> inParams, outParams;
    > Ice::createOutputStream(_ic);
    > for (int i = 0; i < inc; i++) {
    > cerr << "inParam " << i << " is " << in << "END" << " length "
    > << strlen(in) << endl;
    > out->write(in);
    > }
    > out->finished(inParams);
    > proxy->ice_invoke(op, Ice::Normal, inParams, outParams)
    >
    > where "in" is const char*[], and inc is passed in.
    >
    > I am seeing the contents of "in" printed out to standard error without
    > problems, so there isn't obvious corruption. Are there characteristics
    > of the inParams array that I could print out in order to see that it
    > is sensible before I pass it into ice_invoke?
    >
    > Myrosia
  • Mark, no, I don't do "writeSize". Am I supposed to? The dynamic invocation example in the online manual does not call writeSize, it just writes two parameters and calls out->finished(inParams)

    Am I supposed to do that with the latest version? If so, am I also supposed to read something on the server (which uses dynamic Ice as well)?
  • mes
    mes California
    You would need to call writeSize only if the number of arguments can vary from call to call, otherwise how would you know the number of strings to read in the server?

    Regards,
    Mark
  • mes
    mes California
    Ok, perhaps I misunderstood your original example. I thought your code was creating a sequence<string>, but I guess it's really just sending three strings as operation parameters. In this case, you don't need to call writeSize.

    Mark
  • Yes, it's sending the 3 separate strings as parameters, with the server knowing what to expect based on the operation name. I have verified the obvious things like the number of parameters is correct and known on both sides, etc., and in any case the exception in the server comes right at createInputStream call in the server, before any reading is done.

    I will try your suggestion with unmarshalling on the spot, before things go into ICE, and seeing if it works.

    Myrosia
  • mes
    mes California
    You can also set a breakpoint in the server in the throwMemoryLimitException function (Exception.cpp, line 38). It would be useful to see the function call stack trace at that point.

    Mark
  • OK, so I just run against the same problem with too large a message again. This time I had a (permanently inserted) debug in client which asked for unmarshalling the arguments immediately after writing them to a stream, just before they go into ice_invoke. It works just fine, but the server side still raises the memory limit exception once it gets there. I am trying to get a breakpoint in the server as you suggested, but having little luck so far: since the bug is intermittent, I can only do that by attaching gdb to a running process when it starts happening, and the pre-compiled Ice library from the repository doesn't seem to have the debug info attached. I am looking into recompiling it with the right debug flags on my machine...

    FWIW, this may be some sort of unititialized variable problem. Complete restarts tend to cure it. I am trying to double-check my server code for something like that, but not seeing anything immediate.

    Myrosia