Archived

This forum has been archived. Please start a new discussion on GitHub.

Process size increase using proxy calls

Hello again,

We have noticed that when we make many method calls on a proxy -- in this case, C++ client calling a Java server -- our process space increases dramatically and does not decrease. We are sending a double and a long as arguments and have stubbed out the server implementation -- it does nothing. When we comment out our client call to the proxy, our space does not increase.

There is a known issue in Java where ObjectOutputStreams do not reclaim space of objects that are serialized and sent across the wire. The application code must periodically call the reset() method of this class to reclaim space of objects sent that are no longer referenced and can be garbage collected. I bring this up because I'm wondering if there is a similar issue using IceInternal::BasicStream and this is causing our problem. In any case, does anyone know what might be happening and have workarounds?

Thanks,

Brian

Comments

  • When you say the process size increases, is this in the C++ client or the Java server? Could you post a test case that shows the problem? Without this, it will be difficult to work out what is going wrong...

    Thanks,

    Michi.
  • We see the size increase in the Java server.

    Before we take the time to write a test case, just wondering if you might know of an issue like the one I pointed out in Java that could happen in Ice? As far as you know, should this problem not arise at all?
  • marc
    marc Florida
    We are not aware of any such problems. Ice for Java is used in many applications that run for many weeks without restarts. So far we didn't observe the behavior you describe above.

    Which JVM version are you using?
  • mes
    mes California
    Hi Brian,

    I did a quick experiment with a modified version of the "hello" demo in which the client makes 10000 invocations of the sayHello operation. Before the client starts, the RSS value of the server's JVM is 10500. After a couple executions of the client, the RSS value grows to about 35500, where it levels off and remains essentially constant for several more client executions (I've seen the server's RSS grow larger than 35500, but it later returns to that level).

    This was done on Linux with JDK 1.4.2 and Ice 1.4.0.

    Have you checked to see if the process growth of your Java server eventually levels off?

    Take care,
    - Mark
  • Mark,

    Thanks for running that test. Yes, we see the size grow and then it levels off. The problem is, that is grows from about 100Meg to between 400Meg and 1Gig before leveling off. The huge size increase then causes our server process to be swapped out for many seconds at a time on a continual basisi so we do not have continuous flow of information between client and server. So yes, that is the behavior we are seeing.

    We are using JDK version 1.4.2_01 on Unix and 1.4.2_03 on Windows.

    So would you call this an "issue". Is there some workaround we can employ to decrease our process space periodially so that we will not see this swapping problem ala the Java soluation I previously mentioned?

    Thanks again,

    Brian
  • mes
    mes California
    On which platform is your Java server running? If it's Unix, which Unix? If it's both, do you see the same behavior on both Unix and Windows?

    Thanks,
    - Mark
  • That was a good question because we tested the server on both Solaris and Windows and the problem does NOT happen in Windows.

    We are using Solaris SunOS 5.8, version Generic_108528-23. Would this be a know problem using this version of Unix? Is there an OS patch you know of?

    Thanks,

    Brian
  • benoit
    benoit Rennes, France
    I'm not aware of any OS patch for this problem, perhaps one of my colleague is.

    By default the maximum heap size of a Java process is 64 MB (and 128 MB if you use -server). Are you modifying the heap size of the JVM with the -Xmx command line option? Theoritically, your JVM process shouldn't use much more memory than the configured maximum heap size.

    Benoit.
  • Benoit,

    No, we are not using the Xmx option.

    Also, I just noticed another problem. Even when we move the Java server to Windows to fix the space issue, making method calls to the server proxy in our C++ client on Unix causes the CPU utilization to go to 30% ( as viewed using the Unix "top" command). Normally, as shown using our previous JNI middleware implementation, the utilization hits the high 90s. Seems as though the ICE implementation must be blocking on I/O somewhere. I tried setting the Ice.ThreadPool.Server.Size and <adapter-name>.ThreadPool.Size property for the server to 1000, but this had no effect.

    So it seems like there is a related bug in Unix that is causing the space to increase is also causing our CPU utilization to go way down. Unfortunately, our C++ client is not cross-platform so we do not have the option of working around the problem by running the client under Windows.

    Note that this utilization problem happens even if the server implementation of the client proxy call just returns -- i.e. does nothing. Like the space issue, it seems as though the actual marshalling/unmarshalling of information between client and server on Unix is causing the problem.

    FYI,
    Brian
  • benoit
    benoit Rennes, France
    Hi Brian,
    Originally posted by brian
    Benoit,

    No, we are not using the Xmx option.

    I don't understand why the JVM memory is growing to 1GB then, it shouldn't go above the maximum heap size configuration (which is 64 MB if you don't override it with -Xmx). Sounds like a problem with the JVM, you could try to monitor the garbage collection of the JVM to see if it gives you any hints (with -Xloggc:<file>).

    Also, I just noticed another problem. Even when we move the Java server to Windows to fix the space issue, making method calls to the server proxy in our C++ client on Unix causes the CPU utilization to go to 30% ( as viewed using the Unix "top" command). Normally, as shown using our previous JNI middleware implementation, the utilization hits the high 90s. Seems as though the ICE implementation must be blocking on I/O somewhere. I tried setting the Ice.ThreadPool.Server.Size and <adapter-name>.ThreadPool.Size property for the server to 1000, but this had no effect.

    Increasing the initial size of the thread pool will increase the number of threads available to dispatch incoming requests. Why do you think it would improve the CPU utilization?

    So it seems like there is a related bug in Unix that is causing the space to increase is also causing our CPU utilization to go way down. Unfortunately, our C++ client is not cross-platform so we do not have the option of working around the problem by running the client under Windows.

    Note that this utilization problem happens even if the server implementation of the client proxy call just returns -- i.e. does nothing. Like the space issue, it seems as though the actual marshalling/unmarshalling of information between client and server on Unix is causing the problem.

    FYI,
    Brian

    I'm afraid it's very difficult to comment without knowing more about your application. The best would really be a small example that would reproduce the problem. Perhaps you could try to reproduce the problem with one of the Ice demos?

    Benoit.
  • Thanks for the reply Benoit,

    We tried increasing the thread pool size because we pound our server with lots of messages simultaneously and thought that maybe, after reading section 15.3, the problem was that the client was blocking because it had to wait for a single thread to finish processing before sending another message. Anyway, it had no effect and our problem still exists.

    Don't get confused by the 1Gig statement, the main point is that the memory increases to some amount and then levels off as the example that Mark ran from the test suite points out. I would think that same example could be used to show the CPU problem as well.

    The main point here is that, at this point, it appears that the Unix side of marshalling/unmarshalling has a bug in it that does not happen in Windows. Mark was able to recreate the space problem on Linux with a simple example. Now, I'm really curious to see if the same behavior happens in Windows for you as well. The bug is probably not in ICE, but in Unix OS code (and maybe Linux), but it is curious that it also happened in Linux for you. Realize that there might be nothing you guys can do to correct this, but can't believe this hasn't come up before with your other Unix and Linux customers. Perhaps they have not been watching their resources as closely as we have and do not have data rates we do, don't know.

    Not saying that the problem is definitely not ours, but considering we do not see it under Windows and that Mark was able to recreate it, it does not appear that way at this point. Having said that, we would be happy to eat crow if we could fix the space and utilization problems.

    We have invested quite a bit of time rewritting our software that previously used JNI to use ICE. We love using ICE (IceBox, IceStorm, IcePack, Freeze), but this might be a show-stopper for us and cause us to go back to using JNI. We are going to investigate any patches we might be missing from Sun that might fix this problem, but at the same time would appreciate ANY advice that any of you have.

    We are not adverse to putting the time in to create an example, it just seems that if Mark has already recreated the problem via his example, that creating one on our end might be redundant.

    Mark, would love to hear your comments on the situation as well if you have the time please.

    Thanks again,

    Brian
  • mes
    mes California
    Brian,

    I wouldn't consider the behavior I witnessed to necessarily indicate a problem. The size of the JVM process can reasonably be expected to increase as the Ice server allocates resources. In my tests, the size of the server's JVM process leveled off at around 35MB, which is obviously much different than the 1GB size you are reporting.

    A decrease in CPU utilization isn't necessarily bad, unless it is a symptom of other problems. When comparing the Ice implementation to the JNI implementation, are you seeing a decrease in throughput? In other words, were you able to make more requests per second using JNI than using Ice?

    What exactly is your C++ client doing? Does it make continuous requests in a tight loop? Are requests made from one thread, or from many threads? Are all requests made on the same object, or on different objects? What strategy did you end up using to send your data (manually encoding to a byte sequence or the class structure we discussed)?

    - Mark
  • benoit
    benoit Rennes, France
    Originally posted by brian
    Thanks for the reply Benoit,

    We tried increasing the thread pool size because we pound our server with lots of messages simultaneously and thought that maybe, after reading section 15.3, the problem was that the client was blocking because it had to wait for a single thread to finish processing before sending another message. Anyway, it had no effect and our problem still exists.

    Don't get confused by the 1Gig statement, the main point is that the memory increases to some amount and then levels off as the example that Mark ran from the test suite points out. I would think that same example could be used to show the CPU problem as well.

    It's expected for a Java process to grow to a certain level and then stay at this level. The JVM allocates memory in chunks up to a certain limit (fixed by the maximum heap size, the -Xmx option). It then uses this memory for the application allocations. If you're not familiar with how the JVM garbage collection works, here's some useful links:

    http://developers.sun.com/techtopics/mobility/midp/articles/garbage/#1
    http://java.sun.com/docs/hotspot/gc1.4.2/

    In your case, it doesn't seem to be normal that the JVM size grows beyond the limit of the of the maximum heap size (it didn't in Mark's case, it stayed below the 64MB default limit).

    The main point here is that, at this point, it appears that the Unix side of marshalling/unmarshalling has a bug in it that does not happen in Windows. Mark was able to recreate the space problem on Linux with a simple example. Now, I'm really curious to see if the same behavior happens in Windows for you as well. The bug is probably not in ICE, but in Unix OS code (and maybe Linux), but it is curious that it also happened in Linux for you. Realize that there might be nothing you guys can do to correct this, but can't believe this hasn't come up before with your other Unix and Linux customers. Perhaps they have not been watching their resources as closely as we have and do not have data rates we do, don't know.

    We are not aware of any such problems on Linux. Ice and Ice for Java are deployed in a rather large scale application where Java servers are running for several days without any interruptions. The Java servers are started with specific memory limits (e.g.: with -Xmx256m). They don't consume more memory than this limit. The Java garbage collector correctly collects the unused memory indicating that there's no memory leaks (you can monitor the GC with the -Xloggc).

    Not saying that the problem is definitely not ours, but considering we do not see it under Windows and that Mark was able to recreate it, it does not appear that way at this point. Having said that, we would be happy to eat crow if we could fix the space and utilization problems.

    We have invested quite a bit of time rewritting our software that previously used JNI to use ICE. We love using ICE (IceBox, IceStorm, IcePack, Freeze), but this might be a show-stopper for us and cause us to go back to using JNI. We are going to investigate any patches we might be missing from Sun that might fix this problem, but at the same time would appreciate ANY advice that any of you have.

    I would recommend to first try to figure out why your Java process is growing so much. Looking into what the garbage collector is doing might give you some hints. We would be happy to investigate this further if we could reproduce it as well, so anything you can send us to reproduce it would help.

    We are not adverse to putting the time in to create an example, it just seems that if Mark has already recreated the problem via his example, that creating one on our end might be redundant.

    Mark, would love to hear your comments on the situation as well if you have the time please.

    Thanks again,

    Brian

    Benoit.
  • Thanks again guys for your thoughtful responses. We will do some more investigation to try to answer all of your good questions as best as we can and get back to you. I/we apprecate your help.

    For now, I just want to clarify that our Java server is started by IcePack using IceBox on either Windows or Solaris. Don't know if that gives you more insight, but we are not setting any command line options -- e.g. Xmx -- on the JVM using the XML configuration.

    Also, I understand the point you are making about JVM size not growning past a certain point, but the interesting thing is that again, the space problem does not occur running on Windows. I am using a slightly older version of Java on Solaris, but wouldn't think that would be the problem. On Windows, any space increased caused by the message sending gets garbage collected so that it does not go up appreciably. If our same exact software is running on Windows without this problem, any ideas why that might be the case?


    Brian
  • Thanks again guys for your thoughtful responses. We will do some more investigation to try to answer all of your good questions as best as we can and get back to you. I/we apprecate your help.

    For now, I just want to clarify that our Java server is started by IcePack using IceBox on either Windows or Solaris. Don't know if that gives you more insight, but we are not setting any command line options -- e.g. Xmx -- on the JVM using the XML configuration.

    Also, I understand the point you are making about JVM size not growning past a certain point, but the interesting thing is that again, the space problem does not occur running on Windows. I am using a slightly older version of Java on Solaris, but wouldn't think that would be the problem. On Windows, any space increased caused by the message sending gets garbage collected so that it does not go up appreciably. If our same exact software is running on Windows without this problem, any ideas why that might be the case?


    Brian
  • benoit
    benoit Rennes, France
    No, I'm not sure why the same software would behave differently wrt the memory allocation on Windows or Solaris. It shouldn't be the case, Java is supposed to be platform independent ;) I would try to compare the output of the garbage collection logs on Windows and Solaris to try to see if the GC behaves differently.

    To enable the logging of the garbage collector you can add the following in your IceBox or Java server deployment descriptor:
      <vm-options>
         <vm-option>-Xloggc:/tmp/gc.log</vm-option>
      </vm-options>
    

    This will log the GC in the /tmp/gc.log. You could also try to explicitly set the -Xmx option to see if that makes a difference.

    Benoit.