IceInternal.BasicStream seems to hold on to objects

Venkat Seeth

Hi there,

Howdy. I'm profiling the application where in I see that IceInternal.BasicStream seems to hold on to Slice defined objects which are basically domain objects acting either as input to a service request or output from a request.

The chain is shown below. This seems to hold on to these objects even after a forced GC from the profiling tool.

- IceInternal.BasicStream (#ffffae51) [Java Frame]
- _readEncapsCache IceInternal.BasicStream$ReadEncaps (#00001242)
- unmarshaledMap java.util.TreeMap (#0000150f)
- root java.util.TreeMap$Entry (#00002336)
- value model.retrieve.Request (#00009c24)

- IceInternal.BasicStream (#ffffb985) [Java Frame]
- _readEncapsCache IceInternal.BasicStream$ReadEncaps (#00000385)
- unmarshaledMap java.util.TreeMap (#00001361)
- root java.util.TreeMap$Entry (#00000d41)
- value model.retrieve.Response (#0000275b)

Any thoughts are greatly appreciated.

Thanks,
Venkatesh

mes

In Ice 2.1.0 and prior versions, the Java and C# run times actually cache BasicStream objects and eventually reuse them, which is why your objects are living longer than you might expect. Once that BasicStream instance is reused, the internal maps it keeps are cleared and your objects should be eligible for GC.

In the next release, the Ice run time will no longer cache BasicStream objects, which means any Ice objects that are created during marshaling should be eligible for GC as soon as the request completes, assuming the application isn't holding references to them.

- Mark

Venkat Seeth

Thanks for the quick reply. It makes sense now.

But, If say a service is gettting multiple requests, say 10 requests, I observe that 10 object instances are cached by the BasicStream.

Is this expected behavior?

Thanks,
Venkatesh

mes

Sorry, I need to correct what I stated above: the next release removes some unnecessary caching in BasicStream. However, we will make sure that BasicStream's internal tables are cleared as soon as possible.

But, If say a service is gettting multiple requests, say 10 requests, I observe that 10 object instances are cached by the BasicStream.

Is this expected behavior?

The Ice run time creates a number of BasicStream objects. For example, each thread in the server's thread pool creates a BasicStream to hold an incoming request message. So if there are 10 threads in your thread pool and each has serviced an incoming request, then it is quite possible that there are 10 instances of an unmarshaled Ice object still being referenced.

Hope that helps,
- Mark

Venkat Seeth

Thanks for the clarification.

It means that the BasicStream cache is directly proportional to the number of Threads in the Server Thread Pool.

Will the cache be cleared when the threads are released into the thread pool?

Venkatesh

mes

Venkat Seeth wrote:

Will the cache be cleared when the threads are released into the thread pool?

Yes. As idle threads are reaped by the Ice run time, the BasicStream objects they hold are released.

- Mark

mes

mes wrote:

Yes. As idle threads are reaped by the Ice run time, the BasicStream objects they hold are released.

To clarify this statement, what I mean is that a thread pool thread does not release its BasicStream object until the thread terminates. However, in the next release the thread will ensure that the BasicStream object does not retain any references to unmarshaled objects after each request completes.

- Mark

Venkat Seeth

Thanks again for your clarification.

Can you throw some light on when the next version is going to be released.

This is going to affect if you have a large thread pool.

Venkatesh

Venkat Seeth

Hi Mark,

Howdy. I did some rudimentary load tests with about 700 requests on a server with the following config.

Version 2.1.0
Env: Win2K and Java 1.5

Ice.ThreadPool.Client.Size=30
Ice.ThreadPool.Server.Size=30

I see that there are about 122 instances of the model object sent as a parameter in request. This is more than the number of threads in the ThreadPool.

The surprising thing is the response objects created for each request are only 27, which is less than the number of threads in the ThreadPool.

There are many held on to by BasicStream as the Root in the GC path and many others held by Ice.Connection as the Root which finally has a reference to BasicStream.

Can you throw some light on this leak.

Thanks,
Venkatesh

mes

Hi,

Which tool are you using to examine the JVM state?

Have you tried running your test for a longer period of time to see if the number of objects steadily increases?

I'll look into this further.

Regarding the timing of the next release: We are in "release testing" mode now, so assuming there are no major problems discovered, I expect 2.1.1 will be released within the next two weeks.

Take care,
- Mark

Venkat Seeth

Which tool are you using to examine the JVM state?

YourKit Java Profiler 4.0.10

Have you tried running your test for a longer period of time to see if the number of objects steadily increases?

I ran the test last eveing and left the application running all night, came back this morning, forced a GC, 0 bytes collected and still see the same dangkling references.

I'll look into this further.

Thanks.

I expect 2.1.1 will be released within the next two weeks.

Thats super. I can live with this for 2 weeks but curious to know your findings.

Thanks,
Venkatesh

Venkat Seeth

Have you tried running your test for a longer period of time to see if the number of objects steadily increases?

I did a series of tests and I dont see that the number of objects steadily increases.

I did try reducing the number of Server Threads and the number of objects reduced too. I cannot see a pattern but it hovers around some steady number.

30 Server Threads - 120 Objects
10 Server Threads - 48 objects - stays around this though it seems to go down at times but comes back up to this number.

Hope this helps,
Venkatesh

mes

Hi Venkat,

Thanks for that.

These results are consistent with my expectations that objects are eventually released, but it would be better if they were released earlier. This will be addressed in the 2.1.1 release.

I would be interested in seeing the Slice definitions of your test case. If you don't want to post them here, feel free to send them to me by email.

Thanks,
- Mark