Archived

This forum has been archived. Please start a new discussion on GitHub.

C# Memory Leak

Hello,

we noticed a memory leak in our c# application using Ice. While we were looking deeper into the problem we experimented with a minimal application and different Ice versions (3.4, 3.5 and 3.6). Depending on the ice version we noticed different behavior. And strangely ONLY with 3.4 there is no memory leak.

This is the slice file of the minimal application:


module MemoryLeaker {

    sequence<byte> byteVec;

    struct leakyObject {
      int id;
      byteVec leakBytes;
    };

    interface Leaker {
        void leakMemory(int amount, leakyObject inObj, out leakyObject outObj);
    };
};



The server does the following: Everytime the method "leakMemory" is called 50 MB are allocated for the leakyObject and the object is returned.

The client contains a loop that creates a new ice connection and proxy and calls the "leakMemory" method on each iteration. Afterwards "destroy" on the communicator is called.
Calling ice initialize every iteration seems to cause the memory leak. We know that the memory leak does not occur if we would call initalize only once. However we can't do this in our production application.

With 3.4 there's no memory usage increase. With 3.5 client AND server's memory usage increases. With 3.6 only the server's memory usage increases.

Our current workaround is to stick with ice3.4. However we need to upgrade soon due to our changing development environment. E.g. there's no ice3.4 visual studio addin for VS2012 available.


You can find the sources of the test applications in the attachment.

Do you have any idea why the behavior changed between the different ice versions?


Best regards and thanks in advance,
Johanna

Comments

  • benoit
    benoit Rennes, France
    Hi Johanna,

    I can't reproduce the leak with 3.6 and C# executables built with VS2015. Which Visual Studio version do you use? Does the server ends up throwing an OutOfMemory exception when it can't allocate any more memory?

    One thing I noticed, there's a different memory usage behavior between the client and the server. The client releases quickly the memory whereas the server hangs on to it a bit longer. The server allocates close to 2GB of memory before the garbage collector starts collecting memory. When the GC kicks in, the server memory usually goes down back under 1GB of memory usage. I wasn't able to get an OutOfMemory exception so the memory appears to be properly released.

    Cheers,
    Benoit.
  • Hi Benoit,

    first of all: thank you for looking into it!

    We tried on different machines and with different Visual Studio versions (Windows 10 and Visual Studio 2015, Windows 7 and Visual Studio 2012).
    Thank you for pointing out the server behavior. We did not notice that the memory usage can go down before. However we also reproduced this behavior now.
    On Windows 10, VS2015 it seems the GC cleans up after an undefined period of time (at least sometimes...), while on windows 7, VS2012 the GC behaves somewhat different and the cleanup seems to be triggered only when a huge amout of memory (5GB+) has been used (no matter how long we wait). Even after cleaning up, the memory usage in most cases still remains about 1,5 - 2,5 GB.

    Nevertheless we still have an issue because our application needs to run on 32bit systems as well and is part of a larger application which also consumes a lot of memory.
    Therefore in our case we can get out of memory exceptions.

    As you can see in the attached screenshot the memory usage with Ice 3.6 can get rather high. With Ice 3.4 we don't get this behavior of the server at all: Memory usage of client and server never exceed 1 gb and the memory is released rapidly (usually remains about 300 mb).

    Best regards,

    Johanna
  • benoit
    benoit Rennes, France
    Which Visual Studio / .NET version do you use with Ice 3.4? With VS2012 and Windows 8, I'm seeing a behavior similar to what I described in my first post (with VS2015/Windows 10).

    The only major difference I can think of between 3.4 and 3.5/3.6 is the use of the new asynchronous socket APIs (i.e.: use of SendAsync instead of BeginSend). You could try re-compiling Ice 3.6 to use the old socket API and see if this makes a difference (comment out #define ICE_SOCKET_ASYNC_API in Ice-3.6.1/csharp/src/Ice/StreamSocket.cs line 15 and recompile Ice for C#).

    How much memory does your machine have? Did you actually get OutOfMemory exceptions on your 32 bits systems?

    With .NET one issue which can occur when you allocate large objects is large object heap fragmentation. Recent .NET versions improve the memory management of large object heaps (and 4.5.1 also allows compacting it), so I recommend using a recent .NET version. See this article for instance of large object compaction: https://www.simple-talk.com/dotnet/.net-framework/large-object-heap-compaction-should-you-use-it/

    Cheers,
    Benoit.
  • Hi Benoit,

    The application was build with .NET framework 4 as target and we were running it on machines that had .NET 4.5.1 and 4.6 installed.
    For ice3.4 we used Visual Studio 2012, Win7, .NET framwork 4.5.1 (target 4).

    We recompiled Ice3.6.1 according to your advice. But the problem still occured.

    Our development machines are actually 64-Bit machines with a lot of memory (8-16 GB). However we could reproduce the out of memory exception by compiling the minimal example as x86 application (see attached file).

    Do you have any further ideas?

    Cheers,
    Johanna
  • benoit
    benoit Rennes, France
    Hi,

    Can you try setting Ice.CacheMessageBuffers=0 in the server configuration?

    I can reproduce the issue with 3.6 when running the .NET server with a 32 bits process. The server eventually throws an OutOfMemory exception if the requests are sent quickly enough (the GC seems to be able to reclaim the memory if the requests are sent slowly however). Disabling the message caching on the server appears to fix the issue with 3.6. It's not clear to me what is the difference between 3.4 and 3.6, message caching is also enabled with 3.4.

    Cheers,
    Benoit.
  • Hi Benoit,

    setting Ice.CacheMessageBuffers=0 in the server seems to solve the problem. But only if I comment out #define ICE_SOCKET_ASYNC_API in StreamSockete.cs. Otherwise I still have the issue with Ice 3.6.

    Using Ice 3.5 and setting Ice.CacheMessageBuffers=0 in the server does not change anything: Memory usage increases in both - client and server. But for Ice 3.5 I did not recompile Ice.dll.

    Cheers,
    Johanna
  • benoit
    benoit Rennes, France
    Hi Johanna,

    Can you try to apply the patch attached to this post on a fresh 3.6.1 distribution and see if it fixes the issue?

    To apply the patch:
    $ cd Ice-3.6.1
    $ patch -p1 < patch.cs.txt
    

    Cheers,
    Benoit.
  • Hi Benoit,

    this patch seems to fix the issue for ice 3.6.1.

    Will this fix be included in the next Ice bugfix release? If that's the case, will it also be backported to Ice 3.5?

    Can you explain the actual reason for the memory leak? And is this the same issue in Ice3.5 (leak in client and server)?

    Cheers,
    Johanna
  • benoit
    benoit Rennes, France
    Hi Johanna,

    I don't think it's a memory leak. As you can see in the patch, we actually don't fix any leaks but just release references sooner. Your test case also works fine for me as long as the allocations are done at a slow pace.

    The patch helps the garbage collector collecting large objects sooner than later. I can't say exactly why the garbage collector fails to reclaim the memory without this fix, it could be a timing issue (garbage collector isn't fast enough to reclaim memory for large objects) or the fact that the fix helps reducing LOH fragmentation. Figuring this out would require to analyze the .NET heaps with a memory profiler.

    This fix will be included in the next Ice minor release.

    Cheers,
    Benoit.