IcePy, large sequences

huberthoegl

Hello,

we implemented a client/server image visualization program which represents
images as sequence<float>. The server is written in C++, the client in Python.
We face the problem, that after the images get transfered to the client and
are temporarily stored into a Python list, the memory footprint of the client
process increases drastically. E.g. for a 128 MByte large image more than 450 MByte
of virtual memory space get consumed. Our first naive approach to relax the
situtation was to copy the image to a more memory-efficient Python array and to free the large list of image data. But it seems, that the original list is not actually released
from memory. Has anybody an idea how IcePy can handle large sequences in a memory
efficient way?

Thank you!

Hubert

benoit

Hello Hubert and welcome to the forums! We'll be happy to answer your question once you set your signature. See [thread=1697]this thread[/thread] for more information on our support policy on the forums.

Cheers,
Benoit.

huberthoegl

MedNet repository access

Hello,

here is the MedNet repository, just for the case that somebody
wants to have a look at the code:

http://141.82.30.67/cgi-bin/viewcvs.cgi/?root=mednet

The memory consuming list is in file (just search for array.array):
http://141.82.30.67/cgi-bin/viewcvs.cgi/client/OrthoviewerFrame.py?root=mednet&rev=108&view=markup

Cheers,

Hubert

benoit

Hi Hubert,

We're not aware of any memory leaks with Ice and IcePy. How do you measure the memory consumption of your python application and how did you figure out that the memory wasn't properly released? (note that the size of the virtual memory allocated for the process isn't a good way to figure out how much memory the process actually use, see Michi's post here on this thread).

Cheers,
Benoit.

marc

There is also an FAQ here:

http://www.zeroc.com/faq/iceMemoryUsage.html

mes

Hi,

You've encountered a limitation of using Ice in Python: large sequences are slow and inefficient. I'm not surprised that your program is consuming a lot of memory, especially when you consider what's really happening:

• The Ice for C++ run time allocates a buffer to hold the request.
• A C++ vector<float> is allocated to hold the unmarshalled sequence.
• The Ice extension for Python iterates over the vector and adds elements to a Python tuple, one element at a time.

The first two steps are essentially unavoidable; it's the last step that we would like to optimize. Unfortunately, the Python C API doesn't provide an efficient way for us to do that. The "array" extension might seem like an obvious alternative, but it doesn't provide a C interface; we would essentially have to simulate using Python code to create and populate an instance of the array type. Furthermore, we'd also need to add support for indicating when a Slice sequence should be treated as an array instead of as a Python tuple (e.g., using Slice metadata).

Given these obstacles, we decided to postpone any attempts at optimizing large sequences. As far as we know, however, there are no memory leaks in the Ice for C++ run time or the Ice for Python extension.

If you have a commercial need for this optimization, please contact us at info@zeroc.com.

Take care,
- Mark

huberthoegl

Inefficient large IcePy lists

Hello Mark,

thank you for the detailed comment. By performance measurements
we came to the conclusion that transfering float sequences larger than
a few 100.000 entries is not recommended with IcePy. We try to circumvent
the problem by writing a Python extension module in C++ which encapsulates
the image data so that only a reference is passed to the Python layer.
This should work because we have no need to process the image data directly
with Python - we simply pass the image reference to another program wrapped
with Python (VTK).

I also don't think that there is a memory leak. Probably I was too vague in
my original post.

Cheers,

Hubert