Ice 3.5.1: Patch to fix extremely poor IcePy stringify performance

grembo

IcePy's stringify functions uses IceUtilInternal::Output to convert Ice data types such as struct or exception to strings. I assume that it was never meant to process large amounts of data, since it's extremely slow.

I stumbled over this when a Python process caught an exception containing one 1MB sized string member. The exception was implicitly converted to a string by an exception handler deep inside of the stack. The symptom was a python process running at 100% CPU that couldn't handle any application requests. As such implicit conversions are not uncommon in python I consider this to be extremely suboptimal.

Example:

cat >Test.ice <<EOF
module Test
{
  struct TestStruct
  {
     string message;
  };
};
EOF

cat >test.py <<EOF
import Ice
Ice.loadSlice("Test.ice")
import Test

str(Test.TestStruct("1" * 1000 * 1000)) # 1 million characters
EOF

time python test.py

real    1m21.264s
user    1m19.846s
sys     0m0.032s

The patches replaces IceUtilInternal::Output in IcePy with a local helper class called PrintHelper . It received some basic testing, you might want to double check yourself just to be sure.

Performance after applying the patch:

time python test.py

real    0m0.059s
user    0m0.024s
sys     0m0.033s

The patch assumes that you already applied https://www.zeroc.com/forums/patches/6183-patch-3-ice-3-5-1-marshaling-bug-nested-optionals.html. If you didn't, apply that one first.

To apply the attached source patch to a patched Ice 3.5.1 source distribution:

cd Ice-3.5.1
patch -p1 </tmp/ice351_icepy_stringify.txt