Crash on exit, after upgrade to Ice 3.2

vanco

I just upgraded from Ice 3.1.1 to Ice 3.2.1. After the upgrade, one of my servers crashes whenever I try to shut it down. The shutdown is through icegridadmin (server stop <server name>).

Here's the crash stack:

*** glibc detected *** /tmp/myexe: double free or corruption (fasttop): 0x00000000007c9400 ***
======= Backtrace: =========
/lib/libc.so.6[0x2afe7f20faad]
/lib/libc.so.6(cfree+0x76)[0x2afe7f211796]
/usr/lib/libIce.so.32(_ZNSt8_Rb_treeISsSt4pairIKSsN11IceInternal6HandleIN3Ice6ObjectEEEESt10_Select1stIS7_ESt4lessISsESaIS7_EE8_M_eraseEPSt13_Rb_tree_nodeIS7_E+0x59)[0x2afe7e579269]
/usr/lib/libIce.so.32(_ZNSt8_Rb_treeISsSt4pairIKSsN11IceInternal6HandleIN3Ice6ObjectEEEESt10_Select1stIS7_ESt4lessISsESaIS7_EE8_M_eraseEPSt13_Rb_tree_nodeIS7_E+0x30)[0x2afe7e579240]
/usr/lib/libIce.so.32(_ZNSt8_Rb_treeIN3Ice8IdentityESt4pairIKS1_St3mapISsN11IceInternal6HandleINS0_6ObjectEEESt4lessISsESaIS2_IKSsS8_EEEESt10_Select1stISF_ES9_IS1_ESaISF_EE8_M_eraseEPSt13_Rb_tree_nodeISF_E+0x36)[0x2afe7e579986]
/usr/lib/libIce.so.32(_ZN11IceInternal14ServantManager7destroyEv+0x34b)[0x2afe7e57799b]
/usr/lib/libIce.so.32(_ZN3Ice14ObjectAdapterI7destroyEv+0x1ab)[0x2afe7e51c32b]
/usr/lib/libIce.so.32(_ZSt8for_eachISt17_Rb_tree_iteratorISt4pairIKSsN7IceUtil6HandleIN3Ice14ObjectAdapterIEEEEEN15IceUtilInternal16SecondVoidMemFunIS2_S6_S7_EEET0_T_SE_SD_+0x7d)[0x2afe7e51bbfd]
/usr/lib/libIce.so.32(_ZN11IceInternal20ObjectAdapterFactory7destroyEv+0xbc)[0x2afe7e519cac]
/usr/lib/libIce.so.32(_ZN11IceInternal8Instance7destroyEv+0x39f)[0x2afe7e4e431f]
/usr/lib/libIce.so.32(_ZN3Ice13CommunicatorI7destroyEv+0x28)[0x2afe7e4a14f8]
/usr/lib/libIce.so.32(_ZN3Ice11Application4mainEiPPcRKNS_18InitializationDataE+0x3cc)[0x2afe7e4916fc]
/usr/lib/libIce.so.32(_ZN3Ice11Application4mainEiPPc+0x43)[0x2afe7e491d03]
/tmp/myexe[0x414cb0]
/lib/libc.so.6(__libc_start_main+0xf4)[0x2afe7f1c0b74]
/tmp/myexe(_ZN12IceDelegateD3Ice6Object7ice_isAERKSsPKSt3mapISsSsSt4lessISsESaISt4pairIS2_SsEEE+0x69)[0x414609]

The only extraordinary thing about the server is that it attempts to implement two interfaces. The management of all the memory is done through smart pointers:

class MyApp : public Ice::Application
{
virtual int run(int argc, char* argv[]) {
//...initialization cut out
Ice::CommunicatorPtr ic = this->communicator();

// implementation of the two interfaces. Doesn't directly derive from anything
boost::shared_ptr<MyImpl> myImpl(new MyImpl(ic, <some args>));

// First interface, stores myImpl as a shared_ptr
MyI1Ptr p1 = new MyI1(myImpl);
// Second interface, stores myImpl as a shared_ptr
MyI2Ptr p2 = new MyI2(myImpl);

// Make the facets and activate them
Ice::ObjectPrx prx = adapter->add(p1, ic->stringToIdentity("myname"));
adapter->addFacet(p2, prx->ice_getIdentity(), "facetname");
adapter->activate();

// OK, the server is now up and running. Wait for it to shutdown.
ic->waitForShutdown();
}
};

Any help would be appreciated...

<Edit - added valgrind output>
For kicks I ran valgrind on the executable. The output is full of 'Invalid read of size 8' and 'Address is 24 bytes inside a block of size 48 free'd'. I've attached two of the pertinent reports, all caused on application exit.

matthew

The stack trace indicates that its crashing while releasing the servants, so I would point my finger squarely at the memory management of that shared object. Without more details, or a code example, its hard to tell you exactly what is going wrong however. You could try changing the use of boost:shared_ptr to IceUtil::Handle, and derive MyImpl for IceUtil::Shared and see if that fixes your problem.

vanco

Didn't work...

Sorry - changing boost::shared_ptr to IceUtil::Handle and deriving MyImpl from IceUtil::Shared didn't work. That makes sense - since both just do reference counting.

I'll try to cook up a simple example that replicates my problem. In the meantime, any suggestions are welcome.

vanco

One more hint

Forgot to mention this: the crash on exit only happens once at least one of the servants have been invoked at least once. When I stop the server immediately after it starts up, without executing any of its functions, I see no crash on exit.

vanco

no luck reproducing on a small testcase

No luck reproducing the problem on a small testcase - the small testcase works like a charm.
Instead, I tried going top-down - started removing code from my server, until it started working. Interestingly, it started working when I removed some completely innocuous line of code (a=b*c, where all three are floating points). It almost looks like either an uninitialized variable, or a race condition, most likely in the guts of Ice (since valgrind cannot find it in my code).
Building the debug version of Ice to see if that will show anything...

vanco

bug was in user code

Managed to track the bug to user code. There was a bug in the destruction sequence of the MyImpl object.

Curiously, though - this bug was never triggered in 3.1.1. It's almost like the MyImpl object was never destroyed in 3.1.1... If it were destroyed, this bug would have shown up immediately.