Segmentation Fault in Garbage Collection

MirkoWaechter

Hello,

we have occasionally seg faults in the garbage collection of Ice:

Error: Segmentation Fault

Backtrace:

Backtrace:
1 0x7fe0ba2c1b39 armarx::Application::HandlerFault(int) + 185
2 0x7fe0b66b9150 /lib/x86_64-linux-gnu/libc.so.6(+0x36150) [0x7fe0b66b9150]
3 0x7fe0b8cb3502 memoryx::EntityAttributeBase::__gcReachable(std::map<IceInternal::GCShared*, int, std::less<IceInternal::GCShared*>, std::allocator<std::pair<IceInternal::GCShared* const, int> > >&) const + 178
4 0x7fe0bb2ceb3e IceInternal::GC::collectGarbage() + 478
5 0x7fe0bb2cf10e IceInternal::GC::run() + 446
6 0x7fe0baf6d96f /usr/lib/libIceUtil.so.34(+0x2f96f) [0x7fe0baf6d96f]
7 0x7fe0b7c94e9a /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fe0b7c94e9a]
8 0x7fe0b677638d clone + 109

The structure of the classes that are involved is:
Entity has a list of EntityAttributes and EntityWrappers.
EntityWrapper has an IceHandle to the Entity it belongs to (so a cycle and a case for the Garbage Collection).
When Cloning Entities, I first create an empty instance, assign it to a Handle and then copy the values inside, so that the ref count is > 1 and the GC cannot collect it during copying of the EntityWrappers.

The slice interface definitions (without functions) look like this:

	interface Serializable 
	{
		["cpp:const"]
		void serialize(ObjectSerializerBase serializer);
		void deserialize(ObjectSerializerBase serializer);
	};

    class AttributeBase implements armarx::Serializable
    {

        ["protected"]
        string name;
    };
    
    class EntityAttributeBase extends AttributeBase
    {


        ["protected"]
        AttributeElementList values;
    };

    dictionary<string, EntityAttributeBase> AttributeMap;
 class EntityBase implements armarx::Serializable
    {


        ["protected"]
        string id;

        ["protected"]
        string name;

        ["protected"]
        AttributeMap attributes;

        ["protected"]
        [B]AbstractEntityWrapperBaseList wrappers;[/B] // <- causes cycle
    };

    class AbstractEntityWrapperBase
    {
        ["protected"]
        [B]EntityBase entity;[/B]// <- causes cycle
    };

I dont really understand why the garbage collection is doing something with the EntityAttributeBase, since there cannot be any cycle with this class.

All instances of any of these types are stored in IceHandles.

So, are there any known bugs with the GC?
Is the instance not locked while examined by the GC?
Why does it examine the EntityAttributeBase, although this class cannot have any cycles?

We are using Ice 3.4.2, Ubuntu 12.04 and gcc 4.6.



thanks
Mirko

MirkoWaechter

After looking into the Garbage Collection, I see that the GC is iterating over the member containers. What happens when I change the member containers while the GC is running? This is doomed to fail I guess.
The mutex of the GC is inaccesible.
How is this supposed to work?

cheers
Mirko

MirkoWaechter

It just crashed with debugger attached:
It is as suspected. The GC is running while clearing the member container.

The stacktraces of both threads:

0 memoryx::EntityAttributeBase::__gcReachable EntityBase.cpp 5144 0x7f3a95eb9502
1 IceInternal::GC::collectGarbage GC.cpp 279 0x7f3a984d5b3e
2 IceInternal::GC::run GC.cpp 192 0x7f3a984d610e
3 startHook Thread.cpp 413 0x7f3a9817496f
4 start_thread pthread_create.c 308 0x7f3a94e9ae9a
5 clone clone.S 112 0x7f3a9397c38d
6 ??

0 __lll_lock_wait lowlevellock.S 132 0x7f3a94ea189c
1 _L_lock_903 /lib/x86_64-linux-gnu/libpthread.so.0 0x7f3a94e9d080
2 __pthread_mutex_lock pthread_mutex_lock.c 82 0x7f3a94e9cf19
3 IceUtil::RecMutex::lock RecMutex.cpp 159 0x7f3a98170dcd
4 MutexPtrLock MutexPtrLock.h 30 0x7f3a984d5683
5 IceInternal::GCShared::__decRef GC.cpp 101 0x7f3a984d5683
6 IceInternal::Handle<armarx::VariantBase>::~Handle Handle.h 106 0x429472
7 memoryx::AttributeElement::~AttributeElement EntityBase.h 182 0x7f3a96bbaf67
8 std::_Destroy<memoryx::AttributeElement> stl_construct.h 94 0x7f3a96c0c98c
9 std::_Destroy_aux<false>::__destroy<memoryx::AttributeElement*> stl_construct.h 104 0x7f3a96c04eea
10 std::_Destroy<memoryx::AttributeElement*> stl_construct.h 127 0x7f3a96bf94b1
11 std::_Destroy<memoryx::AttributeElement*, memoryx::AttributeElement> stl_construct.h 153 0x7f3a96be8ae7
12 std::vector<memoryx::AttributeElement>::_M_erase_at_end stl_vector.h 1255 0x7f3a967ccdec
13 std::vector<memoryx::AttributeElement>::clear stl_vector.h 1040 0x7f3a967cc04a
14 memoryx::EntityAttribute::clear EntityAttribute.cpp 209 0x7f3a967c8fb1
15 memoryx::EntityAttribute::setValueWithUncertainty EntityAttribute.cpp 96 0x7f3a967c8148
16 memoryx::ObjectInstance::setPositionUncertainty ObjectInstance.cpp 111 0x7f3a9229e50e
17 memoryx::KalmanFilterFusion::fuseEntity KalmanFilterFusion.h 141 0x7f3a96bd6f74
18 memoryx::WorkingMemoryEntitySegment<memoryx::ObjectInstance>::updateEntity WorkingMemoryEntitySegment.h 163 0x7f3a96fe7f42
19 memoryx::WorkingMemoryEntitySegment<memoryx::ObjectInstance>::updateEntity WorkingMemoryEntitySegment.h 129 0x7f3a92301cfb
20 memoryx::ObjectLocalizationMemoryUpdater::localizationFinished ObjectLocalizationMemoryUpdater.cpp 703 0x7f3a96fc2cb3
... <More>

It is the same object, on which both threads are currently operating.

benoit

Hi,

As you've discovered the garbage collection has thread-safety issues. We have reworked its implementation with the upcoming Ice 3.6. There's no more garbage collection thread. See Garbage Collection Improvements for the details.

Cheers,
Benoit.

MirkoWaechter

That is unfortunate but thanks for the answer.
Is there anyway to get the mutex of the garbage collection?

benoit

Hi,

Sorry, no, it's not possible to access the mutex of the garbage collection.

You would need to patch Ice to provide access to it through the IceInternal::GCShared interface (defined in cpp/include/Ice/GCShared.h and implemented in cpp/src/Ice/GC.cpp).

The best would be to try out 3.6b. The final release is due in few weeks.

Cheers,
Benoit.