new thread for push some data will case deadlock

dreamy · May 2016

In the example of simpleChat. add a new thread for push some data. when destroy() and push() are invoked in the same time, that can cause deadlock. How to deal with this problem ?

class ThreadTask : public IceUtil::Thread //public IceUtil::TimerTask
{
public:

//virtual void runTimerTask()
virtual void run()
{
    while(true)
    {

        ChatRoomPtr chatRoom = ChatRoom::instance();
        chatRoom->push();
    }
}

};

}

void ChatRoom::push()
{
Lock sync(*this);

for(list<ChatRoom::MemberInfo>::const_iterator p = _members.begin(); p != _members.end(); ++p)
{
    try{
        p->callback->message("aaaaaaaaaaaaa");
        cout << "TimerTask send " << p->callback << endl;;
    }catch(...){
        cout << "TimerTask callback exception\n";
    }
}

}

void
ChatSessionI::destroy(const Ice::Current& current)
{
Lock sync(*this);
if(_callback)
{
ChatRoomPtr chatRoom = ChatRoom::instance();
chatRoom->leave(_callback);
_callback = 0;
chatRoom->message(_userId + " has left the chat room.");
}
current.adapter->remove(current.id);
}

dreamy · May 2016

Asynchronous Method Dispatch can resolve this question. but glacier2router will block for large data to route。

benoit · May 2016

Hi,

It's not clear from the code you posted what could cause the deadlock. The best way to investigate a deadlock is to capture the stack trace of all the threads. In general, this provide enough information to diagnose the deadlock.

One possible cause for deadlock is that you run out of threads, see this FAQ for more information on this.

Cheers,
Benoit.

dreamy · May 2016

Hi Benoit,

You can refer to the Glacier2 simpleChat demo for example. As the above Modify, add a new thread for push data that will lead to a deadlock. I don't know if this is right.

Thread 7 (Thread 0x7fda910a7700 (LWP 3778)):
#0 0x00007fda92091e91 in sigwait () from /lib64/libpthread.so.0
#1 0x00007fda92af3e63 in sigwaitThread () at CtrlCHandler.cpp:166
#2 0x00007fda9208adc5 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fda91db828d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fda908a6700 (LWP 3779)):
#0 0x00007fda9208ea82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fda93011712 in IceUtil::Cond::timedWaitImpl (this=0x1da4cb8, mutex=..., timeout=...) at ../../include/IceUtil/Cond.h:301
#2 0x00007fda92b16011 in IceUtil::Monitor::timedWait (this=0x1da4cb8, timeout=...) at ../../include/IceUtil/Monitor.h:180
#3 0x00007fda92b1504b in IceUtil::Timer::run (this=0x1da4c70) at Timer.cpp:195
#4 0x00007fda92b11526 in startHook (arg=0x1da4c70) at Thread.cpp:659
#5 0x00007fda9208adc5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fda91db828d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fda900a5700 (LWP 3780)):
#0 0x00007fda9208e6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fda934869c5 in IceUtil::Cond::waitImpl (this=0x1da5078, mutex=...) at ../../include/IceUtil/Cond.h:266
#2 0x00007fda92e97e63 in IceUtil::Monitor::wait (this=0x1da5078) at ../../include/IceUtil/Monitor.h:152
#3 0x00007fda92fb6982 in IceInternal::EndpointHostResolver::run (this=0x1da5030) at IPEndpointI.cpp:605
#4 0x00007fda92b11526 in startHook (arg=0x1da5030) at Thread.cpp:659
#5 0x00007fda9208adc5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fda91db828d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fda8f8a4700 (LWP 3781)):
#0 0x00007fda91db8863 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fda9305bde6 in IceInternal::Selector::select (this=0x1da56c8, handlers=std::vector of length 0, capacity 1, timeout=0) at Selector.cpp:470
#2 0x00007fda9307795d in IceInternal::ThreadPool::run (this=0x1da5630, thread=...) at ThreadPool.cpp:714
#3 0x00007fda93079df4 in IceInternal::ThreadPool::EventHandlerThread::run (this=0x1da6a60) at ThreadPool.cpp:1308
#4 0x00007fda92b11526 in startHook (arg=0x1da6a60) at Thread.cpp:659
#5 0x00007fda9208adc5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fda91db828d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fda8f0a3700 (LWP 3782)):
#0 0x00007fda92090f4d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fda9208cd02 in _L_lock_791 () from /lib64/libpthread.so.0
#2 0x00007fda9208cc08 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x0000000000411931 in IceUtil::Mutex::lock (this=0x7fda88001e70) at /usr/local/Ice-3.6.3/include/IceUtil/Mutex.h:294
#4 0x00000000004125e0 in IceUtil::LockT::LockT (this=0x7fda8f0a1ec0, mutex=...) at /usr/local/Ice-3.6.3/include/IceUtil/Lock.h:49
#5 0x0000000000410a94 in ChatRoom::leave (this=0x7fda88001e60, callback=...) at ChatSessionI.cpp:160
#6 0x0000000000411590 in ChatSessionI::destroy (this=0x7fda880024b0, current=...) at ChatSessionI.cpp:252
#7 0x00007fda934a9e2d in Glacier2::Session::___destroy (this=0x7fda880024b0, __inS=..., __current=...) at Session.cpp:1525
#8 0x000000000040badd in Demo::ChatSession::__dispatch (this=0x7fda880024b0, in=..., current=...) at Chat.cpp:463
#9 0x00007fda92f221b8 in IceInternal::Incoming::invoke (this=0x7fda8f0a21a0, servantManager=..., stream=0x7fda8f0a2958) at Incoming.cpp:688
#10 0x00007fda92eee951 in Ice::ConnectionI::invokeAll (this=0x7fda88000d00, stream=..., invokeNum=1, requestId=9, compress=0 '\000', servantManager=..., adapter=...) at ConnectionI.cpp:3421
#11 0x00007fda92ee55f9 in Ice::ConnectionI::dispatch (this=0x7fda88000d00, startCB=..., sentCBs=std::vector of length 0, capacity 0, compress=0 '\000', requestId=9, invokeNum=1, servantManager=..., adapter=..., outAsync=...,
heartbeatCallback=..., stream=...) at ConnectionI.cpp:1817
#12 0x00007fda92ee4d28 in Ice::ConnectionI::message (this=0x7fda88000d00, current=...) at ConnectionI.cpp:1726
#13 0x00007fda93077928 in IceInternal::ThreadPool::run (this=0x1da82f0, thread=...) at ThreadPool.cpp:692
#14 0x00007fda93079df4 in IceInternal::ThreadPool::EventHandlerThread::run (this=0x1da7fa0) at ThreadPool.cpp:1308
#15 0x00007fda92b11526 in startHook (arg=0x1da7fa0) at Thread.cpp:659
#16 0x00007fda9208adc5 in start_thread () from /lib64/libpthread.so.0
#17 0x00007fda91db828d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fda8e8a2700 (LWP 4088)):
#0 0x00007fda9208e6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fda934869c5 in IceUtil::Cond::waitImpl (this=0x7fda8e8a1c60, mutex=...) at ../../include/IceUtil/Cond.h:266
#2 0x00007fda92e97e63 in IceUtil::Monitor::wait (this=0x7fda8e8a1c60) at ../../include/IceUtil/Monitor.h:152
#3 0x00007fda9300f324 in IceInternal::ProxyOutgoingBase::invokeImpl (this=0x7fda8e8a1b80) at Outgoing.cpp:181
#4 0x00007fda930100d3 in IceInternal::Outgoing::invoke (this=0x7fda8e8a1b80) at Outgoing.cpp:388
---Type to continue, or q to quit---
#5 0x00007fda9303558d in IceProxy::Ice::Object::__invoke (this=0x7fda88002b48, __og=...) at Proxy.cpp:1558
#6 0x000000000040a4c1 in IceProxy::Demo::ChatCallback::message (this=0x7fda88002b40, __p_data='a' , __ctx=0x0) at Chat.cpp:86
#7 0x0000000000411db8 in IceProxy::Demo::ChatCallback::message (this=0x7fda88002b40, __p_data='a' ) at ./Chat.h:115
#8 0x0000000000410820 in ChatRoom::push (this=0x7fda88001e60) at ChatSessionI.cpp:137
#9 0x00000000004103fc in (anonymous namespace)::ReapTask::run (this=0x7fda880020b0) at ChatSessionI.cpp:98
#10 0x00007fda92b11526 in startHook (arg=0x7fda880020b0) at Thread.cpp:659
#11 0x00007fda9208adc5 in start_thread () from /lib64/libpthread.so.0
#12 0x00007fda91db828d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fda938e4740 (LWP 3777)):
#0 0x00007fda9208e6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fda92f3c6f9 in IceUtil::Cond::waitImpl (this=0x1da48c0, mutex=...) at ../../include/IceUtil/Cond.h:266
#2 0x00007fda92f3a9c9 in IceUtil::Monitor::wait (this=0x1da48c0) at ../../include/IceUtil/Monitor.h:152
#3 0x00007fda92ff8058 in IceInternal::ObjectAdapterFactory::waitForShutdown (this=0x1da48b0) at ObjectAdapterFactory.cpp:68
#4 0x00007fda92ec7a53 in Ice::CommunicatorI::waitForShutdown (this=0x1da3380) at CommunicatorI.cpp:50
#5 0x0000000000414c32 in ChatServer::run (this=0x7ffdff7f8970, argc=1) at Server.cpp:59
#6 0x00007fda92e91205 in Ice::Application::doMain (this=0x7ffdff7f8970, argc=1, argv=0x7ffdff7f8a78, initData=...) at Application.cpp:679
#7 0x00007fda92e8fc67 in Ice::Application::main (this=0x7ffdff7f8970, argc=1, argv=0x7ffdff7f8a78, initializationData=...) at Application.cpp:424
#8 0x00007fda92e8f397 in Ice::Application::main (this=0x7ffdff7f8970, argc=1, argv=0x7ffdff7f8a78, configFile=0x416450 "config.server") at Application.cpp:341
#9 0x00000000004139da in main (argc=1, argv=0x7ffdff7f8a78) at Server.cpp:73

Cheers,
Dreamy.

bernard · May 2016

Hi Dreamy,

You should not never make synchronous Ice calls with a mutex locked, as in your new Chatroom::push function. Even if this does not cause a deadlock immediately, it's asking for trouble.

If you need to lock a mutex and make remote calls, use asynchronous calls (AMI = Async Method Invocation), as shown by the Chatroom::message function:

void
ChatRoom::message(const string& data) const
{
    Lock sync(*this);
    for(list<MemberInfo>::const_iterator p = _members.begin(); p != _members.end(); ++p)
    {
        p->callback->begin_message(data);
    }
}

Best regards,
Bernard

dreamy · May 2016

Hi Bernard,

Using  AMI  on the server side can avoid this situation.  Because I need to push  on one connection, so  session management is indispensable and it  needs to be locked.

Cheers,
Dreamy.

Archived

new thread for push some data will case deadlock

Comments

Categories