Archived

This forum has been archived. Please start a new discussion on GitHub.

Question regarding shutdown deadlock

gmh
gmh
in Help Center

Hello,

We recently had a customer experience a process hang of our sender on Linux (RHEL7) when shutting down. The problem can be avoided by changing a seemingly unrelated PAM configuration. The shutdown (Thread 1) is waiting on Thread 2, which is trying to get a lock. My guess is that Thread 5 is holding the lock, but not returning from "interrupted". This issue does only seem to happen when we load our libIceLiveSocket transport plugin that we had ZeroC develop, even if it had not been in use for the instance of the application. Any pointers as to what to investigate to avoid this deadlock would be appreciated.

Thread 19 (Thread 0x7fdf9425f700 (LWP 9122)):
0 0x00007fdfa1d713c1 in sigwait () at /lib64/libpthread.so.0

3 0x00007fdf9be349fd in clone () at /lib64/libc.so.6

Thread 18 (Thread 0x7fdf9093e700 (LWP 9123)):
0 0x00007fdf9be29ccd in poll () at /lib64/libc.so.6
1 0x00007fdf90c71321 in () at /lib64/libcuda.so

5 0x00007fdf9be349fd in clone () at /lib64/libc.so.6

Thread 17 (Thread 0x7fdf7ffff700 (LWP 9124)):
0 0x00007fdf9be29ccd in poll () at /lib64/libc.so.6
1 0x00007fdf90c71321 in () at /lib64/libcuda.so

5 0x00007fdf9be349fd in clone () at /lib64/libc.so.6

Thread 16 (Thread 0x7fdf7f7fe700 (LWP 9125)):
0 0x00007fdfa1d6da35 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0
1 0x00007fdf90b3864b in () at /lib64/libcuda.so

4 0x00007fdf9be349fd in clone () at /lib64/libc.so.6

Thread 15 (Thread 0x7fdf7ebe3700 (LWP 9189)):
0 0x00007fdfa1d6da35 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0
1 0x00007fdf9d2ff820 in IceUtil::Timer::run() () at /opt/hpremote/rgsender/lib64/libIce.so.37
2 0x00007fdf9d2cb44f in startHook () at /opt/hpremote/rgsender/lib64/libIce.so.37
3 0x00007fdfa1d69ea5 in start_thread () at /lib64/libpthread.so.0
4 0x00007fdf9be349fd in clone () at /lib64/libc.so.6

Thread 14 (Thread 0x7fdf7e3e2700 (LWP 9190)):
0 0x00007fdfa1d6da35 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0
1 0x00007fdf9d332543 in IceInternal::EndpointHostResolver::run() () at /opt/hpremote/rgsender/lib64/libIce.so.37
2 0x00007fdf9d2cb44f in startHook () at /opt/hpremote/rgsender/lib64/libIce.so.37
3 0x00007fdfa1d69ea5 in start_thread () at /lib64/libpthread.so.0
4 0x00007fdf9be349fd in clone () at /lib64/libc.so.6

Thread 13 (Thread 0x7fdf7dbe1700 (LWP 9191)):
0 0x00007fdf9be34fd3 in epoll_wait () at /lib64/libc.so.6
1 0x00007fdf9d26c5be in IceInternal::Selector::select(int) () at /opt/hpremote/rgsender/lib64/libIce.so.37
2 0x00007fdf9d387f18 in IceInternal::ThreadPool::run(IceUtil::Handle const&) () at /opt/hpremote/rgsender/lib64/libIce.so.37
3 0x00007fdf9d3888b2 in IceInternal::ThreadPool::EventHandlerThread::run() () at /opt/hpremote/rgsender/lib64/libIce.so.37
4 0x00007fdf9d2cb44f in startHook () at /opt/hpremote/rgsender/lib64/libIce.so.37
5 0x00007fdfa1d69ea5 in start_thread () at /lib64/libpthread.so.0
6 0x00007fdf9be349fd in clone () at /lib64/libc.so.6

Thread 12 (Thread 0x7fdf6b7fe700 (LWP 9195)):
0 0x00007fdfa1d6da35 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0

4 0x00007fdf9be349fd in clone () at /lib64/libc.so.6

Thread 11 (Thread 0x7fdf6affd700 (LWP 9196)):
0 0x00007fdf9be34fd3 in epoll_wait () at /lib64/libc.so.6

3 0x00007fdf9be349fd in clone () at /lib64/libc.so.6

Thread 10 (Thread 0x7fdf6a7fc700 (LWP 9197)):
0 0x00007fdf9be34fd3 in epoll_wait () at /lib64/libc.so.6
1 0x000000000090a6d0 in processIngressRxLoop ()
2 0x00007fdfa1d69ea5 in start_thread () at /lib64/libpthread.so.0
3 0x00007fdf9be349fd in clone () at /lib64/libc.so.6

Thread 9 (Thread 0x7fdf69ffb700 (LWP 9198)):
0 0x00007fdfa1d6da35 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0
1 0x00000000009051a0 in os_sem_wait ()
2 0x000000000090ad0f in cleanupManager ()
3 0x00007fdfa1d69ea5 in start_thread () at /lib64/libpthread.so.0
4 0x00007fdf9be349fd in clone () at /lib64/libc.so.6

Thread 8 (Thread 0x7fdf697fa700 (LWP 9199)):
0 0x00007fdfa1d6da35 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0
1 0x00007fdf9d2ff820 in IceUtil::Timer::run() () at /opt/hpremote/rgsender/lib64/libIce.so.37
2 0x00007fdf9d2cb44f in startHook () at /opt/hpremote/rgsender/lib64/libIce.so.37
3 0x00007fdfa1d69ea5 in start_thread () at /lib64/libpthread.so.0
4 0x00007fdf9be349fd in clone () at /lib64/libc.so.6

Thread 7 (Thread 0x7fdf68ff9700 (LWP 9200)):
0 0x00007fdfa1d6da35 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0
1 0x00007fdf9d332543 in IceInternal::EndpointHostResolver::run() () at /opt/hpremote/rgsender/lib64/libIce.so.37
2 0x00007fdf9d2cb44f in startHook () at /opt/hpremote/rgsender/lib64/libIce.so.37
3 0x00007fdfa1d69ea5 in start_thread () at /lib64/libpthread.so.0
4 0x00007fdf9be349fd in clone () at /lib64/libc.so.6

Thread 6 (Thread 0x7fdf5bfff700 (LWP 9201)):
0 0x00007fdf9be34fd3 in epoll_wait () at /lib64/libc.so.6
1 0x00007fdf9d26c5be in IceInternal::Selector::select(int) () at /opt/hpremote/rgsender/lib64/libIce.so.37
2 0x00007fdf9d387f18 in IceInternal::ThreadPool::run(IceUtil::Handle const&) () at /opt/hpremote/rgsender/lib64/libIce.so.37
3 0x00007fdf9d3888b2 in IceInternal::ThreadPool::EventHandlerThread::run() () at /opt/hpremote/rgsender/lib64/libIce.so.37
4 0x00007fdf9d2cb44f in startHook () at /opt/hpremote/rgsender/lib64/libIce.so.37
5 0x00007fdfa1d69ea5 in start_thread () at /lib64/libpthread.so.0
6 0x00007fdf9be349fd in clone () at /lib64/libc.so.6

Thread 5 (Thread 0x7fdf59ffb700 (LWP 9206)):
0 0x00007fdf9d2ad609 in IceInternal::interrupted() () at /opt/hpremote/rgsender/lib64/libIce.so.37
1 0x00007fdf9d2ad629 in IceInternal::acceptInterrupted() () at /opt/hpremote/rgsender/lib64/libIce.so.37
2 0x00007fdf9d803155 in IceLiveSocket::doLiveAccept(void*) () at /opt/hpremote/rgsender/lib64/libIceLiveSocket.so.37
3 0x00007fdf9d80ec5c in IceLiveSocket::AcceptThread::run() () at /opt/hpremote/rgsender/lib64/libIceLiveSocket.so.37
4 0x00007fdf9d2cb44f in startHook () at /opt/hpremote/rgsender/lib64/libIce.so.37
5 0x00007fdfa1d69ea5 in start_thread () at /lib64/libpthread.so.0
6 0x00007fdf9be349fd in clone () at /lib64/libc.so.6

Thread 4 (Thread 0x7fdf597fa700 (LWP 9207)):
0 0x00007fdf9be34fd3 in epoll_wait () at /lib64/libc.so.6
1 0x00007fdf9d26c5be in IceInternal::Selector::select(int) () at /opt/hpremote/rgsender/lib64/libIce.so.37
2 0x00007fdf9d387f18 in IceInternal::ThreadPool::run(IceUtil::Handle const&) () at /opt/hpremote/rgsender/lib64/libIce.so.37
3 0x00007fdf9d3888b2 in IceInternal::ThreadPool::EventHandlerThread::run() () at /opt/hpremote/rgsender/lib64/libIce.so.37
4 0x00007fdf9d2cb44f in startHook () at /opt/hpremote/rgsender/lib64/libIce.so.37
5 0x00007fdfa1d69ea5 in start_thread () at /lib64/libpthread.so.0
6 0x00007fdf9be349fd in clone () at /lib64/libc.so.6

Thread 3 (Thread 0x7fdf52ffd700 (LWP 9213)):
0 0x00007fdf9be2ba33 in select () at /lib64/libc.so.6
1 0x000000000078bbb9 in Utilities::InterruptibleSelect::waitForDataOrInterrupt() ()
2 0x0000000000575279 in Sender::XEventPump::internalRun() ()
3 0x000000000094a1bf in execute_native_thread_routine ()
4 0x00007fdfa1d69ea5 in start_thread () at /lib64/libpthread.so.0
5 0x00007fdf9be349fd in clone () at /lib64/libc.so.6

Thread 2 (Thread 0x7fdf5affd700 (LWP 9329)):
0 0x00007fdfa1d7054d in __lll_lock_wait () at /lib64/libpthread.so.0
1 0x00007fdfa1d6be9b in _L_lock_883 () at /lib64/libpthread.so.0
2 0x00007fdfa1d6bd68 in pthread_mutex_lock () at /lib64/libpthread.so.0
3 0x00007fdf9d808ffb in IceUtil::Mutex::lock() const () at /opt/hpremote/rgsender/lib64/libIceLiveSocket.so.37
4 0x00007fdf9d27424c in IceInternal::IncomingConnectionFactory::finished(IceInternal::ThreadPoolCurrent&, bool) () at /opt/hpremote/rgsender/lib64/libIce.so.37
5 0x00007fdf9d380cc6 in (anonymous namespace)::FinishedWorkItem::execute(IceInternal::ThreadPoolCurrent&) () at /opt/hpremote/rgsender/lib64/libIce.so.37
6 0x00007fdf9d38726d in IceInternal::ThreadPoolWorkQueue::message(IceInternal::ThreadPoolCurrent&) () at /opt/hpremote/rgsender/lib64/libIce.so.37
7 0x00007fdf9d387d00 in IceInternal::ThreadPool::run(IceUtil::Handle const&) () at /opt/hpremote/rgsender/lib64/libIce.so.37
8 0x00007fdf9d3888b2 in IceInternal::ThreadPool::EventHandlerThread::run() () at /opt/hpremote/rgsender/lib64/libIce.so.37
9 0x00007fdf9d2cb44f in startHook () at /opt/hpremote/rgsender/lib64/libIce.so.37
10 0x00007fdfa1d69ea5 in start_thread () at /lib64/libpthread.so.0
11 0x00007fdf9be349fd in clone () at /lib64/libc.so.6

Thread 1 (Thread 0x7fdfa2151980 (LWP 9113)):
0 0x00007fdfa1d6b017 in pthread_join () at /lib64/libpthread.so.0
1 0x00007fdf9d2c9e42 in IceUtil::ThreadControl::join() () at /opt/hpremote/rgsender/lib64/libIce.so.37
2 0x00007fdf9d80f191 in IceLiveSocket::Acceptor::close() () at /opt/hpremote/rgsender/lib64/libIceLiveSocket.so.37
3 0x00007fdf9d272733 in IceInternal::IncomingConnectionFactory::closeAcceptor() () at /opt/hpremote/rgsender/lib64/libIce.so.37
4 0x00007fdf9d2728bf in IceInternal::IncomingConnectionFactory::setState(IceInternal::IncomingConnectionFactory::State) [clone .constprop.308] () at /opt/hpremote/rgsender/lib64/libIce.so.37
5 0x00007fdf9d274205 in IceInternal::IncomingConnectionFactory::destroy() () at /opt/hpremote/rgsender/lib64/libIce.so.37
6 0x00007fdf9d29dfd5 in Ice::ObjectAdapterI::deactivate() () at /opt/hpremote/rgsender/lib64/libIce.so.37
7 0x00007fdf9d29b199 in Ice::ObjectAdapterI::destroy() () at /opt/hpremote/rgsender/lib64/libIce.so.37
8 0x00000000004fc81f in Sender::SenderSystem::deactivateObjectAdapters() ()
9 0x0000000000517ff9 in Sender::RWApplication::~RWApplication() ()
10 0x000000000051a20d in Sender::Application::run(int, char) ()
11 0x00000000004662aa in Sender::Application::main(int&, char) ()
12 0x00000000004643e4 in main ()

Categories