Archived

This forum has been archived. Please start a new discussion on GitHub.

Ice Storm hangs on getPublisher() call

ddelago
edited June 2019 in Help Center

As the title says, whenever I make this call Ice Storm will pause at this point for some reason. Although, it does not always occur. Sometimes it will make it past this point but that is rare. See the log below and any help would be appreciated!

GDB log
Thread 5 (Thread 0x730ff470 (LWP 1804)):
0 0x76e34294 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
1 0x75bfbc38 in IceInternal::Selector::select (this=this@entry=0x587e5c, timeout=-1) at src/ice/cpp/src/Ice/Selector.cpp:724
2 0x75ca2094 in IceInternal::ThreadPool::run (this=this@entry=0x587dd8, thread=...) at src/ice/cpp/src/Ice/ThreadPool.cpp:640
3 0x75ca2834 in IceInternal::ThreadPool::EventHandlerThread::run (this=0xa4e0b0) at src/ice/cpp/src/Ice/ThreadPool.cpp:1205
4 0x75d46394 in startHook (arg=0xa4e0b0) at src/ice/cpp/src/Ice/Thread.cpp:653
5 0x76f73fc4 in start_thread (arg=0x730ff470) at pthread_create.c:335
6 0x76e33bc8 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:76 from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 4 (Thread 0x738ff470 (LWP 1803)):
0 0x76f7a94c in __pthread_cond_wait (cond=cond@entry=0x587a90, mutex=mutex@entry=0x587ac0) at pthread_cond_wait.c:186
1 0x75d04f4c in IceUtil::Cond::waitImpl (mutex=..., this=) at src/ice/cpp/include/IceUtil/Cond.h:260
2 IceUtil::Monitor::wait (this=) at src/ice/cpp/include/IceUtil/Monitor.h:147
3 IceInternal::EndpointHostResolver::run (this=0x587a50) at src/ice/cpp/src/Ice/IPEndpointI.cpp:610
4 0x75d46394 in startHook (arg=0x587a50) at src/ice/cpp/src/Ice/Thread.cpp:653
5 0x76f73fc4 in start_thread (arg=0x738ff470) at pthread_create.c:335
6 0x76e33bc8 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:76 from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

---Type to continue, or q to quit---
Thread 3 (Thread 0x740ff470 (LWP 1802)):
0 0x76f7ace8 in __pthread_cond_timedwait (cond=cond@entry=0x5878b8, mutex=mutex@entry=0x5878e8, abstime=0x740fede8, abstime@entry=0x5878e8) at pthread_cond_timedwait.c:198
1 0x75c22530 in IceUtil::Cond::timedWaitImpl (timeout=..., mutex=..., this=0x5878b8) at src/ice/cpp/include/IceUtil/Cond.h:295
2 IceUtil::Monitor::timedWait (timeout=..., this=0x5878b8) at src/ice/cpp/include/IceUtil/Monitor.h:175
3 IceUtil::Timer::run (this=0x587878) at src/ice/cpp/src/Ice/Timer.cpp:197
4 0x75d46394 in startHook (arg=0x587878) at src/ice/cpp/src/Ice/Thread.cpp:653
5 0x76f73fc4 in start_thread (arg=0x740ff470) at pthread_create.c:335
6 0x76e33bc8 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:76 from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 2 (Thread 0x74ac4470 (LWP 1801)):
0 0x76f7e800 in recv () at ../sysdeps/unix/syscall-template.S:84
1 0x0019d128 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 1 (Thread 0x76feb8c0 (LWP 1799)):
0 0x76f7a94c in __pthread_cond_wait (cond=cond@entry=0xb48a88, mutex=mutex@entry=0xb48ab8) at pthread_cond_wait.c:186
1 0x75c62568 in IceUtil::Cond::waitImpl (mutex=..., this=0xb48a88) at src/ice/cpp/include/IceUtil/Cond.h:260
2 IceUtil::Monitor::wait (this=) at src/ice/cpp/include/IceUtil/Monitor.h:147
3 IceInternal::OutgoingAsyncBase::_waitForResponse (this=0xb48a78) at src/ice/cpp/src/Ice/OutgoingAsync.cpp:478
4 0x75d57024 in IceProxy::Ice::Object::end_ice_invoke (this=this@entry=0xb48a30, outEncaps=std::vector of length 0, capacity 0, result=...) at src/ice/cpp/src/Ice/Proxy.cpp:500
---Type to continue, or q to quit---
5 0x75b8d368 in IceProxy::Ice::Object::ice_invoke (context=..., outParams=std::vector of length 0, capacity 0, inParams=..., mode=, operation=..., this=0xb48a30)
at src/ice/cpp/include/Ice/Proxy.h:2232
6 IcePy::SyncTypedInvocation::invoke (this=0xad0fc8, args=) at src/Operation.cpp:2259
7 0x75b811fc in operationInvoke (self=0x757ef4f0, args=) at src/Operation.cpp:569
8 0x0008c2ec in PyEval_EvalFrameEx ()
9 0x00084494 in PyEval_EvalCodeEx ()
10 0x0008c8dc in PyEval_EvalFrameEx ()
11 0x00084494 in PyEval_EvalCodeEx ()
12 0x000841f8 in PyEval_EvalCode ()
13 0x000bd8b8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

IceStorm Log
-- 06/25/19 15:50:23.677 C:/Program Files (x86)/ZeroC/Ice-3.5.1/bin/icebox-IceStorm: Protocol: sending validate connection
message type = 3 (validate connection)
compression status = 0 (not compressed; do not compress response, if any)
message size = 14
-- 06/25/19 15:50:23.704 C:/Program Files (x86)/ZeroC/Ice-3.5.1/bin/icebox-IceStorm: Protocol: received request
message type = 0 (request)
compression status = 0 (not compressed; do not compress response, if any)
message size = 86
request id = 1
identity = DemoIceStorm/TopicManager
facet =
operation = ice_isA
mode = 1 (nonmutating)
context =
encoding = 1.1
-- 06/25/19 15:50:23.741 C:/Program Files (x86)/ZeroC/Ice-3.5.1/bin/icebox-IceStorm: Protocol: sending reply
message type = 2 (reply)
compression status = 0 (not compressed; do not compress response, if any)
message size = 26
request id = 1
reply status = 0 (ok)
-- 06/25/19 15:50:23.768 C:/Program Files (x86)/ZeroC/Ice-3.5.1/bin/icebox-IceStorm: Protocol: received request
message type = 0 (request)
compression status = 0 (not compressed; do not compress response, if any)
message size = 78
request id = 2
identity = DemoIceStorm/TopicManager
facet =
operation = retrieve
mode = 1 (nonmutating)
context =
encoding = 1.1
-- 06/25/19 15:50:23.807 C:/Program Files (x86)/ZeroC/Ice-3.5.1/bin/icebox-IceStorm: Protocol: sending reply
message type = 2 (reply)
compression status = 0 (not compressed; do not compress response, if any)
message size = 66
request id = 2
reply status = 1 (user exception)
-- 06/25/19 15:50:23.836 C:/Program Files (x86)/ZeroC/Ice-3.5.1/bin/icebox-IceStorm: Protocol: received close connection
message type = 4 (close connection)
compression status = 1 (not compressed; compress response, if any)
message size = 14

Comments

  • benoit
    benoit Rennes, France

    Hi Daniel,

    According to the IceStorm tracing, the last invocation received by IceStorm is "retrieve" on the IceStorm/TopicManager object. IceStorm replies to it with a user exception.

    Could you also enable tracing on the Python client with --Ice.Trace.Protocol, --Ice.Trace.Network=2 and provide both tracing from the same run?

    Also, how do you configure the endpoints on the IceStorm server? One common cause for hangs is that the server publishes endpoints with IP adresses that are not accessible by the client. The client will in turn hang when it tries to establish a connection to this inaccessible endpoint until a timeout occurs.

    See this FAQ entry for information on this.

    You should make sure to configure both <service-name>.Publish.Endpoints and <service-name>.TopicManager.Endpoints in the IceStorm configuration if your IceStorm service runs on a Windows how with multiple network interfaces with some interfaces that can't be reach by the client. See the Ice manual for information on IceStorm configuration properties.

    Cheers,
    Benoit.