Archived

This forum has been archived. Please start a new discussion on GitHub.

IceBox or IceStorm hang up?

Hi, :confused:
i have a project which used Icestorm server in linux, and i start it by normal method: icebox --Ice.Config=config.icebox

in recent time , icebox frequently hang up in running , and the icestorm can't work together, then the publisher and subscriber can't connect to the Topicmanager.

when the icebox hand up, it have to type "kill -9 pid" to forcely stop the icebox , in opposition, i only need to type "killall icebox" to stop it ,and it can quit elegance.

the ice version i used is 3.2.0

i set the icestorm all trace open , but each time it hang up, it didn't throw something exception. and i show some log below:

logTrace1:

............more
[ 03/20/08 19:11:09.593 Protocol: received request
message type = 0 (request)
compression status = 1 (not compressed; compress response, if any)
message size = 62
request id = 1527
identity = DemoIceStorm/TopicManager
facet =
operation = ice_ping
mode = 1 (nonmutating)
context = ]
[ 03/20/08 19:11:09.593 Protocol: sending reply
message type = 2 (reply)
compression status = 1 (not compressed; compress response, if any)
message size = 25
request id = 1527
reply status = 0 (ok) ]
[ 03/20/08 19:11:09.813 Protocol: received request
message type = 0 (request)
compression status = 1 (not compressed; compress response, if any)
message size = 62
request id = 1540
identity = DemoIceStorm/TopicManager
facet =
operation = ice_ping
mode = 1 (nonmutating)
context = ]
[ 03/20/08 19:11:09.813 Protocol: sending reply
message type = 2 (reply)
compression status = 1 (not compressed; compress response, if any)
message size = 25
request id = 1540
reply status = 0 (ok) ]

[ 03/20/08 19:11:11.553 Network: attempting to accept ssl connection
local address = 192.168.1.25:8334
remote address = 192.168.1.74:2153 ]
[ 03/20/08 19:11:11.554 Protocol: sending validate connection
message type = 3 (validate connection)
compression status = 0 (not compressed; do not compress response, if any)
message size = 14 ]
[ 03/20/08 19:12:56.564 Protocol: sending close connection
message type = 4 (close connection)
compression status = 1 (not compressed; compress response, if any)
message size = 14 ]
[ 03/20/08 19:12:56.564 Protocol: sending close connection
message type = 4 (close connection)
compression status = 1 (not compressed; compress response, if any)
message size = 14 ]
[ 03/20/08 19:12:56.566 Network: shutting down ssl connection for writing
local address = 192.168.1.25:13415
remote address = 192.168.1.25:32824 ]
[ 03/20/08 19:12:56.566 Network: closing ssl connection
local address = 192.168.1.25:13415
remote address = 192.168.1.25:32824 ]
[ 03/20/08 19:12:56.566 Network: shutting down ssl connection for writing
local address = 192.168.1.25:40357
remote address = 192.168.1.25:8336 ]
[ 03/20/08 19:12:56.566 Network: closing ssl connection
local address = 192.168.1.25:40357
remote address = 192.168.1.25:8336 ]
[ 03/20/08 19:35:40.557 Protocol: sending close connection
message type = 4 (close connection)
compression status = 1 (not compressed; compress response, if any)
message size = 14 ]
[ 03/20/08 19:35:40.558 Protocol: sending close connection
message type = 4 (close connection)
compression status = 1 (not compressed; compress response, if any)
message size = 14 ]
[ 03/20/08 19:35:40.558 Protocol: sending close connection
message type = 4 (close connection)
compression status = 1 (not compressed; compress response, if any)
message size = 14 ]

LogTrace2:

.................more
[ 03/20/08 16:33:11.148 Protocol: sending asynchronous request
message type = 0 (request)
compression status = 2 (compressed; compress response, if any)
message size = 516
request id = 134
identity = D5C7DB50-A694-4567-A69C-FBEF79004BD9
facet =
operation = report
mode = 0 (normal)
context = ]
[ 03/20/08 16:33:11.303 Protocol: received reply
message type = 2 (reply)
compression status = 1 (not compressed; compress response, if any)
message size = 25
request id = 134
reply status = 0 (ok) ]
[ 03/20/08 16:33:11.690 Network: attempting to accept ssl connection
local address = 192.168.1.25:8334
remote address = 192.168.102.108:2472 ]
[ 03/20/08 16:33:11.690 Protocol: sending validate connection
message type = 3 (validate connection)
compression status = 0 (not compressed; do not compress response, if any)
message size = 14 ]
[ 03/20/08 16:34:44.768 Protocol: sending close connection
message type = 4 (close connection)
compression status = 1 (not compressed; compress response, if any)
message size = 14 ]

[ 03/20/08 16:34:44.769 Network: shutting down ssl connection for writing
local address = 192.168.1.25:31570
remote address = 192.168.1.25:8336 ]
[ 03/20/08 16:34:44.769 Network: closing ssl connection
local address = 192.168.1.25:31570
remote address = 192.168.1.25:8336 ]

[ 03/20/08 16:47:24.252 Protocol: sending close connection
message type = 4 (close connection)
compression status = 1 (not compressed; compress response, if any)
message size = 14 ]
[ 03/20/08 16:47:24.253 Protocol: sending close connection
message type = 4 (close connection)
compression status = 1 (not compressed; compress response, if any)
message size = 14 ]
[ 03/20/08 16:47:24.253 Protocol: sending close connection
message type = 4 (close connection)
compression status = 1 (not compressed; compress response, if any)
message size = 14 ]

what is the problem at icebox? isn't it stable?

Comments

  • matthew
    matthew NL, Canada
    There are no known issues with IceBox. You should upgrade to Ice 3.2.1, and then when this occurs again attach a debugger to icebox and see what the various threads are doing. If you are using Linux you can use:

    gdb - <icebox-pid>

    thread apply all bt

    Under windows you can attach the debugger to the running process at inspect the stacks.

    It is also worthwhile to see whether you can open the external port for IceBox and IceStorm (telnet <ip> <port>). You connect and see "IceP". If this doesn't work then the server has stopped accepting connections for some reason.
  • :confused:
    I upgrade to Ice 3.2.1, and try again, but it occurs again. and i attach a debugger to icebox by linux cmd : gdb - < icebox-pid> at four conditions.


    first conditions :

    Icebox is running normally. and gdb bt is :

    #0 0x00c7c402 in __kernel_vsyscall ()
    #1 0x4731c1a6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
    #2 0x002d748f in IceUtil::Cond::waitImpl<IceUtil::RecMutex> (this=0x89bc3dc,
    mutex=@0x89bc40c) at ../../include/IceUtil/Cond.h:203
    #3 0x002d755f in IceUtil::Monitor<IceUtil::RecMutex>::wait (this=0x89bc3dc)
    at ../../include/IceUtil/Monitor.h:152
    #4 0x002d4970 in IceInternal::ObjectAdapterFactory::waitForShutdown (
    this=0x89bc3d0) at ObjectAdapterFactory.cpp:69
    #5 0x0024ab1a in Ice::CommunicatorI::waitForShutdown (this=0x89be3c8)
    at CommunicatorI.cpp:119
    #6 0x0033831b in Ice::Service::waitForShutdown (this=0xbfa01994)
    at Service.cpp:1156
    #7 0x00338b92 in Ice::Service::run (this=0xbfa01994, argc=@0xbfa019f0,
    argv=0xbfa01a74, initData=@0xbfa019b4) at Service.cpp:767
    #8 0x00339626 in Ice::Service::main (this=0xbfa01994, argc=@0xbfa019f0,
    argv=0xbfa01a74, initData=@0xbfa019b4) at Service.cpp:666
    #9 0x0805e09e in main (argc=1, argv=0xbfa01a74) at Service.cpp:143

    second conditions :

    Icebox look like running normally ,but subscriber could not connect to the IceStorm again, and gdb - <icebox-pid> again, the gdb bt seems ok , it is:

    (gdb) bt
    #0 0x008e7402 in __kernel_vsyscall ()
    #1 0x423aa1a6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
    #2 0x0066b48f in IceUtil::Cond::waitImpl<IceUtil::RecMutex> (this=0x9d9a3d4,
    mutex=@0x9d9a404) at ../../include/IceUtil/Cond.h:203
    #3 0x0066b55f in IceUtil::Monitor<IceUtil::RecMutex>::wait (this=0x9d9a3d4)
    at ../../include/IceUtil/Monitor.h:152
    #4 0x00668970 in IceInternal::ObjectAdapterFactory::waitForShutdown (
    this=0x9d9a3c8) at ObjectAdapterFactory.cpp:69
    #5 0x005deb1a in Ice::CommunicatorI::waitForShutdown (this=0x9d9c300)
    at CommunicatorI.cpp:119
    #6 0x006cc31b in Ice::Service::waitForShutdown (this=0xbfc0d334)
    at Service.cpp:1156
    #7 0x006ccb92 in Ice::Service::run (this=0xbfc0d334, argc=@0xbfc0d390,
    argv=0xbfc0d414, initData=@0xbfc0d354) at Service.cpp:767
    #8 0x006cd626 in Ice::Service::main (this=0xbfc0d334, argc=@0xbfc0d390,
    argv=0xbfc0d414, initData=@0xbfc0d354) at Service.cpp:666
    #9 0x0805e09e in main (argc=1, argv=0xbfc0d414) at Service.cpp:143

    Third conditions :

    i try to telnet the TopicManager Endpoint and get the gdb bt again:

    [root@CyberNsms ~]# telnet 192.168.1.25 8334
    Trying 192.168.1.25...
    Connected to CyberNsms (192.168.1.25).
    Escape character is '^]'.

    (gdb) bt
    #0 0x008e7402 in __kernel_vsyscall ()
    #1 0x423aa1a6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
    #2 0x0066b48f in IceUtil::Cond::waitImpl<IceUtil::RecMutex> (this=0x9d9a3d4,
    mutex=@0x9d9a404) at ../../include/IceUtil/Cond.h:203
    #3 0x0066b55f in IceUtil::Monitor<IceUtil::RecMutex>::wait (this=0x9d9a3d4)
    at ../../include/IceUtil/Monitor.h:152
    #4 0x00668970 in IceInternal::ObjectAdapterFactory::waitForShutdown (
    this=0x9d9a3c8) at ObjectAdapterFactory.cpp:69
    #5 0x005deb1a in Ice::CommunicatorI::waitForShutdown (this=0x9d9c300)
    at CommunicatorI.cpp:119
    #6 0x006cc31b in Ice::Service::waitForShutdown (this=0xbfc0d334)
    at Service.cpp:1156
    #7 0x006ccb92 in Ice::Service::run (this=0xbfc0d334, argc=@0xbfc0d390,
    argv=0xbfc0d414, initData=@0xbfc0d354) at Service.cpp:767
    #8 0x006cd626 in Ice::Service::main (this=0xbfc0d334, argc=@0xbfc0d390,
    argv=0xbfc0d414, initData=@0xbfc0d354) at Service.cpp:666
    #9 0x0805e09e in main (argc=1, argv=0xbfc0d414) at Service.cpp:143


    Forth conditions :

    i try to terminate the icebox which hang up and i send it a SIGTERM( ctr+c or kill <icebox-pid> ),but icebox can't terminate successfully. and i get the gdb bt :

    (gdb) bt
    #0 0x008e7402 in __kernel_vsyscall ()
    #1 0x423ac98e in __lll_mutex_lock_wait () from /lib/libpthread.so.0
    #2 0x423a87fc in _L_mutex_lock_85 () from /lib/libpthread.so.0
    #3 0x423a833d in pthread_mutex_lock () from /lib/libpthread.so.0
    #4 0x0805996e in IceUtil::Mutex::lock (this=0x9e2ad48)
    at ../../include/IceUtil/Mutex.h:305
    #5 0x08059a8b in LockT (this=0xbfc0cb04, mutex=@0x9e2ad48)
    at ../../include/IceUtil/Lock.h:49
    #6 0x005f2718 in Ice::ConnectionI::setState (this=0x9e2ab20,
    state=Ice::ConnectionI::StateClosed) at ConnectionI.cpp:1891
    #7 0x005f2ddf in Ice::ConnectionI::setState (this=0x9e2ab20,
    state=Ice::ConnectionI::StateClosing, ex=@0xbfc0cc74)
    at ConnectionI.cpp:1779
    #8 0x005fb3d4 in Ice::ConnectionI::destroy (this=0x9e2ab20,
    reason=Ice::ConnectionI::ObjectAdapterDeactivated) at ConnectionI.cpp:265
    #9 0x005eae7f in IceUtilInternal::VoidMemFun1<Ice::ConnectionI, IceInternal::Handle<Ice::ConnectionI>, Ice::ConnectionI::DestructionReason>::operator() (
    this=0xbfc0cd2c, handle=@0xbfc0ccec,
    arg=Ice::ConnectionI::ObjectAdapterDeactivated)
    at ../../include/IceUtil/Functional.h:80
    #10 0x005ed851 in std::binder2nd<IceUtilInternal::VoidMemFun1<Ice::ConnectionI, IceInternal::Handle<Ice::ConnectionI>, Ice::ConnectionI::DestructionReason> >::operator() (this=0xbfc0cd2c, __x=@0x9e2c810)
    at /usr/lib/gcc/i386-redhat-linux/4.1.1/../../../../include/c++/4.1.1/bits/stl_function.h:446
    #11 0x005ed8b2 in std::for_each<std::_List_iterator<IceInternal::Handle<Ice::Con---Type <return> to continue, or q <return> to quit---
    nectionI> >, std::binder2nd<IceUtilInternal::VoidMemFun1<Ice::ConnectionI, IceInternal::Handle<Ice::ConnectionI>, Ice::ConnectionI::DestructionReason> > > (
    __first={_M_node = 0x9e2c808}, __last={_M_node = 0x9dad7ac}, __f=
    {<std::unary_function<IceInternal::Handle<Ice::ConnectionI>,void>> = {<No data fields>}, op = {<std::binary_function<IceInternal::Handle<Ice::ConnectionI>,Ice::ConnectionI::DestructionReason,void>> = {<No data fields>}, _mfn = {__pfn = 0x5fb35c <Ice::ConnectionI::destroy(Ice::ConnectionI::DestructionReason)>, __delta = 0}}, value = Ice::ConnectionI::ObjectAdapterDeactivated})
    at /usr/lib/gcc/i386-redhat-linux/4.1.1/../../../../include/c++/4.1.1/bits/stl_algo.h:159
    #12 0x005e36e2 in IceInternal::IncomingConnectionFactory::setState (
    this=0x9dad6a8, state=IceInternal::IncomingConnectionFactory::StateClosed)
    at ConnectionFactory.cpp:1021
    #13 0x005e3747 in IceInternal::IncomingConnectionFactory::destroy (
    this=0x9dad6a8) at ConnectionFactory.cpp:533
    #14 0x006782fe in IceUtilInternal::VoidMemFun<IceInternal::IncomingConnectionFactory, IceInternal::Handle<IceInternal::IncomingConnectionFactory> >::operator()
    (this=0xbfc0ce9c, handle=@0xbfc0ce7c)
    at ../../include/IceUtil/Functional.h:65
    #15 0x0067d0f5 in std::for_each<__gnu_cxx::__normal_iterator<IceInternal::Handle<IceInternal::IncomingConnectionFactory>*, std::vector<IceInternal::Handle<IceInternal::IncomingConnectionFactory>, std::allocator<IceInternal::Handle<IceInternal::IncomingConnectionFactory> > > >, IceUtilInternal::VoidMemFun<IceInternal::IncomingConnectionFactory, IceInternal::Handle<IceInternal::IncomingConnectionFactory> > > (__first={_M_current = 0x9e2c72c}, __last={_M_current = 0x9e2c730},
    __f=
    ---Type <return> to continue, or q <return> to quit---
    {<std::unary_function<IceInternal::Handle<IceInternal::IncomingConnectionFactory>,void>> = {<No data fields>}, _mfn = {__pfn = 0x5e370a <IceInternal::IncomingConnectionFactory::destroy()>, __delta = 0}})
    at /usr/lib/gcc/i386-redhat-linux/4.1.1/../../../../include/c++/4.1.1/bits/stl_algo.h:159
    #16 0x00677924 in Ice::ObjectAdapterI::deactivate (this=0x9dad120)
    at ObjectAdapterI.cpp:242
    #17 0x006724ce in Ice::ObjectAdapterI::destroy (this=0x9dad120)
    at ObjectAdapterI.cpp:322
    #18 0x00319422 in IceStorm::ServiceI::stop (this=0x9d9ae18) at Service.cpp:166
    #19 0x08052518 in IceBox::ServiceManagerI::stopAll (this=0x9d9a730)
    at ServiceManagerI.cpp:549
    #20 0x08052e07 in IceBox::ServiceManagerI::stop (this=0x9d9a730)
    at ServiceManagerI.cpp:263
    #21 0x0805d5e2 in IceBox::IceBoxService::stop (this=0xbfc0d334)
    at Service.cpp:90
    #22 0x006ccba4 in Ice::Service::run (this=0xbfc0d334, argc=@0xbfc0d390,
    argv=0xbfc0d414, initData=@0xbfc0d354) at Service.cpp:772
    #23 0x006cd626 in Ice::Service::main (this=0xbfc0d334, argc=@0xbfc0d390,
    argv=0xbfc0d414, initData=@0xbfc0d354) at Service.cpp:666
    #24 0x0805e09e in main (argc=1, argv=0xbfc0d414) at Service.cpp:143


    at last , i have to terminate it with SIGKILL by 'kill -9 <icebox-pid>'。
    what is the problem with icebox?
  • matthew
    matthew NL, Canada
    Just a single stack trace isn't very helpful. You must supply all of the stacks from each thread: "thread apply all bt" from within gdb will do that.

    The telnet trace above does not show a "IceP" output. Is that really what occurred, or did you not paste the entire output?

    The 4th trace you gave indicates that IceStorm is trying to shutdown an existing incoming connection which is hanging. To avoid this you can set a timeout on the connection using the -t parameter (see the Ice Manual section 32.4.6). tcp -h ... -p ... -t 10000, for example, would set a 10 second timeout.
  • Hi,

    I think I am experiencing similar problems. When trying to use IceStorm (Ice version 3.2.1), I noticed that my app did not pass the line of code where I try to cast the topic manager like this:

    IceStorm::TopicManagerPrx topicManager = IceStorm::TopicManagerPrx::checkedCast(obj);

    preceeded by
    Ice::ObjectPrx obj = ic->stringToProxy("IceStorm/TopicManager:tcp -p 9999");

    I have IceBox running with IceStorm. I looked at the network trace of both the client and IceBox and found the following:

    IceBox/IceCast gives:
    [ 03/25/08 14:12:20.765 Network: accepted tcp connection
    local address = 127.0.0.1:9999
    remote address = 127.0.0.1:2234 ]
    [ 03/25/08 14:12:20.765 Protocol: sending validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/25/08 14:12:20.781 Network: sent 14 of 14 bytes via tcp
    local address = 127.0.0.1:9999
    remote address = 127.0.0.1:2234 ]
    [ 03/25/08 14:12:20.843 Network: received 14 of 14 bytes via tcp
    local address = 127.0.0.1:9999
    remote address = 127.0.0.1:2234 ]
    [ 03/25/08 14:12:20.843 Network: received 68 of 68 bytes via tcp
    local address = 127.0.0.1:9999
    remote address = 127.0.0.1:2234 ]
    [ 03/25/08 14:12:20.859 Protocol: received request
    message type = 0 (request)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 82
    request id = 1
    identity = IceStorm/TopicManager
    facet =
    operation = ice_isA
    mode = 1 (nonmutating)
    context = ]
    [ 03/25/08 14:12:20.875 Protocol: sending reply
    message type = 2 (reply)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 26
    request id = 1
    reply status = 0 (ok) ]
    [ 03/25/08 14:12:20.875 Network: sent 26 of 26 bytes via tcp
    local address = 127.0.0.1:9999
    remote address = 127.0.0.1:2234 ]

    The client application states:
    [ 03/25/08 14:12:20.750 Network: trying to establish tcp connection to 127.0.0.1:9999 ]
    [ 03/25/08 14:12:20.765 Network: tcp connection established
    local address = 127.0.0.1:2234
    remote address = 127.0.0.1:9999 ]
    [ 03/25/08 14:12:20.812 Network: received 14 of 14 bytes via tcp
    local address = 127.0.0.1:2234
    remote address = 127.0.0.1:9999 ]
    [ 03/25/08 14:12:20.812 Protocol: received validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/25/08 14:12:20.828 Protocol: sending request
    message type = 0 (request)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 82
    request id = 1
    identity = IceStorm/TopicManager
    facet =
    operation = ice_isA
    mode = 1 (nonmutating)
    context = ]
    [ 03/25/08 14:12:20.843 Network: sent 82 of 82 bytes via tcp
    local address = 127.0.0.1:2234
    remote address = 127.0.0.1:9999 ]

    As far as I can see the client is stuck at:
    int rc = WaitForSingleObject(_sem, INFINITE); (line 35 of cond.cpp)

    I'm I doing anything wrong?

    Regards,
    Christian
  • matthew
    matthew NL, Canada
    As far as I can see the client is stuck at:
    int rc = WaitForSingleObject(_sem, INFINITE); (line 35 of cond.cpp)

    I'm I doing anything wrong?

    Regards,
    Christian

    From the traces it looks like your client never receives the response to the ice_isA sent from the server. I assume you are running this on a local network? If that is the case, this would be highly unusual! Do the Ice demos work?

    If not, then it is likely that you have some problem with your Windows install, or perhaps some network issue (such as a firewall eating your packets).

    If the Ice demos work and your application does not, then it is more likely an issue with your app.
  • Thanks,

    The clock sample worked, so it is something in my app. I was trying all the ice setup during the initialization of a dll, so inside DLLmain. It might just be that this is the wrong part here. I moved the code to some "normal" function which is called during lifetime of the DLL and apparently this works. Maybe this is the logical explanation for the problem?

    Regards,
    Christian
  • matthew
    matthew NL, Canada
    It certainly seems like you did something wrong in the initialization of your DLL, yes.
  • happen alike

    ;) hi,there

    The same problem also happened to me , the strange thing is that I can not find out when the Icebox hanging up or dead , I only use icebox for Icestorm service , there will be lots of reports to the server every minute and the server itself subscribe to such topic .How can I trace the strange thing ,I use ice trace but it seems not to give useful infomation .Can you show me the way ?

    Another question is that if two or more messages come at the same time ,how the icestorm handle them ,I doubt the messages will be mixed?

    many thanks !
  • matthew
    matthew NL, Canada
    I'm afraid that without more information its impossible to give you advice on tracking down your problem. What exactly is occurring? Why don't you find the trace useful?
    Another question is that if two or more messages come at the same time ,how the icestorm handle them ,I doubt the messages will be mixed?

    There are no ordering guarantees between different event publishers, since that is impossible. However, from a single publisher, assuming you use the correct QoS, then you can guarantee that the orders are delivered in the same order they arrive. You can find more information on ordering in the Ice manual (see section 44.10 Delivery Mode)
  • sorry for delay reply,hehe. because the icebox didn't occur frequently these days, and i have no idea how to debug.

    these are the gdb bt below when the icebox hang up.


    (gdb) info threads
    11 Thread -1207985264 (LWP 11544) 0x00ccf402 in __kernel_vsyscall ()
    10 Thread -1218475120 (LWP 11545) 0x00ccf402 in __kernel_vsyscall ()
    9 Thread -1228964976 (LWP 11546) 0x00ccf402 in __kernel_vsyscall ()
    8 Thread -1239454832 (LWP 11547) 0x00ccf402 in __kernel_vsyscall ()
    7 Thread -1249944688 (LWP 11548) 0x00ccf402 in __kernel_vsyscall ()
    6 Thread -1260434544 (LWP 11549) 0x00ccf402 in __kernel_vsyscall ()
    5 Thread -1270924400 (LWP 11550) 0x00ccf402 in __kernel_vsyscall ()
    4 Thread -1281414256 (LWP 11551) 0x00ccf402 in __kernel_vsyscall ()
    3 Thread -1291904112 (LWP 11552) 0x00ccf402 in __kernel_vsyscall ()
    2 Thread -1304429680 (LWP 11669) 0x00ccf402 in __kernel_vsyscall ()
    1 Thread -1207982384 (LWP 11543) 0x00ccf402 in __kernel_vsyscall ()
    (gdb) thread apply all bt

    Thread 11 (Thread -1207985264 (LWP 11544)):
    #0 0x00ccf402 in __kernel_vsyscall ()
    #1 0x423adebe in do_sigwait () from /lib/libpthread.so.0
    #2 0x423adf5f in sigwait () from /lib/libpthread.so.0
    #3 0x0074977f in sigwaitThread () at CtrlCHandler.cpp:124
    #4 0x423a63db in start_thread () from /lib/libpthread.so.0
    #5 0x4230006e in clone () from /lib/libc.so.6

    Thread 10 (Thread -1218475120 (LWP 11545)):
    #0 0x00ccf402 in __kernel_vsyscall ()
    #1 0x423aa42c in pthread_cond_timedwait@@GLIBC_2.3.2 ()
    from /lib/libpthread.so.0
    #2 0x005660bb in IceUtil::Cond::timedWaitImpl<IceUtil::Mutex> (
    this=0x87ed4b4, mutex=@0x87ed4e4, timeout=@0x87ed504)
    at ../../include/IceUtil/Cond.h:224
    #3 0x005661b0 in IceUtil::Monitor<IceUtil::Mutex>::timedWait (this=0x87ed4b4,
    timeout=@0x87ed504) at ../../include/IceUtil/Monitor.h:180
    #4 0x00566305 in IceInternal::ConnectionMonitor::run (this=0x87ed490)
    at ConnectionMonitor.cpp:78
    #5 0x007653cb in startHook (arg=0x87ed490) at Thread.cpp:357
    #6 0x423a63db in start_thread () from /lib/libpthread.so.0
    #7 0x4230006e in clone () from /lib/libc.so.6

    Thread 9 (Thread -1228964976 (LWP 11546)):
    #0 0x00ccf402 in __kernel_vsyscall ()
    #1 0x423006e6 in __epoll_wait_nocancel () from /lib/libc.so.6
    #2 0x0063f21a in IceInternal::ThreadPool::run (this=0x87edb88)
    at ThreadPool.cpp:462
    #3 0x00640e36 in IceInternal::ThreadPool::EventHandlerThread::run (
    this=0x87edc68) at ThreadPool.cpp:1066
    #4 0x007653cb in startHook (arg=0x87edc68) at Thread.cpp:357
    #5 0x423a63db in start_thread () from /lib/libpthread.so.0
    #6 0x4230006e in clone () from /lib/libc.so.6

    Thread 8 (Thread -1239454832 (LWP 11547)):
    #0 0x00ccf402 in __kernel_vsyscall ()
    #1 0x423aa42c in pthread_cond_timedwait@@GLIBC_2.3.2 ()
    from /lib/libpthread.so.0
    #2 0x005660bb in IceUtil::Cond::timedWaitImpl<IceUtil::Mutex> (
    this=0x87fe384, mutex=@0x87fe3b4, timeout=@0x87fe3d4)
    at ../../include/IceUtil/Cond.h:224
    ---Type <return> to continue, or q <return> to quit---
    #3 0x005661b0 in IceUtil::Monitor<IceUtil::Mutex>::timedWait (this=0x87fe384,
    timeout=@0x87fe3d4) at ../../include/IceUtil/Monitor.h:180
    #4 0x00566305 in IceInternal::ConnectionMonitor::run (this=0x87fe360)
    at ConnectionMonitor.cpp:78
    #5 0x007653cb in startHook (arg=0x87fe360) at Thread.cpp:357
    #6 0x423a63db in start_thread () from /lib/libpthread.so.0
    #7 0x4230006e in clone () from /lib/libc.so.6

    Thread 7 (Thread -1249944688 (LWP 11548)):
    #0 0x00ccf402 in __kernel_vsyscall ()
    #1 0x422f6503 in poll () from /lib/libc.so.6
    #2 0x00a3a24a in selectReadWrite (fd=18, read=true, timeout=-1)
    at Util.cpp:312
    #3 0x00a3a334 in IceSSL::selectRead (fd=18, timeout=-1) at Util.cpp:337
    #4 0x00a36c74 in IceSSL::TransceiverI::initialize (this=0x886f5a8, timeout=-1)
    at TransceiverI.cpp:444
    #5 0x0055da74 in Ice::ConnectionI::validate (this=0x88690c0)
    at ConnectionI.cpp:128
    #6 0x005486e3 in IceInternal::IncomingConnectionFactory::message (
    this=0x8800980, threadPool=@0xb57f519c) at ConnectionFactory.cpp:786
    #7 0x00640679 in IceInternal::ThreadPool::run (this=0x8800350)
    at ThreadPool.cpp:843
    #8 0x00640e36 in IceInternal::ThreadPool::EventHandlerThread::run (
    this=0x8800458) at ThreadPool.cpp:1066
    #9 0x007653cb in startHook (arg=0x8800458) at Thread.cpp:357
    #10 0x423a63db in start_thread () from /lib/libpthread.so.0
    #11 0x4230006e in clone () from /lib/libc.so.6

    Thread 6 (Thread -1260434544 (LWP 11549)):
    #0 0x00ccf402 in __kernel_vsyscall ()
    #1 0x423aa1a6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
    #2 0x0054f003 in IceUtil::Cond::waitImpl<IceUtil::Mutex> (this=0x8800b6c,
    mutex=@0x8800b9c) at ../../include/IceUtil/Cond.h:203
    #3 0x0054f0d3 in IceUtil::Monitor<IceUtil::Mutex>::wait (this=0x8800b6c)
    at ../../include/IceUtil/Monitor.h:152
    #4 0x0018920a in IceStorm::BatchFlusher::run (this=0x8800b48)
    at BatchFlusher.cpp:82
    #5 0x007653cb in startHook (arg=0x8800b48) at Thread.cpp:357
    #6 0x423a63db in start_thread () from /lib/libpthread.so.0
    #7 0x4230006e in clone () from /lib/libc.so.6

    Thread 5 (Thread -1270924400 (LWP 11550)):
    ---Type <return> to continue, or q <return> to quit---
    #0 0x00ccf402 in __kernel_vsyscall ()
    #1 0x423aa1a6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
    #2 0x0054f003 in IceUtil::Cond::waitImpl<IceUtil::Mutex> (this=0x8800d54,
    mutex=@0x8800d84) at ../../include/IceUtil/Cond.h:203
    #3 0x0054f0d3 in IceUtil::Monitor<IceUtil::Mutex>::wait (this=0x8800d54)
    at ../../include/IceUtil/Monitor.h:152
    #4 0x00190820 in IceStorm::SubscriberPoolMonitor::run (this=0x8800d30)
    at SubscriberPool.cpp:122
    #5 0x007653cb in startHook (arg=0x8800d30) at Thread.cpp:357
    #6 0x423a63db in start_thread () from /lib/libpthread.so.0
    #7 0x4230006e in clone () from /lib/libc.so.6

    Thread 4 (Thread -1281414256 (LWP 11551)):
    #0 0x00ccf402 in __kernel_vsyscall ()
    #1 0x423aa1a6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
    #2 0x0054f003 in IceUtil::Cond::waitImpl<IceUtil::Mutex> (this=0x8800c84,
    mutex=@0x8800cb4) at ../../include/IceUtil/Cond.h:203
    #3 0x0054f0d3 in IceUtil::Monitor<IceUtil::Mutex>::wait (this=0x8800c84)
    at ../../include/IceUtil/Monitor.h:152
    #4 0x0018e189 in IceStorm::SubscriberPool::dequeue (this=0x8800c78,
    subscriber=@0xb39f2358, requeue=false, interval=@0xb39f235c,
    computeInterval=@0xb39f2357) at SubscriberPool.cpp:393
    #5 0x0018e3cf in run (this=0x8800b10) at SubscriberPool.cpp:48
    #6 0x007653cb in startHook (arg=0x8800b10) at Thread.cpp:357
    #7 0x423a63db in start_thread () from /lib/libpthread.so.0
    #8 0x4230006e in clone () from /lib/libc.so.6

    Thread 3 (Thread -1291904112 (LWP 11552)):
    #0 0x00ccf402 in __kernel_vsyscall ()
    #1 0x423aa42c in pthread_cond_timedwait@@GLIBC_2.3.2 ()
    from /lib/libpthread.so.0
    #2 0x005660bb in IceUtil::Cond::timedWaitImpl<IceUtil::Mutex> (
    this=0x8869f1c, mutex=@0x8869f4c, timeout=@0x8869f70)
    at ../../include/IceUtil/Cond.h:224
    #3 0x005661b0 in IceUtil::Monitor<IceUtil::Mutex>::timedWait (this=0x8869f1c,
    timeout=@0x8869f70) at ../../include/IceUtil/Monitor.h:180
    #4 0x00e3abbf in Freeze::CheckpointThread::run (this=0x8869ef8)
    at SharedDbEnv.cpp:380
    #5 0x007653cb in startHook (arg=0x8869ef8) at Thread.cpp:357
    #6 0x423a63db in start_thread () from /lib/libpthread.so.0
    #7 0x4230006e in clone () from /lib/libc.so.6

    ---Type <return> to continue, or q <return> to quit---
    Thread 2 (Thread -1304429680 (LWP 11669)):
    #0 0x00ccf402 in __kernel_vsyscall ()
    #1 0x423006e6 in __epoll_wait_nocancel () from /lib/libc.so.6
    #2 0x0063f21a in IceInternal::ThreadPool::run (this=0x88b2d48)
    at ThreadPool.cpp:462
    #3 0x00640e36 in IceInternal::ThreadPool::EventHandlerThread::run (
    this=0x88b67b0) at ThreadPool.cpp:1066
    #4 0x007653cb in startHook (arg=0x88b67b0) at Thread.cpp:357
    #5 0x423a63db in start_thread () from /lib/libpthread.so.0
    #6 0x4230006e in clone () from /lib/libc.so.6

    Thread 1 (Thread -1207982384 (LWP 11543)):
    #0 0x00ccf402 in __kernel_vsyscall ()
    #1 0x423aa1a6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
    #2 0x005cd48f in IceUtil::Cond::waitImpl<IceUtil::RecMutex> (this=0x87ed3d4,
    mutex=@0x87ed404) at ../../include/IceUtil/Cond.h:203
    #3 0x005cd55f in IceUtil::Monitor<IceUtil::RecMutex>::wait (this=0x87ed3d4)
    at ../../include/IceUtil/Monitor.h:152
    #4 0x005ca970 in IceInternal::ObjectAdapterFactory::waitForShutdown (
    this=0x87ed3c8) at ObjectAdapterFactory.cpp:69
    #5 0x00540b1a in Ice::CommunicatorI::waitForShutdown (this=0x87ef300)
    at CommunicatorI.cpp:119
    #6 0x0062e31b in Ice::Service::waitForShutdown (this=0xbff80a94)
    at Service.cpp:1156
    #7 0x0062eb92 in Ice::Service::run (this=0xbff80a94, argc=@0xbff80af0,
    argv=0xbff80b74, initData=@0xbff80ab4) at Service.cpp:767
    #8 0x0062f626 in Ice::Service::main (this=0xbff80a94, argc=@0xbff80af0,
    argv=0xbff80b74, initData=@0xbff80ab4) at Service.cpp:666
    #9 0x0805e09e in main (argc=1, argv=0xbff80b74) at Service.cpp:143
    #0 0x00ccf402 in __kernel_vsyscall ()
  • and the trace of the icebox before it hang up

    [ 03/28/08 09:18:00.079 Protocol: sending reply
    message type = 2 (reply)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 25
    request id = 148
    reply status = 0 (ok) ]
    [ 03/28/08 09:18:00.079 Protocol: received request
    message type = 0 (request)
    compression status = 1 (not compressed; compress response, if any)
    message size = 62
    request id = 50
    identity = DemoIceStorm/FFFF.publish
    facet =
    operation = ice_ping
    mode = 1 (nonmutating)
    context = ]
    [ 03/28/08 09:18:00.079 Protocol: sending reply
    message type = 2 (reply)
    compression status = 1 (not compressed; compress response, if any)
    message size = 25
    request id = 50
    reply status = 0 (ok) ]
    [ 03/28/08 09:18:00.080 Protocol: received request
    message type = 0 (request)
    compression status = 1 (not compressed; compress response, if any)
    message size = 62
    request id = 56
    identity = DemoIceStorm/FFFF.publish
    facet =
    operation = ice_ping
    mode = 1 (nonmutating)
    context = ]
    [ 03/28/08 09:18:00.080 Protocol: sending reply
    message type = 2 (reply)
    compression status = 1 (not compressed; compress response, if any)
    message size = 25
    request id = 56
    reply status = 0 (ok) ]
    [ 03/28/08 09:18:00.080 Protocol: received request
    message type = 0 (request)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 62
    request id = 263
    identity = DemoIceStorm/FFFF.publish
    facet =
    operation = ice_ping
    mode = 1 (nonmutating)
    context = ]
    [ 03/28/08 09:18:00.080 Protocol: sending reply
    message type = 2 (reply)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 25
    request id = 263
    reply status = 0 (ok) ]
    [ 03/28/08 09:18:00.080 Network: attempting to accept ssl connection
    local address = 127.0.0.1:8334
    remote address = 127.0.0.1:57854 ]
    [ 03/28/08 09:18:00.080 Protocol: sending validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:18:00.102 Network: closing ssl connection
    local address = 127.0.0.1:8334
    remote address = 127.0.0.1:57854 ]
    [ 03/28/08 09:18:00.103 Network: attempting to accept ssl connection
    local address = 192.168.2.102:8334
    remote address = 192.168.102.21:23270 ]
    [ 03/28/08 09:18:00.103 Protocol: sending validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:18:00.123 Network: closing ssl connection
    local address = 192.168.2.102:8334
    remote address = 192.168.102.21:23270 ]
    [ 03/28/08 09:18:00.123 Network: attempting to accept ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.1.25:47004 ]
    [ 03/28/08 09:18:00.123 Protocol: sending validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:18:00.140 Security: IceSSL: ignoring certificate validation failure:
    peer certificate does not contain `192.168.2.102' in its subjectAltName extension ]
    [ 03/28/08 09:18:00.140 Network: ssl connection established
    local address = 192.168.2.102:18341
    remote address = 192.168.2.102:8336 ]
    [ 03/28/08 09:18:00.141 Security: SSL summary for outgoing connection
    cipher = DHE-RSA-AES256-SHA
    bits = 256
    protocol = TLSv1
    local address = 192.168.2.102:18341
    remote address = 192.168.2.102:8336 ]
    [ 03/28/08 09:18:00.141 Protocol: received validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:18:00.141 Protocol: sending request
    message type = 0 (request)
    compression status = 1 (not compressed; compress response, if any)
    message size = 74
    request id = 0 (oneway)
    identity = 1D1A150A-B830-4305-8888-06939D904FCF
    facet =
    operation = ice_ping
    mode = 1 (nonmutating)
    context = ]

    more

    Sorry ,i don't know how to post long text that longer than 10000, hehe

  • continue

    [ 03/28/08 09:18:00.142 Network: trying to establish ssl connection to 192.168.2.102:8337 ]
    [ 03/28/08 09:18:00.143 Network: closing ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.1.25:47004 ]
    [ 03/28/08 09:18:00.143 Network: shutting down ssl connection for writing
    local address = 192.168.2.102:8334
    remote address = 192.168.2.102:2825 ]
    [ 03/28/08 09:18:00.144 Network: closing ssl connection
    local address = 192.168.2.102:8334
    remote address = 192.168.2.102:2825 ]
    [ 03/28/08 09:18:00.144 Network: shutting down ssl connection for writing
    local address = 192.168.2.102:8333
    remote address = 192.168.2.146:4592 ]
    [ 03/28/08 09:18:00.144 Network: closing ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.2.146:4592 ]
    [ 03/28/08 09:18:00.144 Network: shutting down ssl connection for writing
    local address = 192.168.2.102:8333
    remote address = 192.168.2.140:4064 ]
    [ 03/28/08 09:18:00.144 Network: closing ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.2.140:4064 ]
    [ 03/28/08 09:18:00.144 Network: shutting down ssl connection for writing
    local address = 192.168.2.102:8334
    remote address = 192.168.2.102:2881 ]
    [ 03/28/08 09:18:00.144 Network: closing ssl connection
    local address = 192.168.2.102:8334
    remote address = 192.168.2.102:2881 ]
    [ 03/28/08 09:18:00.145 Network: shutting down ssl connection for writing
    local address = 192.168.2.102:8333
    remote address = 192.168.102.21:20452 ]
    [ 03/28/08 09:18:00.145 Network: closing ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.102.21:20452 ]
    [ 03/28/08 09:18:00.145 Network: shutting down ssl connection for writing
    local address = 192.168.2.102:8333
    remote address = 192.168.1.25:46391 ]
    [ 03/28/08 09:18:00.145 Network: closing ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.1.25:46391 ]
    [ 03/28/08 09:18:00.145 Network: shutting down ssl connection for writing
    local address = 192.168.2.102:8334
    remote address = 192.168.2.102:58874 ]
    [ 03/28/08 09:18:00.145 Network: closing ssl connection
    local address = 192.168.2.102:8334
    remote address = 192.168.2.102:58874 ]
    [ 03/28/08 09:18:00.146 Network: shutting down ssl connection for writing
    local address = 192.168.2.102:8333
    remote address = 192.168.2.81:1064 ]
    [ 03/28/08 09:18:00.146 Network: closing ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.2.81:1064 ]
    [ 03/28/08 09:18:00.146 Network: shutting down ssl connection for writing
    local address = 192.168.2.102:8333
    remote address = 192.168.102.21:12905 ]
    [ 03/28/08 09:18:00.146 Network: closing ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.102.21:12905 ]
    [ 03/28/08 09:18:00.146 Network: shutting down ssl connection for writing
    local address = 192.168.2.102:8333
    remote address = 192.168.1.25:9709 ]
    [ 03/28/08 09:18:00.146 Network: closing ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.1.25:9709 ]
    [ 03/28/08 09:18:00.146 Network: shutting down ssl connection for writing
    local address = 192.168.2.102:8333
    remote address = 192.168.2.102:61155 ]
    [ 03/28/08 09:18:00.146 Network: closing ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.2.102:61155 ]
    [ 03/28/08 09:18:00.147 Network: shutting down ssl connection for writing
    local address = 192.168.2.102:8333
    remote address = 192.168.2.75:1038 ]
    [ 03/28/08 09:18:00.147 Network: closing ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.2.75:1038 ]
    [ 03/28/08 09:18:00.147 Network: shutting down ssl connection for writing
    local address = 192.168.2.102:8333
    remote address = 192.168.2.159:4572 ]
    [ 03/28/08 09:18:00.147 Network: closing ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.2.159:4572 ]
    [ 03/28/08 09:18:00.147 Network: shutting down ssl connection for writing
    local address = 192.168.2.102:8333
    remote address = 192.168.1.25:9713 ]
    [ 03/28/08 09:18:00.147 Network: closing ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.1.25:9713 ]
    [ 03/28/08 09:18:00.147 Network: attempting to accept ssl connection
    local address = 127.0.0.1:8334
    remote address = 127.0.0.1:57860 ]
    [ 03/28/08 09:18:00.148 Protocol: sending validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:18:00.160 Network: closing ssl connection
    local address = 127.0.0.1:8334
    remote address = 127.0.0.1:57860 ]
    [ 03/28/08 09:18:00.160 Network: attempting to accept ssl connection
    local address = 192.168.2.102:8334
    remote address = 192.168.1.25:12173 ]
    [ 03/28/08 09:18:00.160 Protocol: sending validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:18:00.174 Network: closing ssl connection
    local address = 192.168.2.102:8334
    remote address = 192.168.1.25:12173 ]
    [ 03/28/08 09:18:00.175 Network: attempting to accept ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.1.25:47014 ]
    [ 03/28/08 09:18:00.175 Protocol: sending validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:18:00.195 Network: closing ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.1.25:47014 ]
    [ 03/28/08 09:18:00.195 Network: attempting to accept ssl connection
    local address = 127.0.0.1:8334
    remote address = 127.0.0.1:57865 ]
    [ 03/28/08 09:18:00.195 Protocol: sending validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:18:00.224 Security: IceSSL: ignoring certificate validation failure:
    peer certificate does not contain `192.168.2.102' in its subjectAltName extension ]
    [ 03/28/08 09:18:00.224 Network: ssl connection established
    local address = 192.168.2.102:42817
    remote address = 192.168.2.102:8337 ]
    [ 03/28/08 09:18:00.224 Security: SSL summary for outgoing connection
    cipher = DHE-RSA-AES256-SHA
    bits = 256
    protocol = TLSv1
    local address = 192.168.2.102:42817
    remote address = 192.168.2.102:8337 ]
    [ 03/28/08 09:18:00.225 Protocol: received validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:18:00.225 Protocol: sending batch request
    message type = 1 (batch request)
    compression status = 2 (compressed; compress response, if any)
    message size = 127
    number of requests = 2
    request #0:
    identity = 42F20353-953A-410C-80A2-F5F2312BBE55
    facet =
    operation = ice_ping
    mode = 1 (nonmutating)
    context =
    request #1:
    identity = 42F20353-953A-410C-80A2-F5F2312BBE55
    facet =
    operation = ice_ping
    mode = 1 (nonmutating)
    context = ]
    [ 03/28/08 09:18:00.226 Protocol: sending batch request
    message type = 1 (batch request)
    compression status = 2 (compressed; compress response, if any)
    message size = 146
    number of requests = 8
    request #0:
    identity = 1D1A150A-B830-4305-8888-06939D904FCF
    facet =
    operation = ice_ping
    mode = 1 (nonmutating)
    context =
    request #1:
    identity = 1D1A150A-B830-4305-8888-06939D904FCF
    facet =
    operation = ice_ping
    mode = 1 (nonmutating)
    context =
    request #2:
    identity = 1D1A150A-B830-4305-8888-06939D904FCF
    facet =
    operation = ice_ping
    mode = 1 (nonmutating)
    context =
    request #3:
    identity = 1D1A150A-B830-4305-8888-06939D904FCF
    facet =
    operation = ice_ping
    mode = 1 (nonmutating)
    context =
    request #4:
    identity = 1D1A150A-B830-4305-8888-06939D904FCF
    facet =
    operation = ice_ping
    mode = 1 (nonmutating)
    context =
    request #5:
    identity = 1D1A150A-B830-4305-8888-06939D904FCF
    facet =
    operation = ice_ping
    mode = 1 (nonmutating)
    context =
    request #6:
    identity = 1D1A150A-B830-4305-8888-06939D904FCF
    facet =
    operation = ice_ping
    mode = 1 (nonmutating)
    context =
    request #7:
    identity = 1D1A150A-B830-4305-8888-06939D904FCF
    facet =
    operation = ice_ping
    mode = 1 (nonmutating)
    context = ]
    [ 03/28/08 09:18:00.233 Network: closing ssl connection
    local address = 127.0.0.1:8334
    remote address = 127.0.0.1:57865 ]

    more
  • [ 03/28/08 09:18:00.245 Network: closing ssl connection
    local address = 192.168.2.102:8334
    remote address = 192.168.102.21:23271 ]
    [ 03/28/08 09:18:00.245 Network: attempting to accept ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.2.102:16951 ]
    [ 03/28/08 09:18:00.245 Protocol: sending validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:18:00.257 Network: closing ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.2.102:16951 ]
    [ 03/28/08 09:18:00.258 Network: attempting to accept ssl connection
    local address = 127.0.0.1:8334
    remote address = 127.0.0.1:57868 ]
    [ 03/28/08 09:18:00.258 Protocol: sending validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:18:00.270 Network: closing ssl connection
    local address = 127.0.0.1:8334
    remote address = 127.0.0.1:57868 ]
    [ 03/28/08 09:18:00.270 Network: attempting to accept ssl connection
    local address = 192.168.2.102:8334
    remote address = 192.168.1.25:12175 ]
    [ 03/28/08 09:18:00.270 Protocol: sending validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:18:00.282 Network: closing ssl connection
    local address = 192.168.2.102:8334
    remote address = 192.168.1.25:12175 ]
    [ 03/28/08 09:18:00.282 Network: attempting to accept ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.2.102:16954 ]
    [ 03/28/08 09:18:00.282 Protocol: sending validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:18:00.294 Network: closing ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.2.102:16954 ]
    [ 03/28/08 09:18:00.294 Network: attempting to accept ssl connection
    local address = 127.0.0.1:8334
    remote address = 127.0.0.1:57871 ]
    [ 03/28/08 09:18:00.295 Protocol: sending validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:18:00.306 Network: closing ssl connection
    local address = 127.0.0.1:8334
    remote address = 127.0.0.1:57871 ]
    [ 03/28/08 09:18:00.307 Network: attempting to accept ssl connection
    local address = 192.168.2.102:8334
    remote address = 192.168.1.25:12176 ]
    [ 03/28/08 09:18:00.307 Protocol: sending validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:18:00.319 Network: closing ssl connection
    local address = 192.168.2.102:8334
    remote address = 192.168.1.25:12176 ]
    [ 03/28/08 09:18:00.319 Network: attempting to accept ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.2.102:16965 ]
    [ 03/28/08 09:18:00.319 Protocol: sending validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:18:00.331 Network: closing ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.2.102:16965 ]
    [ 03/28/08 09:18:00.331 Network: attempting to accept ssl connection
    local address = 127.0.0.1:8334
    remote address = 127.0.0.1:57874 ]
    [ 03/28/08 09:18:00.334 Protocol: sending validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:18:00.346 Network: closing ssl connection
    local address = 127.0.0.1:8334
    remote address = 127.0.0.1:57874 ]
    [ 03/28/08 09:18:00.346 Network: attempting to accept ssl connection
    local address = 192.168.2.102:8334
    remote address = 192.168.1.25:12177 ]
    [ 03/28/08 09:18:00.346 Protocol: sending validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:18:00.358 Network: closing ssl connection
    local address = 192.168.2.102:8334
    remote address = 192.168.1.25:12177 ]
    [ 03/28/08 09:18:00.359 Network: attempting to accept ssl connection
    local address = 192.168.2.102:8333
    remote address = 192.168.1.25:47205 ]
    [ 03/28/08 09:18:00.359 Protocol: sending validate connection
    message type = 3 (validate connection)
    compression status = 0 (not compressed; do not compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:19:27.474 Protocol: sending close connection
    message type = 4 (close connection)
    compression status = 1 (not compressed; compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:19:27.474 Protocol: sending close connection
    message type = 4 (close connection)
    compression status = 1 (not compressed; compress response, if any)
    message size = 14 ]
    [ 03/28/08 09:19:27.475 Network: shutting down ssl connection for writing
    local address = 192.168.2.102:42817
    remote address = 192.168.2.102:8337 ]
    [ 03/28/08 09:19:27.475 Network: closing ssl connection
    local address = 192.168.2.102:42817
    remote address = 192.168.2.102:8337 ]
    [ 03/28/08 09:19:27.476 Network: shutting down ssl connection for writing
    local address = 192.168.2.102:18341
    remote address = 192.168.2.102:8336 ]
    [ 03/28/08 09:19:27.476 Network: closing ssl connection
    local address = 192.168.2.102:18341
    remote address = 192.168.2.102:8336 ]

    these below are the telnet . sorry ,i don't know what i can do next for IceP, that is all when i type the telnet.

    [root@oracle ~]# telnet 192.168.1.25 8334
    Trying 192.168.1.25...
    Connected to 192.168.1.25 (192.168.1.25).
    Escape character is '^]'.


    and what info i could provide? what the next i could debug?
    could it the problem in my app in pub or sub?
  • matthew
    matthew NL, Canada
    It looks like your client shutdown non-gracefully during the SSL handshake and your server hasn't discovered that yet. Due to the way that TCP/IP works that could take quite some time. When using the thread pool concurrency model until the server discovers that your client has disappeared your server will not accept further connections, nor shutdown correctly (since graceful shutdown has to wait until all pending accepts are completed).

    To avoid this you should set an endpoint timeout -- -t <ms> on the endpoint in question. With the upcoming Ice 3.3 release this situation cannot occur.
  • I set the Ice.Override.Timeout at the client by without icestorm service.
    Oh ,i would have it try, it looks lile the key to the problem .
    Thanks for your help, hehe :)