icegridnode error message and crash

jae

I'm running Ice 3.1.1. I've been up and running for a number of months. Looking back over my logs (which are piped to syslog and formated by syslog-ng), I see numerous messages like this:

2007-10-30T12:20:23-0700 err /usr/bin/icegridnode[14502]: a node with the same name is already registered and active
2007-10-30T12:20:28-0700 err /usr/bin/icegridnode[14502]: a node with the same name is already registered and active
2007-10-30T12:20:33-0700 err /usr/bin/icegridnode[14502]: a node with the same name is already registered and active

These messages are posted every five seconds. There does not appear to be any additional information. Up until today, I have not noticed any problems caused by this.

Looking over the log, I notice this in the time just before this started:

2007-09-28T01:34:51-0700 warning /usr/bin/icegridnode[14502]: unexpected exception while reaping node session: TcpTransceiver.cpp:291: Ice::SocketException: socket exception: Connection timed out
2007-09-28T01:34:59-0700 warning /usr/bin/icegridnode[14502]: couldn't contact the IceGrid registry: Network.cpp:841: Ice::DNSException: DNS error: Name or service not known host: cl2

These errors repeated for a number of hours, after which time, the node errors started to appear. Interestingly, about 7 hours later, I started to see the same DNS errors on an independent IceGridRegistry. Both cleared up at about the same time and then I started seeing the messages about "a node with the same name".

I've now restarted icegridnode in both locations and the messages have stopped.

In one of the two places where I was running an icegrid registry, icegridnode crashed. This is what prompted me to notice the above.

Here is the Backtrace, etc. Does it appear that there is any relation? Or, would you suspect it to be something completely unrelated?

*** glibc detected *** /usr/bin/icegridnode: malloc(): memory corruption: 0x00000000008007c1 ***
======= Backtrace: =========
/lib/libc.so.6[0x2b3cf1107b0d]
/lib/libc.so.6[0x2b3cf11099a6]
/lib/libc.so.6(malloc+0x7d)[0x2b3cf110b4fd]
/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/libstdc++.so.6(_Znwm+0x1d)[0x2b3cf0cf6a3d]
/usr/bin/icegridnode[0x50c151]
/usr/bin/icegridnode[0x50cfd0]
/usr/bin/icegridnode[0x50daa9]
/usr/lib/libIce.so.31(_ZNK3Ice7Locator18___findAdapterByIdERN11IceInternal8IncomingERKNS_7CurrentE+0x126)[0x2b3cf0803936]
/usr/lib/libIce.so.31(_ZN11IceInternal8Incoming6invokeERKNS_6HandleINS_14ServantManagerEEE+0xc80)[0x2b3cf07db510]
/usr/lib/libIce.so.31(_ZN3Ice11ConnectionI9invokeAllERN11IceInternal11BasicStreamEiihRKNS1_6HandleINS1_14ServantManagerEEERKNS4_INS_13ObjectAdapterEEE+0x17e)[0x2b3cf07b2c6e]
/usr/lib/libIce.so.31(_ZN3Ice11ConnectionI7messageERN11IceInternal11BasicStreamERKNS1_6HandleINS1_10ThreadPoolEEE+0x137)[0x2b3cf07b9a47]
/usr/lib/libIce.so.31(_ZN11IceInternal10ThreadPool3runEv+0x997)[0x2b3cf08817d7]
/usr/lib/libIce.so.31(_ZN11IceInternal10ThreadPool18EventHandlerThread3runEv+0x62)[0x2b3cf0882d22]
/usr/lib/libIceUtil.so.31[0x2b3cf0a16456]
/lib/libpthread.so.0[0x2b3cf0b28135]
/lib/libc.so.6(__clone+0x6d)[0x2b3cf115b62d]

jae

Memory Map to above

The forum software complained that my post was too long. Here's a portion of the memory map section of the dump:

======= Memory map: ========
00400000-005c2000 r-xp 00000000 09:00 75500791                           /usr/bin/icegridnode
006c2000-006d9000 rw-p 001c2000 09:00 75500791                           /usr/bin/icegridnode
006d9000-00837000 rw-p 006d9000 00:00 0                                  [heap]
40000000-40001000 ---p 40000000 00:00 0
40001000-40801000 rw-p 40001000 00:00 0
40801000-40802000 ---p 40801000 00:00 0
40802000-41002000 rw-p 40802000 00:00 0
41002000-41003000 ---p 41002000 00:00 0
41003000-41803000 rw-p 41003000 00:00 0
41803000-41804000 ---p 41803000 00:00 0
41804000-42004000 rw-p 41804000 00:00 0
42004000-42005000 ---p 42004000 00:00 0
42005000-42805000 rw-p 42005000 00:00 0
42805000-42806000 ---p 42805000 00:00 0
42806000-43006000 rw-p 42806000 00:00 0
43006000-43007000 ---p 43006000 00:00 0
43007000-43807000 rw-p 43007000 00:00 0
43807000-43808000 ---p 43807000 00:00 0
43808000-44008000 rw-p 43808000 00:00 0
44008000-44009000 ---p 44008000 00:00 0
44009000-44809000 rw-p 44009000 00:00 0
44809000-4480a000 ---p 44809000 00:00 0
4480a000-4500a000 rw-p 4480a000 00:00 0
4500a000-4500b000 ---p 4500a000 00:00 0
4500b000-4580b000 rw-p 4500b000 00:00 0
4580b000-4580c000 ---p 4580b000 00:00 0
4580c000-4600c000 rw-p 4580c000 00:00 0
4600c000-4600d000 ---p 4600c000 00:00 0
4600d000-4680d000 rw-p 4600d000 00:00 0
4680d000-4680e000 ---p 4680d000 00:00 0
4680e000-4700e000 rw-p 4680e000 00:00 0
4700e000-4700f000 ---p 4700e000 00:00 0
4700f000-4780f000 rw-p 4700f000 00:00 0
4780f000-47810000 ---p 4780f000 00:00 0
47810000-48010000 rw-p 47810000 00:00 0
48010000-48011000 ---p 48010000 00:00 0
48011000-48811000 rw-p 48011000 00:00 0
48811000-48812000 ---p 48811000 00:00 0
48812000-49012000 rw-p 48812000 00:00 0
49012000-49013000 ---p 49012000 00:00 0
49013000-49813000 rw-p 49013000 00:00 0
49813000-49814000 ---p 49813000 00:00 0
49814000-4a014000 rw-p 49814000 00:00 0
4a014000-4a015000 ---p 4a014000 00:00 0
4a015000-4a815000 rw-p 4a015000 00:00 0
4a815000-4a816000 ---p 4a815000 00:00 0
4a816000-4b016000 rw-p 4a816000 00:00 0
4b016000-4b017000 ---p 4b016000 00:00 0
4b017000-4b817000 rw-p 4b017000 00:00 0
4b817000-4b818000 ---p 4b817000 00:00 0
4b818000-4c018000 rw-p 4b818000 00:00 0

jae

Last section of Memory Map

2aaaaaaab000-2aaaaaaad000 rw-p 2aaaaaaab000 00:00 0
2aaaaaabc000-2aaaaaac6000 r-xp 00000000 09:00 79179556                  /lib64/libnss_files-2.5.so
2aaaaaac6000-2aaaaabc5000 ---p 0000a000 09:00 79179556                  /lib64/libnss_files-2.5.so
2aaaaabc5000-2aaaaabc7000 rw-p 00009000 09:00 79179556                  /lib64/libnss_files-2.5.so
2aaaaabc7000-2aaaaabcb000 r-xp 00000000 09:00 79179554                  /lib64/libnss_dns-2.5.so
2aaaaabcb000-2aaaaacca000 ---p 00004000 09:00 79179554                  /lib64/libnss_dns-2.5.so
2aaaaacca000-2aaaaaccc000 rw-p 00003000 09:00 79179554                  /lib64/libnss_dns-2.5.so
2aaaaaccc000-2aaaaacdc000 r-xp 00000000 09:00 79179552                  /lib64/libresolv-2.5.so
2aaaaacdc000-2aaaaaddb000 ---p 00010000 09:00 79179552                  /lib64/libresolv-2.5.so
2aaaaaddb000-2aaaaaddd000 rw-p 0000f000 09:00 79179552                  /lib64/libresolv-2.5.so
2aaaaaddd000-2aaaaaddf000 rw-p 2aaaaaddd000 00:00 0
2aaaac000000-2aaaac06e000 rw-p 2aaaac000000 00:00 0
2aaaac06e000-2aaab0000000 ---p 2aaaac06e000 00:00 0
2b3cef685000-2b3cef6a0000 r-xp 00000000 09:00 8388924                   /var/tmp/portage/sys-libs/glibc-2.5-r4/image/lib64/ld-2.5.so (deleted)
2b3cef6a0000-2b3cef6a1000 rw-p 2b3cef6a0000 00:00 0
2b3cef6b0000-2b3cef6b1000 rw-p 2b3cef6b0000 00:00 0
2b3cef7a0000-2b3cef7a1000 r--p 0001b000 09:00 8388924                   /var/tmp/portage/sys-libs/glibc-2.5-r4/image/lib64/ld-2.5.so (deleted)
2b3cef7a1000-2b3cef7a2000 rw-p 0001c000 09:00 8388924                   /var/tmp/portage/sys-libs/glibc-2.5-r4/image/lib64/ld-2.5.so (deleted)
2b3cef7a2000-2b3cef904000 r-xp 00000000 09:00 83899677                  /usr/lib64/libIceGrid.so.3.1.1
2b3cef904000-2b3cefa04000 ---p 00162000 09:00 83899677                  /usr/lib64/libIceGrid.so.3.1.1
2b3cefa04000-2b3cefa23000 rw-p 00162000 09:00 83899677                  /usr/lib64/libIceGrid.so.3.1.1
2b3cefa23000-2b3cefaa4000 r-xp 00000000 09:00 83899674                  /usr/lib64/libIceStormService.so.3.1.1
2b3cefaa4000-2b3cefba4000 ---p 00081000 09:00 83899674                   /usr/lib64/libIceStormService.so.3.1.1
2b3cefba4000-2b3cefbae000 rw-p 00081000 09:00 83899674                   /usr/lib64/libIceStormService.so.3.1.1
2b3cefbae000-2b3cefbc0000 r-xp 00000000 09:00 83899662                   /usr/lib64/libIceBox.so.3.1.1
2b3cefbc0000-2b3cefcbf000 ---p 00012000 09:00 83899662                   /usr/lib64/libIceBox.so.3.1.1
2b3cefcbf000-2b3cefcc1000 rw-p 00011000 09:00 83899662                   /usr/lib64/libIceBox.so.3.1.1
2b3cefcc1000-2b3cefcc2000 rw-p 2b3cefcc1000 00:00 0
2b3cefcc2000-2b3cefd26000 r-xp 00000000 09:00 83899665                  /usr/lib64/libGlacier2.so.3.1.1
2b3cefd26000-2b3cefe26000 ---p 00064000 09:00 83899665                   /usr/lib64/libGlacier2.so.3.1.1
2b3cefe26000-2b3cefe31000 rw-p 00064000 09:00 83899665                   /usr/lib64/libGlacier2.so.3.1.1
2b3cefe31000-2b3cefe79000 r-xp 00000000 09:00 83899668                   /usr/lib64/libIcePatch2.so.3.1.1
2b3cefe79000-2b3ceff79000 ---p 00048000 09:00 83899668                   /usr/lib64/libIcePatch2.so.3.1.1
2b3ceff79000-2b3ceff7d000 rw-p 00048000 09:00 83899668                   /usr/lib64/libIcePatch2.so.3.1.1
2b3ceff7d000-2b3cefff7000 r-xp 00000000 09:00 83899659                   /usr/lib64/libFreeze.so.3.1.1
2b3cefff7000-2b3cf00f6000 ---p 0007a000 09:00 83899659                   /usr/lib64/libFreeze.so.3.1.1
2b3cf00f6000-2b3cf00fc000 rw-p 00079000 09:00 83899659                   /usr/lib64/libFreeze.so.3.1.1
2b3cf00fc000-2b3cf00fd000 rw-p 2b3cf00fc000 00:00 0
2b3cf00fd000-2b3cf0109000 r-xp 00000000 09:00 83899653                   /usr/lib64/libIceXML.so.3.1.1
2b3cf0109000-2b3cf0208000 ---p 0000c000 09:00 83899653                   /usr/lib64/libIceXML.so.3.1.1
2b3cf0208000-2b3cf0209000 rw-p 0000b000 09:00 83899653                   /usr/lib64/libIceXML.so.3.1.1
2b3cf0209000-2b3cf0248000 r-xp 00000000 09:00 83899656                   /usr/lib64/libIceSSL.so.3.1.1
2b3cf0248000-2b3cf0348000 ---p 0003f000 09:00 83899656                   /usr/lib64/libIceSSL.so.3.1.1
2b3cf0348000-2b3cf034b000 rw-p 0003f000 09:00 83899656                   /usr/lib64/libIceSSL.so.3.1.1
2b3cf034b000-2b3cf038c000 r-xp 00000000 09:00 58970453                   /usr/lib64/libssl.so.0.9.8
2b3cf038c000-2b3cf048c000 ---p 00041000 09:00 58970453                   /usr/lib64/libssl.so.0.9.8
2b3cf048c000-2b3cf0492000 rw-p 00041000 09:00 58970453                   /usr/lib64/libssl.so.0.9.8
2b3cf0492000-2b3cf0493000 rw-p 2b3cf0492000 00:00 0
2b3cf0493000-2b3cf05df000 r-xp 00000000 09:00 58970450                   /usr/lib64/libcrypto.so.0.9.8
2b3cf05df000-2b3cf06de000 ---p 0014c000 09:00 58970450                   /usr/lib64/libcrypto.so.0.9.8
2b3cf06de000-2b3cf0702000 rw-p 0014b000 09:00 58970450                   /usr/lib64/libcrypto.so.0.9.8
2b3cf0702000-2b3cf0705000 rw-p 2b3cf0702000 00:00 0
2b3cf0705000-2b3cf08e1000 r-xp 00000000 09:00 83899650                   /usr/lib64/libIce.so.3.1.1
2b3cf08e1000-2b3cf09e0000 ---p 001dc000 09:00 83899650                   /usr/lib64/libIce.so.3.1.1
2b3cf09e0000-2b3cf09f3000 rw-p 001db000 09:00 83899650                   /usr/lib64/libIce.so.3.1.1
2b3cf09f3000-2b3cf09f4000 rw-p 2b3cf09f3000 00:00 0
2b3cf09f4000-2b3cf0a20000 r-xp 00000000 09:00 83899686                   /usr/lib64/libIceUtil.so.3.1.1
2b3cf0a20000-2b3cf0b1f000 ---p 0002c000 09:00 83899686                   /usr/lib64/libIceUtil.so.3.1.1
2b3cf0b1f000-2b3cf0b21000 rw-p 0002b000 09:00 83899686                   /usr/lib64/libIceUtil.so.3.1.1
2b3cf0b21000-2b3cf0b22000 rw-p 2b3cf0b21000 00:00 0
2b3cf0b22000-2b3cf0b37000 r-xp 00000000 09:00 8388958                    /var/tmp/portage/sys-libs/glibc-2.5-r4/image/lib64/libpthread-2.5.so (deleted)
2b3cf0b37000-2b3cf0c36000 ---p 00015000 09:00 8388958                    /var/tmp/portage/sys-libs/glibc-2.5-r4/image/lib64/libpthread-2.5.so (deleted)
2b3cf0c36000-2b3cf0c37000 r--p 00014000 09:00 8388958                   /var/tmp/portage/sys-libs/glibc-2.5-r4/image/lib64/libpthread-2.5.so (deleted)
2b3cf0c37000-2b3cf0c38000 rw-p 00015000 09:00 8388958                   /var/tmp/portage/sys-libs/glibc-2.5-r4/image/lib64/libpthread-2.5.so (deleted)
2b3cf0c38000-2b3cf0c3c000 rw-p 2b3cf0c38000 00:00 0
2b3cf0c3c000-2b3cf0d1f000 r-xp 00000000 09:00 92276738                   /usr/lib64/gcc/x86_64-pc-linux-gnu/4.1.2/libstdc++.so.6.0.8
2b3cf0d1f000-2b3cf0e1f000 ---p 000e3000 09:00 92276738                   /usr/lib64/gcc/x86_64-pc-linux-gnu/4.1.2/libstdc++.so.6.0.8
2b3cf0e1f000-2b3cf0e25000 r--p 000e3000 09:00 92276738                  /usr/lib64/gcc/x86_64-pc-linux-gnu/4.1.2/libstdc++.so.6.0.8
2b3cf0e25000-2b3cf0e28000 rw-p 000e9000 09:00 92276738                  /usr/lib64/gcc/x86_64-pc-linux-gnu/4.1.2/libstdc++.so.6.0.8
2b3cf0e28000-2b3cf0e3a000 rw-p 2b3cf0e28000 00:00 0
2b3cf0e3a000-2b3cf0e8e000 r-xp 00000000 09:00 8388937                   /var/tmp/portage/sys-libs/glibc-2.5-r4/image/lib64/libm-2.5.so (deleted)
2b3cf0e8e000-2b3cf0f8d000 ---p 00054000 09:00 8388937                   /var/tmp/portage/sys-libs/glibc-2.5-r4/image/lib64/libm-2.5.so (deleted)
2b3cf0f8d000-2b3cf0f8f000 rw-p 00053000 09:00 8388937                   /var/tmp/portage/sys-libs/glibc-2.5-r4/image/lib64/libm-2.5.so (deleted)
2b3cf0f8f000-2b3cf0f90000 rw-p 2b3cf0f8f000 00:00 0
2b3cf0f90000-2b3cf0f9c000 r-xp 00000000 09:00 8388887                   /lib64/libgcc_s.so.1
2b3cf0f9c000-2b3cf109c000 ---p 0000c000 09:00 8388887                   /lib64/libgcc_s.so.1
2b3cf109c000-2b3cf109d000 rw-p 0000c000 09:00 8388887                   /lib64/libgcc_s.so.1
2b3cf109d000-2b3cf11cf000 r-xp 00000000 09:00 8388929                   /var/tmp/portage/sys-libs/glibc-2.5-r4/image/lib64/libc-2.5.so (deleted)
2b3cf11cf000-2b3cf12ce000 ---p 00132000 09:00 8388929                   /var/tmp/portage/sys-libs/glibc-2.5-r4/image/lib64/libc-2.5.so (deleted)
2b3cf12ce000-2b3cf12d1000 r--p 00131000 09:00 8388929                   /var/tmp/portage/sys-libs/glibc-2.5-r4/image/lib64/libc-2.5.so (deleted)
2b3cf12d1000-2b3cf12d3000 rw-p 00134000 09:00 8388929                   /var/tmp/portage/sys-libs/glibc-2.5-r4/image/lib64/libc-2.5.so (deleted)
2b3cf12d3000-2b3cf12d9000 rw-p 2b3cf12d3000 00:00 0
2b3cf12d9000-2b3cf12e7000 r-xp 00000000 09:00 8388987                   /lib64/libbz2.so.1.0.4
2b3cf12e7000-2b3cf13e6000 ---p 0000e000 09:00 8388987                   /lib64/libbz2.so.1.0.4
2b3cf13e6000-2b3cf13e8000 rw-p 0000d000 09:00 8388987                   /lib64/libbz2.so.1.0.4
2b3cf13e8000-2b3cf14eb000 r-xp 00000000 09:00 100977953                 /usr/lib64/libdb_cxx-4.3.so
2b3cf14eb000-2b3cf15ea000 ---p 00103000 09:00 100977953                 /usr/lib64/libdb_cxx-4.3.so
2b3cf15ea000-2b3cf15f0000 rw-p 00102000 09:00 100977953                 /usr/lib64/libdb_cxx-4.3.so
2b3cf15f0000-2b3cf1611000 r-xp 00000000 09:00 101905933                 /usr/lib64/libexpat.so.1.5.2
2b3cf1611000-2b3cf1710000 ---p 00021000 09:00 101905933                 /usr/lib64/libexpat.so.1.5.2
2b3cf1710000-2b3cf1713000 rw-p 00020000 09:00 101905933                 /usr/lib64/libexpat.so.1.5.2
2b3cf1713000-2b3cf1714000 rw-p 2b3cf1713000 00:00 0
2b3cf1714000-2b3cf1716000 r-xp 00000000 09:00 8388961                   /var/tmp/portage/sys-libs/glibc-2.5-r4/image/lib64/libdl-2.5.so (deleted)
2b3cf1716000-2b3cf1816000 ---p 00002000 09:00 8388961                   /var/tmp/portage/sys-libs/glibc-2.5-r4/image/lib64/libdl-2.5.so (deleted)
2b3cf1816000-2b3cf1818000 rw-p 00002000 09:00 8388961                   /var/tmp/portage/sys-libs/glibc-2.5-r4/image/lib64/libdl-2.5.so (deleted)
2b3cf1818000-2b3cf181a000 rw-p 2b3cf1818000 00:00 0
7fffbb410000-7fffbb425000 rw-p 7fffbb410000 00:00 0                     [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                 [vdso]

benoit

Hi,

Can you upgrade to Ice 3.2.1? I'm confident that the issue where the registry warns about "a node with the same name is already registered and active" is fixed in the latest IceGrid version.

Cheers,
Benoit.

jae

3.2 Upgrade Questions

How about the crash? It turns out that a co-worker has seen multiple icegridnode crashes with the same stack. Is that a known problem? Is it fixed in 3.2.1?

A co-worker has looked over the upgrade faq and come to the conclusion that an upgrade would require some amount of re-coding due to changes. At this point, we have a fair amount of Ice code, so we are holding off until we have some project down time (or until we find a need that forces our hand).

Related question; can 3.1.1 and 3.2.1 inter-operate? That is, if we upgrade a registry and collection of nodes to 3.2.1 and leave another at 3.1.1 will they be able to communicate?

Thanks

marc

I'm afraid here in these forums we can only support the latest version of Ice. This includes providing information about version compatibility, or information about what problems existed in older versions of Ice.

Since it seems that you are using Ice for critical parts of your business operations, you should consider a commercial support agreement. Please contact us at info@zeroc.com if you are interested.

jae

patch

I've applied the patch posted here:

http://www.zeroc.com/forums/patches/2745-patch-1-ice-3-1-1-fixes-icegrid-locator-memory-corruption.html

This seems to have corrected the crash.