Archived

This forum has been archived. Please start a new discussion on GitHub.

IceBox segfault

n2503v
n2503v
in Bug Reports
Ice-3.3.0, Linux

IceBox running IceStorm (the only service).
Fairly loaded, total number of clients and servers: up to 180, often subscribing and unsubscribing. This is the highest number we've ever tried. This is also the first time we've seen icebox segfault.

At some point the IceBox segfaulted with the following syslog message:
Apr 29 21:47:36 <hostname> kernel: [416257.979832] icebox[19691]: segfault at 24 ip b7ad7305 sp af7fe950 error 4 in libstdc++.so.6.0.10[b7a41000+e2000]

I have backtraces from the core dump (all 19 threads) but they don't fit into this post. Let me know if you want to examine them.

icebox.cfg 
IceStorm.TopicManager.Proxy=IceStorm/TopicManager:default -p 10000
Ice.Warn.Connections=1
Ice.PrintAdapterReady=1
Ice.MessageSizeMax=30000
IceBox.ServiceManager.Endpoints=tcp -p 9998
IceBox.InheritProperties=1
IceBox.LoadOrder=IceStorm
IceBox.Service.IceStorm=IceStormService,33:createIceStorm --Ice.Config=/etc/marathon/icestorm.cfg
IceBox.UseSharedCommunicator.IceStormService=1

icestorm.cfg 
IceStorm.TopicManager.Endpoints=default -h localhost -p 10000: default -h <hostname> -p 10000
Ice.Warn.Connections=1
Ice.Warn.Dispatch=2
Ice.Warn.Endpoints=1
Ice.Warn.AMICallback=1
Ice.Warn.UnknownProperties=1
Ice.ThreadPool.Client.Size=1
Ice.ThreadPool.Client.SizeWarn=4
Ice.ThreadPool.Client.SizeMax=4
Ice.ThreadPool.Server.Size=1
Ice.ThreadPool.Server.SizeWarn=3
Ice.ThreadPool.Server.SizeMax=3
IceStorm.Send.Timeout=60000
IceStorm.Trace.TopicManager=1
IceStorm.Trace.Topic=2
IceStorm.Trace.Subscriber=1
IceStorm.Publish.Endpoints=default
IceStorm.Publish.ThreadPool.Size=1
IceStorm.Publish.ThreadPool.SizeWarn=5
IceStorm.Publish.ThreadPool.SizeMax=5
IceStorm.TopicManager.ThreadPool.Size=1
IceStorm.TopicManager.ThreadPool.SizeWarn=2
IceStorm.TopicManager.ThreadPool.SizeMax=3
Freeze.DbEnv.IceStorm.DbHome=/home/mrsys/sys/icestorm/stormdb

Thanks for your help,
alex

Comments

  • benoit
    benoit Rennes, France
    Hi Alex,

    Yes, seeing the stack traces would be helpful. Can you attach them to this thread with a text file? Also, any reasons why you set the thread pool sizes of IceBox and IceStorm?

    Cheers,
    Benoit.
  • n2503v
    n2503v
    Hi Benoit,
    the traces are attached.

    Regarding the thread pool sizes, I am not entire sure. The config file has evolved from the pre-Ice3, when the configs were different. Now that I look at it, it seems that the following would apply to the IceBox but would also get inherited by IceStorm (is this correct?).

    Ice.ThreadPool.Client.Size=1
    Ice.ThreadPool.Client.SizeWarn=4
    Ice.ThreadPool.Client.SizeMax=4
    Ice.ThreadPool.Server.Size=1
    Ice.ThreadPool.Server.SizeWarn=3
    Ice.ThreadPool.Server.SizeMax=3

    cheers,
    alex
  • matthew
    matthew NL, Canada
    n2503v wrote: »
    ...
    Regarding the thread pool sizes, I am not entire sure. The config file has evolved from the pre-Ice3, when the configs were different. Now that I look at it, it seems that the following would apply to the IceBox but would also get inherited by IceStorm (is this correct?).
    ...

    It depends how you've configured IceBox & IceStorm, but if you've used separate configuration files, and have not set IceBox.InheritProperties then IceStorm will not inherit the IceBox configuration properties.

    In general, with IceStorm you do not need to configure the thread pools with Ice 3.3, since they default to appropriate sizes.
  • matthew
    matthew NL, Canada
    The crash you've reported is a bug in IceStorm. The simplest workaround is not to trace the subscribers (that is set IceStorm.Trace.Topic=0). If you need a fix let us know, and we'll prepare a patch.
  • n2503v
    n2503v
    Hi Matthew,
    thanks for tracking it down.

    The work around will do us for now.
    In general, with IceStorm you do not need to configure the thread pools with Ice 3.3, since they default to appropriate sizes.

    I'll re-read your article on IceStorm and will get back to you if I still have questions on thread pool configuration.

    Cheers,
    Alex

Categories