Archived
This forum has been archived. Please start a new discussion on GitHub.
Assertion failed in IceDB/FreezeTypes.cpp
Hello,
We have regular unexpected failures of our IceStorm setup. Diving into the logs, we found an occurrence of the line below every time we had a failure:
We run two icestorm replica groups. Each of them have 4 icestorm instances, distributed on 2 servers (2*2). The assertion is raised only on the last instance of the group. We have had the problem for some time and we haven't had a single assertion failed entry in all the icestorm log files except the last one of the group.
The problem is only appearing when we are registering our subscribers. Those subscribers register simultaneously and we first had the problem when increasing their number.
I tried to reproduce the problem in a lab setup but I have not been able to reproduce it so far. Do you have any clue on how to find/resolve the problem ?
Our setup:
* Linux RHEL5, x86_64
* Ice 3.4.1, home-compiled with optimizations disabled
* FreezeDb linked with bdb provided with our distro (RHEL5): libdb_cxx-4.3.so
IceStorm configuration files:
We have regular unexpected failures of our IceStorm setup. Diving into the logs, we found an occurrence of the line below every time we had a failure:
-- 09/25/11 16:30:10.689 RTCSStorm-us-3-IceStorm: Topic: mainDispatchTopic: subscribeAndGetPublisher: B8B9BFC0-CD32-42C5-A7CD-19798F87B9AE icebox: ../../../src/IceDB/FreezeTypes.cpp:118: virtual void FreezeDB::DatabaseConnection::beginTransaction(): Assertion `!_connection->currentTransaction()' failed.
We run two icestorm replica groups. Each of them have 4 icestorm instances, distributed on 2 servers (2*2). The assertion is raised only on the last instance of the group. We have had the problem for some time and we haven't had a single assertion failed entry in all the icestorm log files except the last one of the group.
The problem is only appearing when we are registering our subscribers. Those subscribers register simultaneously and we first had the problem when increasing their number.
I tried to reproduce the problem in a lab setup but I have not been able to reproduce it so far. Do you have any clue on how to find/resolve the problem ?
Our setup:
* Linux RHEL5, x86_64
* Ice 3.4.1, home-compiled with optimizations disabled
* FreezeDb linked with bdb provided with our distro (RHEL5): libdb_cxx-4.3.so
IceStorm configuration files:
# Configuration file (12/23/10 08:32:26.035) # Server configuration Ice.Admin.ServerId=RTCSStorm-us-0 Ice.Admin.Endpoints=tcp -h 127.0.0.1 Ice.ProgramName=RTCSStorm-us-0 IceBox.Service.IceStorm=IceStormService,34:createIceStorm --Ice.Config\='/opt/user/ice-node/node-data/servers/RTCSStorm-us-0/config/config_IceStorm' IceBox.InstanceName=RTCSStorm-us-0 IceBox.LoadOrder=IceStorm\ # Server descriptor properties Ice.StdOut=/opt/collector/ice-node/icestorm/RTCSStorm-us-0.log Ice.StdErr=/opt/collector/ice-node/icestorm/RTCSStorm-us-0.log Ice.Default.Locator=IceGrid/Locator:tcp -p 4061 -h rtcs-registry-master.data.cfm.fr # Configuration file (12/23/10 08:32:26.035) # Object adapter IceStorm.Node IceStorm.Node.Endpoints=default -h usen105l29.us.cfm.fr IceStorm.Node.AdapterId=RTCSStorm-us-0.IceStorm.IceStorm.Node # Object adapter IceStorm.Publish IceStorm.Publish.Endpoints=default -h usen105l29.us.cfm.fr IceStorm.Publish.AdapterId=RTCSStorm-us-0.IceStorm.IceStorm.Publish IceStorm.Publish.ReplicaGroupId=RTCSStorm-us-PublishReplicaGroup # Object adapter IceStorm.TopicManager IceStorm.TopicManager.Endpoints=default -h usen105l29.us.cfm.fr IceStorm.TopicManager.AdapterId=RTCSStorm-us-0.IceStorm.IceStorm.TopicManager IceStorm.TopicManager.ReplicaGroupId=RTCSStorm-us-TopicManagerReplicaGroup # Service descriptor properties IceStorm.InstanceName=RTCSStorm-us IceStorm.Trace.TopicManager=2 IceStorm.Trace.Topic=1 IceStorm.Trace.Subscriber=1 IceStorm.Trace.Election=1 IceStorm.NodeId=0 Ice.StdOut=/opt/user/ice-node/icestorm/RTCSStorm-us-0.log Ice.StdErr=/opt/collector/ice-node/icestorm/RTCSStorm-us-0.log Freeze.DbEnv.IceStorm.DbHome=/opt/collector/ice-node/icestorm/db-RTCSStorm-us-0/ IceStorm.Trace.Network=1 Ice.Default.Locator=IceGrid/Locator:tcp -p 4061 -h rtcs-registry-master.data.cfm.fr
0
Comments
-
Hi,
Could you get the stack trace of the assert from the core file and post it here? You can enable core files by running "ulimit -c unlimited" in the environment where the process is running.
Cheers,
Benoit.0