Archived
IceStorm, db log file unreadable
We use IceBox with IceStorm on Windows with C#. We observed since a long time a sporadically failure while starting up IceBox. We can not start IceStorm because it seems there is a problem with the database.
We saw following log entry in the Ice.LogFile:
- 07/28/16 20:50:29.114 C:/Jenkins/VirtualGCTest/TestEnvironment/Current_TaTU_Version/IceBox/icebox.exe-IceStorm: Freeze.DbEnv: checkpointing environment "IceStorm" -- 07/28/16 20:50:29.139 C:/Jenkins/VirtualGCTest/TestEnvironment/Current_TaTU_Version/IceBox/icebox.exe-IceStorm: Berkeley DB: DbEnv "IceStorm": BDB2520 C:\Users\BATman\AppData\Local\Temp\IceStorm1\log.0000000001: log file unreadable: Operation not permitted -- 07/28/16 20:50:29.139 C:/Jenkins/VirtualGCTest/TestEnvironment/Current_TaTU_Version/IceBox/icebox.exe-IceStorm: Berkeley DB: DbEnv "IceStorm": BDB0061 PANIC: Operation not permitted -- 07/28/16 20:52:29.145 C:/Jenkins/VirtualGCTest/TestEnvironment/Current_TaTU_Version/IceBox/icebox.exe-IceStorm: Freeze.DbEnv: checkpointing environment "IceStorm" -- 07/28/16 20:52:29.145 C:/Jenkins/VirtualGCTest/TestEnvironment/Current_TaTU_Version/IceBox/icebox.exe-IceStorm: Berkeley DB: DbEnv "IceStorm": BDB0060 PANIC: fatal region error detected; run recovery -! 07/28/16 20:52:29.145 C:/Jenkins/VirtualGCTest/TestEnvironment/Current_TaTU_Version/IceBox/icebox.exe-IceStorm: warning: checkpoint on DbEnv "IceStorm" raised DbException: DbEnv::txn_checkpoint: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
That are our startup arguments:
string[] config =
{
// IceBox settings
"--Ice.IPv6=0",
"--Ice.LogFile=" + logfilePath,
"--Ice.Trace.Locator=2",
"--Ice.Trace.Network=2",
"--Ice.Trace.Protocol=0",
"--Ice.Trace.Retry=2",
"--Ice.Trace.Slicing=1",
"--Ice.Trace.ThreadPool=0",
"--Ice.Admin.Endpoints=\"" + iceConfig.ServiceManagerEndpoints + "\"",
"--IceBox.PrintServicesReady=Services",
// IceStorm settings
"--IceStorm.TopicManager.Endpoints=\"" + iceConfig.TopicManagerEndpoints + "\"",
"--IceStorm.Publish.Endpoints=\"" + iceConfig.TopicPublisherEndpoints + "\"",
"--IceStorm.InstanceName=TATU",
"--IceStorm.Trace.Election=1",
"--IceStorm.Trace.Replication=1",
"--IceStorm.Trace.Subscriber=2",
"--IceStorm.Trace.Topic=2",
"--IceStorm.Trace.TopicManager=1",
"--IceStorm.Flush.Timeout=100",
"--IceBox.Service.IceStorm=\"IceStormService," + IceVersionExtra + ":createIceStorm",
"--Freeze.DbEnv.IceStorm.DbHome=" + dbDir,
"--Freeze.Trace.DbEnv=2",
"--Freeze.Trace.Evictor=3",
"--Freeze.Trace.Map=1",
"--Freeze.Trace.Transaction=0",
"\""
};
Can you help us with this issue?
Comments
-
Hi Aldo,
I would start by putting the IceStorm Freeze DbHome in non-temporary directory, owned by the user running IceStorm:
Berkeley DB: DbEnv "IceStorm": BDB2520 C:\Users\BATman\AppData\Local\Temp\IceStorm1\log.0000000001: log file unreadable: Operation not permitted
Cheers,
Bernard0 -
Hi Bernard
Thanks for your answer. I moved the Freeze Db Home to ProgramData but the issue occurs still from time to time.
Berkeley DB: DbEnv "IceStorm": BDB2520 C:\ProgramData\Schindler\TaTU\db\IceStorm1\log.0000000001: log file unreadable: Operation not permitted Berkeley DB: DbEnv "IceStorm": BDB0061 PANIC: Operation not permitted Freeze.DbEnv: checkpointing environment "IceStorm" Berkeley DB: DbEnv "IceStorm": BDB0060 PANIC: fatal region error detected; run recovery warning: checkpoint on DbEnv "IceStorm" raised DbException: DbEnv::txn_checkpoint: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery Freeze.DbEnv: checkpointing environment "IceStorm" Berkeley DB: DbEnv "IceStorm": BDB0060 PANIC: fatal region error detected; run recovery warning: checkpoint on DbEnv "IceStorm" raised DbException: DbEnv::txn_checkpoint: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery Freeze.DbEnv: checkpointing environment "IceStorm" Berkeley DB: DbEnv "IceStorm": BDB0060 PANIC: fatal region error detected; run recovery warning: checkpoint on DbEnv "IceStorm" raised DbException: DbEnv::txn_checkpoint: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
Have you got any other ideas?
0 -
Hi Aldo,
The problem you're seeing is not the initial startup (IceStorm starts up fine), but the periodic checkpointing of the Freeze database. The default checkpoint period is 2 minutes, and it appears to fail each time. See:
and
http://docs.oracle.com/cd/E17076_05/html/programmer_reference/transapp_checkpoint.html
As a work-around, you could disable checkpoints (by setting CheckpointPeriod to 0), but obviously these checkpoints should not fail in the first place.
Could you specify the version of Ice you're using? Are you using our binary distribution or did you build Ice from sources? Does the user running IceBox have full permissions to this C:\ProgramData\Schindler\TaTU\db\IceStorm1 directory? How do you start your IceBox server? Do you start icebox as a Windows service?
Best regards,
Bernard0 -
Hi Bernhard
You're supposing that IceStorm is starting up fine in spite of these database errors we see in the log. So, you have to know that these errors do not occur on every start-up of IceStorm. But every time they appear we cannot access the TopicManager which is actually our main problem. Therefore, we assume that there's a relation between these issues.
We use the binary distribution of Ice 3.6.2 (recently updated it from 3.5.1). We start IceBox by Process.Start() command in our application code.
We made some further investigation and saw that sometimes the icebox is not shuting down. Therefore the DB files are not accessible when we startup the application again. We just kill the icebox.exe process while we shuting down our application. Is there a proper way to shutdown the icebox process? We saw in your demos that the icebox can be shut down by using iceboxadmin. But in this demo is iceboxnet instead of icebox started. We were not able to shut down our icebox process with iceboxadmin. Do we have any mistakes in our startup arguments (see first post)?
Best regards,
Aldo0
