IceStorm, db log file unreadable

HurlymurphyHurlymurphy Member Aldo PlatzOrganization: Schindler Elevator Project: TestTools

We use IceBox with IceStorm on Windows with C#. We observed since a long time a sporadically failure while starting up IceBox. We can not start IceStorm because it seems there is a problem with the database.

We saw following log entry in the Ice.LogFile:

- 07/28/16 20:50:29.114 C:/Jenkins/VirtualGCTest/TestEnvironment/Current_TaTU_Version/IceBox/icebox.exe-IceStorm: Freeze.DbEnv: checkpointing environment "IceStorm"
-- 07/28/16 20:50:29.139 C:/Jenkins/VirtualGCTest/TestEnvironment/Current_TaTU_Version/IceBox/icebox.exe-IceStorm: Berkeley DB: DbEnv "IceStorm": BDB2520 C:\Users\BATman\AppData\Local\Temp\IceStorm1\log.0000000001: log file unreadable: Operation not permitted
-- 07/28/16 20:50:29.139 C:/Jenkins/VirtualGCTest/TestEnvironment/Current_TaTU_Version/IceBox/icebox.exe-IceStorm: Berkeley DB: DbEnv "IceStorm": BDB0061 PANIC: Operation not permitted
-- 07/28/16 20:52:29.145 C:/Jenkins/VirtualGCTest/TestEnvironment/Current_TaTU_Version/IceBox/icebox.exe-IceStorm: Freeze.DbEnv: checkpointing environment "IceStorm"
-- 07/28/16 20:52:29.145 C:/Jenkins/VirtualGCTest/TestEnvironment/Current_TaTU_Version/IceBox/icebox.exe-IceStorm: Berkeley DB: DbEnv "IceStorm": BDB0060 PANIC: fatal region error detected; run recovery
-! 07/28/16 20:52:29.145 C:/Jenkins/VirtualGCTest/TestEnvironment/Current_TaTU_Version/IceBox/icebox.exe-IceStorm: warning: checkpoint on DbEnv "IceStorm" raised DbException: DbEnv::txn_checkpoint: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

That are our startup arguments:

string[] config = 
         {
            // IceBox settings
            "--Ice.IPv6=0",
            "--Ice.LogFile=" + logfilePath,
            "--Ice.Trace.Locator=2",
            "--Ice.Trace.Network=2", 
            "--Ice.Trace.Protocol=0", 
            "--Ice.Trace.Retry=2",
            "--Ice.Trace.Slicing=1",
            "--Ice.Trace.ThreadPool=0",

            "--Ice.Admin.Endpoints=\"" + iceConfig.ServiceManagerEndpoints + "\"",
            "--IceBox.PrintServicesReady=Services",
            // IceStorm settings

            "--IceStorm.TopicManager.Endpoints=\"" + iceConfig.TopicManagerEndpoints + "\"",
            "--IceStorm.Publish.Endpoints=\"" + iceConfig.TopicPublisherEndpoints + "\"",
            "--IceStorm.InstanceName=TATU",
            "--IceStorm.Trace.Election=1",
            "--IceStorm.Trace.Replication=1",
            "--IceStorm.Trace.Subscriber=2",
            "--IceStorm.Trace.Topic=2",
            "--IceStorm.Trace.TopicManager=1",
            "--IceStorm.Flush.Timeout=100",

            "--IceBox.Service.IceStorm=\"IceStormService," + IceVersionExtra + ":createIceStorm",
            "--Freeze.DbEnv.IceStorm.DbHome=" + dbDir,
            "--Freeze.Trace.DbEnv=2",
            "--Freeze.Trace.Evictor=3",
            "--Freeze.Trace.Map=1",
            "--Freeze.Trace.Transaction=0",
            "\""
         };

Can you help us with this issue?

Ice.log 14.5K

Comments

  • bernardbernard Jupiter, FLAdministrators, ZeroC Staff Bernard NormierOrganization: ZeroC, Inc.Project: Ice ZeroC Staff

    Hi Aldo,

    I would start by putting the IceStorm Freeze DbHome in non-temporary directory, owned by the user running IceStorm:

    Berkeley DB: DbEnv "IceStorm": BDB2520 C:\Users\BATman\AppData\Local\Temp\IceStorm1\log.0000000001: log file unreadable: Operation not permitted
    

    Cheers,
    Bernard

  • HurlymurphyHurlymurphy Member Aldo PlatzOrganization: Schindler Elevator Project: TestTools

    Hi Bernard

    Thanks for your answer. I moved the Freeze Db Home to ProgramData but the issue occurs still from time to time.

    Berkeley DB: DbEnv "IceStorm": BDB2520 C:\ProgramData\Schindler\TaTU\db\IceStorm1\log.0000000001: log file unreadable: Operation not permitted
    Berkeley DB: DbEnv "IceStorm": BDB0061 PANIC: Operation not permitted
    Freeze.DbEnv: checkpointing environment "IceStorm"
    Berkeley DB: DbEnv "IceStorm": BDB0060 PANIC: fatal region error detected; run recovery
    warning: checkpoint on DbEnv "IceStorm" raised DbException: DbEnv::txn_checkpoint: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
    Freeze.DbEnv: checkpointing environment "IceStorm"
    Berkeley DB: DbEnv "IceStorm": BDB0060 PANIC: fatal region error detected; run recovery
    warning: checkpoint on DbEnv "IceStorm" raised DbException: DbEnv::txn_checkpoint: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
    Freeze.DbEnv: checkpointing environment "IceStorm"
    Berkeley DB: DbEnv "IceStorm": BDB0060 PANIC: fatal region error detected; run recovery
    warning: checkpoint on DbEnv "IceStorm" raised DbException: DbEnv::txn_checkpoint: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
    

    Have you got any other ideas?

  • bernardbernard Jupiter, FLAdministrators, ZeroC Staff Bernard NormierOrganization: ZeroC, Inc.Project: Ice ZeroC Staff

    Hi Aldo,

    The problem you're seeing is not the initial startup (IceStorm starts up fine), but the periodic checkpointing of the Freeze database. The default checkpoint period is 2 minutes, and it appears to fail each time. See:

    https://doc.zeroc.com/pages/viewpage.action?pageId=14031645#Freeze.*-Freeze.DbEnv.env-name.CheckpointPeriod

    and

    http://docs.oracle.com/cd/E17076_05/html/programmer_reference/transapp_checkpoint.html

    As a work-around, you could disable checkpoints (by setting CheckpointPeriod to 0), but obviously these checkpoints should not fail in the first place.

    Could you specify the version of Ice you're using? Are you using our binary distribution or did you build Ice from sources? Does the user running IceBox have full permissions to this C:\ProgramData\Schindler\TaTU\db\IceStorm1 directory? How do you start your IceBox server? Do you start icebox as a Windows service?

    Best regards,
    Bernard

  • HurlymurphyHurlymurphy Member Aldo PlatzOrganization: Schindler Elevator Project: TestTools
    edited September 2016

    Hi Bernhard

    You're supposing that IceStorm is starting up fine in spite of these database errors we see in the log. So, you have to know that these errors do not occur on every start-up of IceStorm. But every time they appear we cannot access the TopicManager which is actually our main problem. Therefore, we assume that there's a relation between these issues.

    We use the binary distribution of Ice 3.6.2 (recently updated it from 3.5.1). We start IceBox by Process.Start() command in our application code.

    We made some further investigation and saw that sometimes the icebox is not shuting down. Therefore the DB files are not accessible when we startup the application again. We just kill the icebox.exe process while we shuting down our application. Is there a proper way to shutdown the icebox process? We saw in your demos that the icebox can be shut down by using iceboxadmin. But in this demo is iceboxnet instead of icebox started. We were not able to shut down our icebox process with iceboxadmin. Do we have any mistakes in our startup arguments (see first post)?

    Best regards,
    Aldo

Sign In or Register to comment.