Archived

This forum has been archived. Please start a new discussion on GitHub.

Icestorm<->Berkeley DB error

Hello,

i have a function that reads several datasets from an Oracle DB. For each of these datasets I want to create an Icestorm-topic.

After I created approx. 350 topics I get the following error message:

[ icebox-IceStorm: Berkeley DB: DBEnv "db:" Lock table is out of locker entries ]
icebox-IceStorm: warning: dispatch exception: SharedDb.cpp: 180: Freeze::DatabaseException
DB::open: Not enough space
identity: IceStorm/TopicManager
facet:
operation: create

Are there any configuration values that I missed to set?

Thanks in advance.

Comments

  • bernard
    bernard Jupiter, FL
    Hello,

    This is a Berkeley DB configuration issue. By creating lots of topics, you've reached the maximum number of Berkeley DB lockers configured by default, 1000. See http://www.sleepycat.com/docs/api_cxx/env_set_lk_max_lockers.html and http://www.sleepycat.com/docs/ref/lock/max.html.

    The easiest solution is to create a DB_CONFIG file in your db directory and specify set_lk_max_lockers <some larger number>, for example:
    set_lk_max_lockers 10000

    Note that the Berkeley DB documentation at http://www.sleepycat.com/docs corresponds to the DB 4.2.52 release (from Nov 25). Ice 1.2 uses 4.1.25 which is slightly different, for example it does not have the various get_ functions to retrieve configuration settings.

    Cheers,
    Bernard
  • Originally posted by bernard
    Hello,

    This is a Berkeley DB configuration issue. By creating lots of topics, you've reached the maximum number of Berkeley DB lockers configured by default, 1000. See http://www.sleepycat.com/docs/api_cxx/env_set_lk_max_lockers.html and http://www.sleepycat.com/docs/ref/lock/max.html.

    The easiest solution is to create a DB_CONFIG file in your db directory and specify set_lk_max_lockers <some larger number>, for example:
    set_lk_max_lockers 10000

    What's the relationship between DB lockers and topics? Does each topic hold a cursor and DB handle, or more? Is 3-to-1 a decent scaling ratio?

    I've been considering a design that relies on a lot (like, tens of thousands) of IceStorm topics, so I'm interested in any experiences people have had scaling IceStorm to that level.

    Mike
  • Hello,

    I looked into BerkeleyDB-docs and created a config file (DB_CONFIG) with the following entries:

    set_lk_max_lockers 10000
    set_lk_max_locks 10000
    set_lk_max_objects 10000

    On my windows machine I can only create 817 topics. There is no difference if the value of set_lk_max_lockers is set to 10000 or 100000.

    Then I've tested my application on a Solaris machine. After only 116 topics the following error occurs:

    [ icebox-IceStorm: Berkeley DB: DbEnv "db": DB_TXN->abort: Log undo failed for LSN: 1 1946136: DB_NOTFOUND: No matching key/data pair found ]
    [ icebox-IceStorm: Berkeley DB: DbEnv "db": PANIC: DB_NOTFOUND: No matching key/data pair found ]
    [ icebox-IceStorm: Berkeley DB: DbEnv "db": fatal region error detected; run recovery ]
    [ icebox-IceStorm: Berkeley DB: DbEnv "db": fatal region error detected; run recovery ]
    [ icebox-IceStorm: Berkeley DB: DbEnv "db": PANIC: DB_RUNRECOVERY: Fatal error, run database recovery ]
    [ icebox-IceStorm: Berkeley DB: DbEnv "db": fatal region error detected; run recovery ]
    icebox-IceStorm: warning: dispatch exception: SharedDb.cpp:180: Freeze::DatabaseException:
    Db::open: DB_RUNRECOVERY: Fatal error, run database recovery
    identity: IceStorm/TopicManager
    facet:
    operation: create


    An additional problem is the performance of the topic creation under both systems. For example, the creation of approx. 280 topics on the windows machine takes one minute.

    Thanks in advance
  • bernard
    bernard Jupiter, FL
    Hello,

    I've just created 10,000 topics on my Windows XP laptop using the following DB_CONFIG:
    set_lk_max_lockers 40000
    set_lk_max_locks 40000
    set_lk_max_objects 40000
    set_lg_regionmax 2400000
    set_cachesize 0 104857600 1

    set_lg_regionmax is necessary because of the large number of files, and you need a larger set_cachesize most likely as a work-around for bug #6681 (see http://www.sleepycat.com/update/4.2.52/if.4.2.52.html).

    I haven't tried on Solaris. Maybe you reached a file descriptors limit?

    Anyway, creating lots of topics in IceStorm should not be a Berkeley DB scalability exercise. It also takes too long. IceStorm should not create a Freeze Map (and hence Berkeley DB database) per topic ... everything could be combined in one big Map. We'll try to fix this for the next Ice release.

    Cheers,
    Bernard
  • I've been working on pulling out the TopicManager/Topic storage stuff into a separate StormStorage class for the past day or two, with the goal to be able to provide both a Freeze-backed persistent set of topics/links, as well as a transient in-memory-only one. This probably isn't as interesting if the maps get condensed into one, but I'll post a patch anyway when I get things to stop crashing. :)
  • bernard
    bernard Jupiter, FL
    Hello,

    IceStorm in Ice 1.3 was updated to use a single database for all topics. As a result topic creation is a lot faster now. And you can create a very large number of topics without any special Berkeley DB configuration.

    However, if you're creating several thousand topics and want to speed up their creation, I recommend you create a DB_CONFIG file in your Berkeley DB directory (typically .../db/) with the following directive:
    set_flags DB_TXN_NOSYNC

    See http://www.sleepycat.com/docs/api_cxx/env_set_flags.html for a description of DB_TXN_NOSYNC.

    Cheers,
    Bernard
  • Hello,

    I created a DB_CONFIG file and the creation of many topics is much faster. Thanks a lot for this. Anyway, are there any possibilities to improve the performance a little bit more?


    Thanks in advance.
  • bernard
    bernard Jupiter, FL
    If you don't care about topic/links persistence, and really care about topic-creation-speed, removing Freeze altogether would make sense. Did you try vukicevic's patch?

    Cheers,
    Bernard