scan Freeze map slow

rolandsun

we define map key as

struct bufKey {
    int stime;
    int etime;
    short angle;
   short datatype;
   string code;
   string name; 
}

while data is another complex structure, which is about 300k per record.
we use simple code to scan database when program startup as

 Basedb::const_iterator it;
 for(it=baseDatabase.begin();it!=baseDatabase.end();it++)
{
       printf("time %d\n",it->first.stime);
}

We use BDB as database.The scan become very slow (several minutes) when the size of database grow more than 1GB. I add Freeze.Trace.Map config file and found that "duplicating iterator/close iterator" uses most of the time.

Can I speed up the scan ?

bernard

Hi Roland,

You should switch to a prefix increment (++it), that does not create a temporary copy of the iterator.

Then, if baseDatabase is not const, you should use a const variable to avoid the conversion from iterator to const_iterator in it != baseDatabase.end().

Overall:

const Basedb& baseDatabaseConst = baseDatabase;
for(Basedb::const_iterator it = baseDatabaseConst.begin(); it != baseDatabaseConst.end(); ++it)
{
       printf("time %d\n", it->first.stime);
}

Let us know how it goes.

Also, please update your profile: 'proof of concept' is not an acceptable project.

Best regards,
Bernard

rolandsun

Hi Bernard ,

I tried what you told me ,while I got almost no improvement . Actually what I want is only to scan the keys , I think Freeze read both key and data into memory , so it take long time.

I have to scan the database by BDB api if there is no better solution.