Can't insert a record when db has this record

dragzhb

HI bernard:
When I insert one record into db , but it can't be inserted when this record has been in db.
I think ICE insert db using Berkeley DB's DB->put with flag DB_NOOVERWRITE.
this time I insert record into db must use following steps:

_mapPtr->erase(obj.ID);
_mapPtr->insert(make_pair(obj.ID, obj));

but it's very slowly when db has many records.

How can quickly insert records when db has the rcord or has no the record?

Could you have an option to insert record using Berkeley DB's default behavior

The default behavior of the DB->put function is to enter the new key/data pair, replacing any previously existing key if duplicates are disallowed, or adding a duplicate data item if duplicates are allowed.

thanks

bernard

The Freeze Map (C++) has a put function as well.

Cheers,
Bernard

dragzhb

HI bernard:

it's okay now , thanks .
but it's slowly too, when I insert records.

The following is my test:

OS: redhat advanced server 3.0
HD : IBM X340 2 CPU PIII 1.0 G , 1 G memory
db cache size:
# 300M
set_cachesize 0 314572800 0

db size: 3,190,368

inserting speed:

---

Start time : Wed Oct 27 11:14:54 2004
End time : Wed Oct 27 11:15:37 2004
Put Total : 1000
Total Time : 0:0:43
Put Data/s : 23
Put Data/h : 82800

---

bernard

You could use transactions to improve performance; if you don't use transactions, Freeze/Berkeley DB starts and commits a transaction for each write operation.

Next, you could look at various Berkeley DB performance tuning:
set_cachesize
set_lg_bsize
set_flags DB_TXN_WRITE_NOSYNC
set_flags DB_TXN_NOSYNC

Cheers,
Bernard

dragzhb

thanks

dragzhb

HI bernard:
thank you very much, very good. it's very quickly to insert records

The following is my test result:
./phonelistTest -n 10000
Putting Phone Number Statisticts

---

Start time : Fri Oct 29 16:12:44 2004
End time : Fri Oct 29 16:12:47 2004
Put Total : 10000
Total Time : 0:0:3
Put Data/s : 3333
Put Data/h : 11998800

---

thanks

dragzhb

HI bernard:

I found another question for performance.

I want to get db size, I use _Map.size() to get total records , but it's very slowly , my database size:

Database Size

---

phonelist size : 10,879,463

---

it will take about 20 minutes . I found that Berkeley db would go through every record to count the size, I watch this result by watch memory usage.
how do you statistics db size ?
use DB->stat of bt_ndata flag or one by one to cout it ?

The following is my Configuration:

OS: redhat advanced server 3.0
HD : IBM X340 2 CPU PIII 1.0 G , 1 G memory

The following is my DB_CONFIG file:
set_data_dir data
set_lg_dir log
# 300M
set_cachesize 0 314572800 0

# 20M
set_lg_max 20971520
# 5M
set_lg_bsize 5242880
set_flags DB_TXN_WRITE_NOSYNC
set_flags DB_TXN_NOSYNC


How can I do to improve performance to get db size?

another performance is to clear db , using _mapPtr->clear(); it is very slowly too.
did you use DB->truncate to empty db or use DB->del to delete it one by one?

thanks

bernard

Since Ice is available with full source, you can easily find out which Berkeley DB function is used You can also try to tweak the source to get better performance.

Freeze::Map::size is implemented with Db::stat (see src/Freeze/MapI.cpp). The documentation of stat suggests this results in traversing the entire database: see http://www.sleepycat.com/docs/api_cxx/db_stat.html
Let me know if you find a faster way to compute this size!

Freeeze::Map::clear() is implemented with Db::truncate (see again src/Freeze/MapI.cpp). I don't know why it's slow; you could set Freeze.Warn.Deadlocks to see if any deadlock occurs during this clear.

Cheers,
Bernard

dragzhb

I see, Thank you very much !