Archived
This forum has been archived. Please start a new discussion on GitHub.
Why concurrent access to FreezeMap is so slow?
When I test the concurrent access to FreezeMap,I found it is much slower than single-thread method.My code like following:
2. When some threads are invoking g_GetNamesFromMap, I will call g_SetNamesToMap to update the map. In this case, do I have to serialize the two function or just call g_SetNamesToMap directly?
ObjectAdapterPtr g_pAdapter; vector<string> g_GetNamesFromMap(long beginTime,long endTime) { vector<string> back; Freeze::ConnectionPtr pConn = Freeze::createConnection(g_pAdapter->getCommunicator(), "db"); IHisDataMap hisDataMap(pConn, "map"); IHisDataMap::const_iterator pEnd = hisDataMap.upperBoundForTime(endTime); IHisDataMap::const_iterator pos; for(pos = hisDataMap.lowerBoundForTime(beginTime);pos!=pEnd;++pos) { back.push_back(pos->first); } return back; } vector<string> g_SetNamesToMap(string name,long time) { Freeze::ConnectionPtr pConn = Freeze::createConnection(g_pAdapter->getCommunicator(), "db"); IHisDataMap hisDataMap(pConn, "map"); IHisDataMap::iterator pos = hisDataMap.find(name); if(pos != hisDataMap.end()) { pos.set(time); }else { hisDataMap.insert(make_pair(name, time)); } } void JobThread::run() { m_bRunning = true; IceUtil::Time sleepTime = IceUtil::Time::milliSeconds(1000); while(true) { if(!m_bRunning) { break; }else { vector<string> names = g_GetNamesFromMap(0,ULONG_MAX); } ThreadControl::sleep(sleepTime); } } void main() { //Init communicator and g_pAdapter ............................. int iThreadNum = 30; for(int i=0;i<iThreadNum;++i) { JobThread pThread = new JobThread(); pThread->start(); } }1. When iThreadNum = 1, the g_GetNamesFromMap will use 600ms to finish per time. When iThreadNum = 30, it will takes 33s to finish. Is there anything wrong?
2. When some threads are invoking g_GetNamesFromMap, I will call g_SetNamesToMap to update the map. In this case, do I have to serialize the two function or just call g_SetNamesToMap directly?
0
Comments
-
kingbo wrote:1. When iThreadNum = 1, the g_GetNamesFromMap will use 600ms to finish per time. When iThreadNum = 30, it will takes 33s to finish. Is there anything wrong?
Your code is not very clear to me: each thread is an infinite loop that gets the names and sleeps for one full second? Which time are you measuring?
In g_GetNamesFromMap, you can avoid creating transactions by using a const map:const IHisDataMap hisDataMap(pConn, "map");
This could improve performance a little.kingbo wrote:2. When some threads are invoking g_GetNamesFromMap, I will call g_SetNamesToMap to update the map. In this case, do I have to serialize the two function or just call g_SetNamesToMap directly?
You don't need to serialize, since you're using separate connections/maps in each function. However, you need to deal with deadlocks. See the Ice manual for details.
Cheers,
Bernard0 -
vow! So quick response in weekend. Thank you a lot!Your code is not very clear to me: each thread is an infinite loop that gets the names and sleeps for one full second? Which time are you measuring?
In fact,the code is:void JobThread::run() { m_bRunning = true; IceUtil::Time sleepTime = IceUtil::Time::milliSeconds(1000); while(true) { if(!m_bRunning) { break; }else { long beginPos = GetTickCount(); vector<string> names = g_GetNamesFromMap(0,ULONG_MAX); long endPos = GetTickCount(); printf("g_GetNamesFromMap takes %d\n",endPos-beginPos); } ThreadControl::sleep(sleepTime); } }
I just tested your suggestion of "const IHisDataMap hisDataMap(pConn, "map");" and found the performence is improved a little but not satisfied. It still takes 20s to 30s per time.:(0 -
Another improvement would be to keep a Map opened in your main thread:
Freeze::ConnectionPtr pConn = Freeze::createConnection(g_pAdapter->getCommunicator(), "db"); IHisDataMap hisDataMap(pConn, "map");
This way, you would avoid opening/closing the map all the time.
Cheers,
Bernard0 -
There is huge distance between 33s and 600ms. I guess I must miss some important configuration in DB_CONFIG. My DB_CONFIG is:set_cachesize 0 1073741824 1
set_lk_max_lockers 40000
set_lk_max_locks 40000
set_lk_max_objects 40000
set_tx_max 20000
set_lg_bsize 2097152
set_lg_max 2097152
set_flags DB_TXN_NOSYNC
set_flags DB_TXN_WRITE_NOSYNC0