A problem of glacier2router

GlacierX

Hi buddies,

I'm currently using glacier2router 3.2.1 on the server side. Unfortunately, glacier2router occationally stop to work. I even didn't get the magic number when I telnet to the glacier2router's endpoint from the client side. Then if I restart glacier2router everything turned to be ok again.

Client applications are using Ice 3.2 and Ice 3.2.1, if it is the reason make glacier2router die?

Thanks.

P.S. I'm running glacier2router on Win2003

matthew

The problem is not due to mixing different versions of Ice.

If Glacier2 is no long accepting connections that would indicate that your machine likely is suffering from some resource allocation problem... do you have a significant number of clients using Glacier2 at the same time? Are you seeing any warnings from Glacier2, or other error messages?

If you are not the best way would be to get the stack traces from Glacier2 when you get a hang. Should this be the case you should be able to attach to the running process and look at the stacks and report back to us with more info!

GlacierX

Thanks for your reply, matthew:)

I occationally see some same error messages on the glacier2router's console which indicates:"Not enough memory to complete the operation".

I trace the exception it seemed to be something stopped the Ice runtime from creating new thread. I forget the exactly line the exception indicates, I will show you when I see it again.

I installed VC++ 2005 Express on the server. I'll try to debug glacier2router tomorrow.

However, there are someting confused me. Since the glacier2router is suffering from some kind of resource allocation problem. Why didn't it work later on as the thread count and object count of the process decreased? Is that possible the Ice runtime leak socket or other resources anywhere?


God bless me to provide you the call stack info tomorrow

Thanks.

GlacierX

BTW, I don't think the memory or the virtual memory is running out. Because I check the available memory/virtual memory through task manager. There is quite enough memory available.

GlacierX

After I googled "Not Enough Resources Available to Complete This Operation". I was wonderring if the port resource which used to accept the new connections is running out?

You can have a look at :
RPC Services Fail: Not Enough Resources Available to Complete This Operation
DCOM port range configuration problems

matthew

GlacierX wrote: »

Thanks for your reply, matthew:)

I occationally see some same error messages on the glacier2router's console which indicates:"Not enough memory to complete the operation".

I trace the exception it seemed to be something stopped the Ice runtime from creating new thread. I forget the exactly line the exception indicates, I will show you when I see it again.

I installed VC++ 2005 Express on the server. I'll try to debug glacier2router tomorrow.

However, there are someting confused me. Since the glacier2router is suffering from some kind of resource allocation problem. Why didn't it work later on as the thread count and object count of the process decreased? Is that possible the Ice runtime leak socket or other resources anywhere?


God bless me to provide you the call stack info tomorrow

Thanks.

Where exactly in the Ice source does the error message come from?

A stack trace and the exact message displayed on the console will help us resolve this problem!

GlacierX

Here is a snapshot of the callstack.
Since there are many threads running, I post the callstacks of each threads if their callstacks are not exactly the same.

benoit

Hi,

Could you copy/paste the exact error message when the glacier2router complains about resources not being available (it should in theory contain the file/line number where the error occurs)?

Cheers,
Benoit.

GlacierX

hi

I didn't see anything special when the service refuse to accept new connections. I'm afraid the error message had been buried by warning messages (such as ConnectionLostException)or simply didn't appear.

Or would you plz tell me how to block those warnings?

benoit

Hi,

These warnings should be disabled by default. Are you perhaps setting Ice.Warn.Connections to 1 in the Glacier2 configuration? You can also try setting the Ice.Warn.Dispatch property to 0 if the warnings are warnings for unknown exceptions.

Cheers,
Benoit.

GlacierX

Thanks benoit!

I will modify my glacier2router's config file and try again tomorrow.

GlacierX

Hi benoit,

I caught the error message today.
The last sentense is Chinese. I checked it out from msdn and found the HRESULT for this error is 0x80070008.

01/29/08 14:05:30.893 error: cannot create thread for request queue:
Thread.cpp:195: IceUtil::ThreadSyscallException:
thread syscall exception: 存储空间不足，无法处理此命令。

benoit

Hi,

This error indicates that a thread to handle a new connection can't be created because there's not enough memory available. How many clients are connecting to your Glacier2 router?

Each client connection can use up to 3 threads if you use server & client buffered mode. With a high number of clients and if you didn't change the default thread stack size (1MB on Windows), it's possible that you reach the VM address space limit.

You should try lowering the thread stack size with the Ice.ThreadPerConnection.StackSize property (see demo\Glacier2\callback\config.glacier2 for an example).

Also, did you make sure to set the Glacier2.SessionTimeout property to ensure client session correctly time out if the client goes away?

Cheers,
Benoit.

GlacierX

benoit wrote: »

Hi,

This error indicates that a thread to handle a new connection can't be created because there's not enough memory available. How many clients are connecting to your Glacier2 router?

Each client connection can use up to 3 threads if you use server & client buffered mode. With a high number of clients and if you didn't change the default thread stack size (1MB on Windows), it's possible that you reach the VM address space limit.

You should try lowering the thread stack size with the Ice.ThreadPerConnection.StackSize property (see demo\Glacier2\callback\config.glacier2 for an example).

Also, did you make sure to set the Glacier2.SessionTimeout property to ensure client session correctly time out if the client goes away?

Cheers,
Benoit.

There are 1500+ threads at peak time.
I'm sure I changed Ice.ThreadPerConnection.StackSize to 32768.
I checked the process's status when it stop to accept new connections, which indicates there is quite enough free VM(glacier2router only used 300MB+ VM ).

IF the reason of that is lack of VM, why didn't it stop to work from that on?

When I read the message on the console, the thread count dropped dramatically:only 25 threads are running.

benoit

Hi,

It's possible that you're hitting some other resource limits. I'm afraid without more investigation it's difficult to explain the behavior you're seeing. If you want us to further investigate this matter, you should consider purchasing commercial support (you can contact us at info@zeroc.com if you're interested).

In any case, I would recommend to spread the load of the Glacier2 router sessions to multiple Glacier2 processes, this will avoid reaching system limits. With Ice 3.3, you won't have this problem since Glacier2 won't use anymore the thread per connection concurrency model and will only require a fixed number of threads regardless of how many clients are connected.

Cheers,
Benoit.

GlacierX

benoit wrote: »

Hi,

It's possible that you're hitting some other resource limits. I'm afraid without more investigation it's difficult to explain the behavior you're seeing. If you want us to further investigate this matter, you should consider purchasing commercial support (you can contact us at info@zeroc.com if you're interested).

In any case, I would recommend to spread the load of the Glacier2 router sessions to multiple Glacier2 processes, this will avoid reaching system limits. With Ice 3.3, you won't have this problem since Glacier2 won't use anymore the thread per connection concurrency model and will only require a fixed number of threads regardless of how many clients are connected.

Cheers,
Benoit.

Thanks a looooot!
With Ice 3.3, you won't have this problem since Glacier2 won't use anymore the thread per connection concurrency model and will only require a fixed number of threads regardless of how many clients are connected.

Trust me, that's the one of the most exciting news I've heard recently!
You guys are really nice!

BTW, could you please tell me when will Ice 3.3 be released?

benoit

Check out this thread.

Cheers,
Benoit.

GlacierX

Thanks again for you guys' patience:)