Home Help Center

ThreadPerConnection and need for the exact same thread

smusmu Member Shang MuOrganization: University of FloridaProject: JointTrack
I knew this probably would be a problem for my project since you guys sent out the 3.3 release note, but back then I didn't want to post the question here without first trying out the new version, which unfortunately never happened. I am posting now to see if you guys have considered this situation when making the decision to remove ThreadPerConnection and if you have any suggestions on good ways to solve the problem.

I have a servant class that generates data utilizing a third party graphical library which, like most GUI libraries, is generally not thread safe. Unfortunately, because the library was never designed with multithreading in mind, the calls not only have to be serialized, they need to be on exactly the same thread (i.e., the thread id need to be the same). The earlier ThreadPerConnection model before Ice 3.3 worked perfectly for my case. The ThreadPool.Serialize property in newer versions does not guarantee that calls are dispatched on exactly the same thread, so now I am writing code to transfer the calls to a dedicated thread.

Any suggestions on implementing what i'm trying to do here and is there a better way to solve my problem?
Thanks!

Comments

  • mesmes CaliforniaAdministrators, ZeroC Staff Mark SpruiellOrganization: ZeroC, Inc.Project: Ice Developer ZeroC Staff
    Hi,

    Ice 3.4 introduced a new dispatcher facility that will do what you need. Using this feature should make your application quite a bit simpler because there is no need to use AMD or to duplicate the operation parameters in order to queue the request for execution in another thread.

    Regards,
    Mark
  • smusmu Member Shang MuOrganization: University of FloridaProject: JointTrack
    Thanks Mark! That works.

    This last note in the doc was very confusing to me:
    "Never make a blocking invocation from the dispatch thread, such as a synchronous proxy operation or a proxy method that can potentially block, such as ice_getConnection. These invocations depend on the dispatcher for their own completion, therefore blocking the dispatch thread will inevitably lead to deadlock."

    Isn't the dispatching thread just a regular thread in the thread pool so no deadlock would happen as long as there are remaining threads available in the pool? (Reading your source code certainly seems so.)

    By the way the MFC demo referenced in the documentation is missing from your demo package. It is in the source package though.
  • smusmu Member Shang MuOrganization: University of FloridaProject: JointTrack
    On a completely unrelated issue, I'm randomly getting application hang on application exit, which never happened before with Ice 3.2.1. On one side it's hanging here when destroying the communicator:

    void
    IceInternal::IncomingConnectionFactory::waitUntilFinished()
    {
    set<ConnectionIPtr> connections;
    {
    IceUtil::Monitor<IceUtil::Mutex>::Lock sync(*this);


    and I guess on the other side it's probably hanging in the OutgoingConnectionFactory::waitUntilFinished. There are no unfinished calls (as seen by application code). Are there any known issues like this (I'm working with VC2010 debug builds with Ice 3.4.2 on Windows 7)? Otherwise I'll let you know what I find out.
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi,

    Hmm, the MFC demo is in my demo package in the Ice-3.4.2-demos/demo/Ice/MFC directory.

    There are no known issue similar to the one you mention, it would be interesting to see the stack traces of all the threads to figure out what could be the problem.

    Cheers,
    Benoit.
  • xdmxdm La Coruña, SpainAdministrators, ZeroC Staff Jose Gutierrez de la ConchaOrganization: ZeroC, Inc.Project: Ice Developer ZeroC Staff
    "Never make a blocking invocation from the dispatch thread, such as a synchronous proxy operation or a proxy method that can potentially block, such as ice_getConnection. These invocations depend on the dispatcher for their own completion, therefore blocking the dispatch thread will inevitably lead to deadlock."

    That note is under "Dispatcher Implementation Notes" section and is relevant for applications using a custom dispatcher.
  • smusmu Member Shang MuOrganization: University of FloridaProject: JointTrack
    benoit wrote: »
    Hmm, the MFC demo is in my demo package in the Ice-3.4.2-demos/demo/Ice/MFC directory.
    Ah sorry, my bad. Now I remember I actually installed the demo files using your windows installer. It's entirely possible I skipped the MFC demo if you had the option to select individual demos in the installer.
    benoit wrote: »
    it would be interesting to see the stack traces of all the threads to figure out what could be the problem.
    I have attached the stack trace for each threads.
    Basically I have a service manager that takes care of shutting down all components when itself is shutting down. And when doing so there's often a random component stuck in the middle of termination process which hangs both that component and the manager (the call to the component actually already finished). If I kill the component the manager executable is able to get out of the deadlock and terminate, but not the other way around (i.e. when i kill the manager). I could reproduce the hang easily. There had been no other calls other than the components making a call each to register themselves with the manager.

    Thanks for taking a look at them. Meanwhile I guess I'll create some minimal code to reproduce what I just described here.

    Shang
  • smusmu Member Shang MuOrganization: University of FloridaProject: JointTrack
    xdm wrote: »
    That note is under "Dispatcher Implementation Notes" section and is relevant for applications using a custom dispatcher.
    I'm more confused now. For my original problem I did have to derive a class from Ice::Dispatcher following Mark's suggestion.

    By the way I forgot to ask what am I supposed to do with exceptions if they are thrown in the thread that runs DispatcherCall::run()?
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi,

    Yes, it looks like José missed the fact that you were using a dispatcher so you can ignore this comment. In theory the run() method of the dispatcher call isn't supposed to throw so you could assert if it does. I can't explain the deadlock from just looking at the stack traces. Do you make sure that no dispatcher calls are getting lost? It's important to make sure that all the dispatcher calls are correctly being run until the communicator is fully destroyed (i.e.: the communicator destroy() method returns).

    Do you also get this hang if you don't setup any dispatcher? A small test case would definitely help to investigate this.

    Cheers,
    Benoit.
  • smusmu Member Shang MuOrganization: University of FloridaProject: JointTrack
    It mystified me too. That's right, the hang doesn't have anything to do with the dispatcher (I'm not running the component that needed the dispatcher).

    And I think I just found the culprit. The hanging behavior in my new finding is not exactly the same (my service manager was able to terminate instead of waiting for the hanging component). I still don't understand this completely unexpected behavior but I believe it is the cause and I could reproduce the new finding 100% of the time.

    It had something to do with my service manager making use of a win32 system call _spawnlp().

    If I make this call in my manager executable:
    _spawnlp( _P_NOWAIT, "notepad.exe", "notepad.exe", NULL);

    Depending on whether my components are started before or after my manager makes the _spawnlp call, either all or just one of my components will not terminate after my manager executable asks them to shut down. Once I exit the spawn process (notepad.exe in this testing case), the hanging component(s) immediately terminates gracefully.

    The strange thing is the _spawnlp() call is made by my service manager, but the hanging process(s) are my components, which are not started by the manager and had nothing to do with the spawn process at all in this testing. Again, I'm using Ice 3.4.2, VC10, on a Windows 7 machine.
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi,

    Ok, that probably explains it. Most likely, after the spawn system call, the child process inherits the file descriptors from the parent and this includes the socket FDs that were open before the spawn call. When your clients exit, they send a close connection message to the manager and wait for the manager to close the sockets... however the sockets aren't getting closed because the child process (notepad.exe) doesn't close them.

    The use of timeouts could force your clients to not wait for the close of the socket (either using regular timeouts or Ice.Override.CloseTimeout) but the real fix is to ensure that the child process does not inherit the sockets from your service manager. I recommend using CreateProcess instead of the spawn system call, you can have a look at cpp/src/IceGrid/Activator.cpp in the Ice sources for an example on how to safely spawn processes on Windows and Unix.

    Cheers,
    Benoit.
  • smusmu Member Shang MuOrganization: University of FloridaProject: JointTrack
    Thanks a lot for all the help!
Sign In or Register to comment.