Archived
This forum has been archived. Please start a new discussion on GitHub.
Thread problem while trying for a LISP interface
I am working on a large project, a tutorial dialogue system, which is currently is Open Agent Architecture for communication. We are very interested in replacing it with ICE, since OAA has a number of annoying problems.
My difficulty is that many of my system components are written in lisp. My lisp version, Allegro CL 8.1, has a C/C++ interface, so I intended to use it to provide an ICE interface. However, I run into a difficulty with threads.
What I need to do is to have an ICE server running that accepts requests, unmarshalls them, and then calls the appropriate lisp function to handle the request. The problem is, calling lisp functions from C++ is not allowed with multi-threading. The only thing that I am allowed to do is to call C++ from lisp, and then call lisp back only from the same thread before I return control to lisp.
I am trying to figure out what my best options are in this situation. I can see two potential solutions
1) None of my servers need to be multi-threaded, they only get 1 request at a time. So I could call an infinite "event loop" from LISP, which will be which will activate the server code, and then go to sleep. If I can then make the request handlers wake up this event loop thread to get things done, and then let it go to sleep again until the new request comes, then I should be able to call lisp code from it. But I am not sure if it is possible with the C++ thread library.
2) I try to implement a "true" ICE lisp interface by re-implementing the basic necessary functionality in LISP, and calling out to C++ as necessary with requests. For that, I'd need to figure out what is it that "activate" does, and how I can do the same handling in LISP.
Could someone help me figure out if/how 1) is possible, and if not, how much work 2) would be? Or if there are other options for this which I have missed?
My difficulty is that many of my system components are written in lisp. My lisp version, Allegro CL 8.1, has a C/C++ interface, so I intended to use it to provide an ICE interface. However, I run into a difficulty with threads.
What I need to do is to have an ICE server running that accepts requests, unmarshalls them, and then calls the appropriate lisp function to handle the request. The problem is, calling lisp functions from C++ is not allowed with multi-threading. The only thing that I am allowed to do is to call C++ from lisp, and then call lisp back only from the same thread before I return control to lisp.
I am trying to figure out what my best options are in this situation. I can see two potential solutions
1) None of my servers need to be multi-threaded, they only get 1 request at a time. So I could call an infinite "event loop" from LISP, which will be which will activate the server code, and then go to sleep. If I can then make the request handlers wake up this event loop thread to get things done, and then let it go to sleep again until the new request comes, then I should be able to call lisp code from it. But I am not sure if it is possible with the C++ thread library.
2) I try to implement a "true" ICE lisp interface by re-implementing the basic necessary functionality in LISP, and calling out to C++ as necessary with requests. For that, I'd need to figure out what is it that "activate" does, and how I can do the same handling in LISP.
Could someone help me figure out if/how 1) is possible, and if not, how much work 2) would be? Or if there are other options for this which I have missed?
0
Comments
-
Hi,
Welcome to the forum!
I think option #1 is your best bet. There are many ways to implement this strategy but they are highly dependent on your requirements. For example, you would need to consider whether your LISP server will be receiving requests for multiple objects, and whether those objects use the same Slice interface or different Slice interfaces.
In the simplest case, where your server hosts one or more objects that all support the same Slice interface, you could implement a C++ default servant using the normal statically-typed API. Each operation would package up its arguments in a simple container and queue them for processing by the event loop thread. The advantage here is that all of the marshaling and unmarshaling is done for you. Things get a little more complicated if the operations return any results because the C++ servant would need to wait until the event loop completes its dispatch into LISP code and then retrieve the results.
You may need to read up on servant locators if your server hosts an arbitrary number of objects, and asynchronous dispatch might also be useful.
If your objects support different Slice interfaces, you could consider using dynamic dispatch, but then you have to handle the marshaling chores yourself. I recommend avoiding this if at all possible because it is quite easy to make mistakes.
The "workqueue" example in the cpp/demo/IceUtil/workqueue subdirectory of the Ice distribution should get you started on writing your event loop thread.
Hope that helps,
Mark0 -
Thanks a lot! This is very helpful, though I have to admit that I am new enough to ICE that I am not 100% sure that I understand all the terminology.
So let me clarify. I am going to have 10 different ICE servers (corresponding to different LISP components running in different processes). Each server accepts 1 call at a time, and always returns a result. I think I will probably want a singe ICE object/interface per server - I don't see why I would need more, this is a fairly simple application.
I saw the queue application, and I think it gives me a fair idea about how I could organise threading in my C interface, this can work.
The next question is, is it possible for me to write a "generic" c library that I would link to all my LISP servers. I suspect the answer is "no", but I thought I'd ask.
The process with my current architecture is that there is a single *.o file that all my LISP servers load, and then each of them makes a function call to declare which operations it will accept, and the OAA facilitator (sort of equivalent to ICE Grid) figures out how to route the calls to the appropriate server. This makes everything easier to configure, because I only have a single place to manage and re-compile the C files, and I don't need to do complex make files to make sure interfaces stay current for each individual server. But in OAA there are no generated interface files for individual interfaces. From what I understand for ICE, I'd have to compile a separate *.o file for each LISP server, corresponding to the calls it can accept. Is this correct? Can I get around this, or I should just figure out a way to live with it?0 -
If you want to implement a C++ servant using the statically-typed API (i.e., where your C++ class derives from the skeleton class generated by slice2cpp), then you would need to compile at least two source files. I'm assuming that your LISP servers could load a shared library composed of multiple object files if necessary.
You could also do something similar to your existing design, where the LISP server declares its operations by supplying the operation names and parameter and return types. A generic dynamic dispatch servant could use these definitions to convert between Slice types and their LISP equivalents. One disadvantage of this strategy is that it's more error prone. For example, someone could make a change to a Slice file used by a client but neglect to update the corresponding LISP definition in the server. This can cause subtle errors that are only discovered at run time.
If you want to explore this further, I would probably start by writing a translation layer that is able to convert any Slice type (or at least those that your application cares about) to and from LISP types. This should be pretty straightforward as long as you don't need to transfer Slice classes by value. You can review the example in cpp/demo/Ice/invoke to get familiar with the dynamic dispatch and streaming APIs. You can also review the Ice extensions for Python and Ruby, which make extensive use of these APIs.
Note that the generic solution is a pretty ambitious project for someone who is new to Ice. I recommend that you get a statically-typed prototype working first before jumping into the advanced Ice APIs.
Regards,
Mark0 -
Thanks - you were right about the object library, I by now figured out that I can just bundle everything together and load the same library from every LISP process. That was just me not knowing C well enough.
Having worked with the Allegro CL foreign functions (=C) interface, it looks like I am going to back off and do something very simple - essentially, restrict all my SLICE interfaces to passing a single string in, single string out parameter, or maybe a string sequence, at the most. LISP, at least in Allegro implementation, is just not friendly to C++, and passing anything other than strings and integers between LISP and C very quickly becomes a huge hassle. So I either have to do the full work of writing a proper Slice-to-Lisp mapping (for which I don't have the time), or accept that I am not going to get the compile-time benefits that ICE gives. But I will settle for system not crashing when sending very long messages (which is what my current architecture does, causing me no end of grief).0 -
Is there some kind of "contrib" directory for the ICE distribution? I am wondering if others may be interested in the code I have developed for LISP-Ice interface. It's a very limited implementation (only strings can be passed around, no other types, and very little error checking), but it solves a bunch of threading issues as well as LISP-C interface issues related to ICE, and so it may be of interest to someone who wants to develop a similar interface in the future.
We would not be quite ready to release this yet, but if it is relevant, I would consider adding to my work plan a task to make some kind of stand-alone code version in the new year.0 -
Hi,
We have the projects forum for that kind of things.
Please create a post there for your project one you made it public available.
Cheers,
José0
