Archived

This forum has been archived. Please start a new discussion on GitHub.

Can I use Ice to solve this problem?

Here's the project scope:

The goal of our project is to allow third-party scientists to develop their own algorithms using a limited set of programming languages ( e.g. Java, .NET, and C++).

Once they have their algorithms coded up, we are supposed to provide them with a framework (program) that can seamlessly load their algorithms at run-time and utilize multiple PCs, each machine potentially having different number of cores to parallelize the work. Then our framework is supposed to send input data to each of the algorithm so that we can obtain the desired result.

Obviously, since the framework is sending data to the algorithm, the framework needs to define the APIs that all algorithm developers are supposed to follow, along with the input / output data types. (Perhaps this is done via Slice?)

Can Ice middleware be used for such project? If so, what parts of Ice should be used?

I've looked at a number of technologies, and stumbled upon Ice as a modern alternative to DCOM, CORBA, so I want to see how Ice can solve my problem.

Please let me know if you want more clarifications on the project scope.

Comments

  • What I'm looking for is similar to IceGrid example

    We can take the IceGrid MP3 ripper in Ice reference manual as an example. The only difference between that example and my application is as follow:

    1) The ripper algorithm is only available at run-time (compiled with Ice of course). That is, the user would dynamically load the ripper algorithm only when he starts the application. This is the same concept as Plug-ins, more or less.

    2) There can be many ripper algorithms, all need to run concurrently.

    3) Work can be parallelized because many machines are available. Each machine also is multi-core.

    Specific configuration:

    Let's say we have 2 ripper algorithms: RipperA and RipperB.

    And that we have 10 songs to process: song1.wav to song10.wav

    And that we have 3 machines: Machine1, Machine2, and Machine3.

    Machine1 has 4 cores.
    Machine2 has 8 cores.
    Machine3 has 2 cores.

    It would be really nice if the user can dynamically load RipperA and RipperB algorithms from GUI and transmits the algorithms to each of the machine (code mobility) and have them execute in parallel as new songs are distributed.

    If I were to put the above requirements in one sentence, it would be:

    "Distributed Parallel Computing using Plug-ins" ??
  • xdm
    xdm La Coruña, Spain
    Hi
    1) The ripper algorithm is only available at run-time (compiled with Ice of course). That is, the user would dynamically load the ripper algorithm only when he starts the application. This is the same concept as Plug-ins, more or less.
    You could implement a servant as a plug-in, Ice core is unaware of the implementation, it should agree with the interface defined in slice.
    2) There can be many ripper algorithms, all need to run concurrently.
    This is supported by Ice run time, you need to configure thread pools see Ice Threading model in manual.
    It would be really nice if the user can dynamically load RipperA and RipperB algorithms from GUI and transmits the algorithms to each of the machine (code mobility) and have them execute in parallel as new songs are distributed.
    IcePatch2 is an efficient file patching service, and you could integrate it with your application to do that (code mobility). see IcePath2 manual chapter.
    "Distributed Parallel Computing using Plug-ins" ??

    Ice is unaware of client/server implementation, so will be easy to design an application to work in this way with Ice.

    For example in Java you could load a class when required, in c++ this will be a dynamic loaded library, that exports a factory function.

    Ice don't care about this details of the implementation and you are free to implement client and server using plug-in or whatever.
    Obviously, since the framework is sending data to the algorithm, the framework needs to define the APIs that all algorithm developers are supposed to follow, along with the input / output data types. (Perhaps this is done via Slice?)

    Yes, slice is Ice idl, here you define client/server contract.

    In your project you could define interfaces for different algorithms and then users could provide implementation writing a new plug-in. Your server should handle the distribution and load of plug-ins but this shouldn't be difficult.

    You should read the Slice mapping chapters for the languages you are interesting if you have not done it yet.

    Let's us know if you have more questions, or something isn't clear

    Cheers,
    José
  • Thank you!

    Thank you, Jose. Looks like I have a lot of readings I have to do. :)

    Perhaps I will learn best by looking at examples and try to implement a prototype.

    I see that Ice has a lot of example code (in the demo* folders), would you be so kind as to point out some examples (just the project name would be enough) that might be helpful in my adventure :p
  • xdm
    xdm La Coruña, Spain
    Hi Shivesh,

    I think is easier to begin with the basic demos and when these are clear go with more complex things.
    • Ice/hello is mandatory hello world , you will see the basics of an Ice Application here
    • Ice/async show how to send messages asynchronously, that is important in many scenarios
    • Ice/callback shows how to send a callback from a server to a client, not all applications uses callbacks but is very common.

    Do some modifications of that, to see how it works.

    Then you could begin, with demos of services that are a bit more complex, for your application description, i think that Glacier2, IcePath2, IceGrid, will be interesting services to have, IceStorm probably too but it depends on how you want to design the system.

    For a more complete demo see Chat Demo, the design is explained in detail in the web pages and articles.

    And don't forget the news letter

    Cheers,
    José