Archived

This forum has been archived. Please start a new discussion on GitHub.

YafridNG Project

Hello, I am Jose and this is my first huge project using ICE.

I am trying to develop a Grid for rendering 3D Scenes modelled with Blender and rendered with Blender render engine or Yafray. The fact is that I would like to use as less services as possible.

In order for transferring files to clients I am implementing a p2p system. When a client connects to the system, it will get into the network obtaining a stringfied proxy from a WellKnown clients file. Then it queries for the WellKnown client's neighbours.

When a client wishes to render a scene, all its neighbours must have the complete file. First of all, a piece of the file is written in each of the neighbours and then all of them get the remaining pieces contacting all other neighbours. If a piece of information cannot be reached using a neighbour, it will be obtained reading from the client that wishes to
render the scene.

The slice interface is:
/**
  *A task contains all the necessary information for a new task.
  *Node specifies the client who starts the transfer task, this client
  *will act as a server, providing pieces that any of the other neighbours
  *could provide.
  *fileName specifies the name of the file which is being transferred.
  *fileSize specifies the size of the file which is being transferred.
  *pieces specifies the number of pieces that the file has been splitted in.
  **/
struct task{
  p2pClient* node;
  string fileName;
  int fileSize;
  int pieces;
};

/**
  *All clients keep a servant of this interface. It manages the exchange 
of files between
  *clients. When a client wants to transfer a file it invokes the 
createNewProject operation
  *for all of its neighbours.
  **/
interface p2pClient{
/**
  *The read operation reads a piece of a file.
  *
  *@param name is the name of the file that must be readed.
  *@param offset is the offset where the file will be readed.
  *@param num is the number of bytes that will be readed.
  *
  *@return a ByteSeq that contains the info readed.
  *
  *throws FileAccessException if the file could not be opened.
  *
**/
  ["ami", "amd", "cpp:array"] ByteSeq read(string name, int offset, int 
num) throws FileAccessException;
/**
  *The pieceRequest operation ask for a concrete piece of file. If the 
neighbour returns TRUE
  *the piece will be readed.
  *
  *@param projectName specifies the name of the project.
  *@param piece specifies the number of the requested piece.
  *
  *@returns TRUE if the client holds the piece, FALSE in other case.
  *
  **/
  int pieceRequest(string projectName, int piece);
/**
  *The createNewProject operation indicates that a new file will be 
transferred. All Neighbours
  *write a different piece of the file, and start querying all other 
neighbours for the remaining
  *pieces.
  *
  *@param task specifies the information of the new task.
  *@param data is the piece of the file assigned to this neighbour.
  *@param pieceNumber specifies the number of the piece assigned to the 
neighbour.
  *
  **/
  ["ami"] void createNewProject(task t, ByteSeq data, int pieceNumber);
};

My question is, should I use icegrid and have the nodes executing in different computers?
Should I use icegrid and implement a p2p client as an iceBox Service?

Anyway, I must use glacier2 so a client will be able to run behind a router/firewall. But nother problem raises, How could I manage a session with each of the glacier2 routers?

Suggestion are welcome. Thanks in advance.

Comments

  • matthew
    matthew NL, Canada
    IceGrid sounds like a good general solution. Whether or not you use IceBox depends on how you want to deploy your service. Using IceBox allows you to easily change how services are collocation very simple.

    For more background I recommend you read Michi's article "Teach Yourself IceGrid in 10 Minutes" in Issue 19 of connections for information on IceGrid. For file transfer I recommend reading my article "Optimizing Performance of File Transfers" in issue 20. I don't understand your question when about Glacier2. However, for more information I recommend reading "Teach Yourself Glacier2 in 10 Minutes" in issue 22 of Connections. There are numerous other articles that use Glacier2 also in Connections that you might want to reference.
  • Thanks for your reply, matthew.

    My problem with glacier2 is that all of the nodes should run behind a firewall. When a client wants to render a scene, he needs to stablish a session with all of the clients that works for the render task in order to interact with them.

    Is there any way to manage all of these Sessions in a suitable way?

    Thanks in advance.
  • matthew
    matthew NL, Canada
    When you say "clients" it really make things confusing :) Its better to describe the various elements of your system by the role they will play. For example, Renderer and Uploader.

    At any rate, I'm still not sure exactly what your problem is. The Uploader needs to establish a session with Glacier2 in order to interact with the backend objects.

    Once the session has been established the access rights are determined from your configuration of Glacier2. A typical use-case is restrict access to a very limited set of objects (for example, a session object). However, you can configure Glacier2 to allow access to all backend objects if you like.

    It might be easier and less dangerous for you to have the uploader upload the entire scene graph to some object hosted by the session (as I did in the cd ripper app I described in issue 18 & 19) and then have that delegate the rendering to the various renderers.

    I suggest you read the articles I wrote in issue 17, 18, 19 which may make things clearer.