Beginner Streaming question

rdaneelrdaneel Member Iman AwaadOrganization: FH-BRSProject: XPERSIF
Hello,

We're very new with streaming and would like to make sure we have understood correctly. We have our project's datatypes defined in slice in a specific Ice file. We would like to stream only one of these datatypes. It is used in two different interfaces (so all in all, it appears in three ice files).

We would like to send an image frame which is currently simply a sequence of ints: sequence<int> imageBGR; (This will most likely change to a struct which includes the width and height later). Currently, width and height are the two remainig out params of the function in addition to that of imageBGR.

Should we add the --stream option to all three ice files? Also, the example given in the Ice Manual (35.2) ends with a line: out->finished(data). This line is not present in the example present in the Connections (issue 12) article. There, the format is slightly different: ice_writeBasicStruct(out,b);

Is it the case then that this helper function is generated by adding the --stream to the slice file containing the function which is called with imageBGR variableName sent as an outparam? And that in the case the helper function is used (like ice_writeBasicStruct above) that the line out->finished data is then somehow already called when the whole struct is sent?

Many thanks.

Comments

  • mesmes CaliforniaAdministrators, ZeroC Staff Mark SpruiellOrganization: ZeroC, Inc.Project: Ice Developer ZeroC Staff
    Hi,
    rdaneel wrote: »
    Should we add the --stream option to all three ice files?
    You only need to use the --stream option when translating the Slice files that contain the types you wish to stream. If you are only streaming simple types such as a sequence of integers, there is no need to specify the --stream option at all because you would use methods provided by the OutputStream class such as writeIntSeq.
    Also, the example given in the Ice Manual (35.2) ends with a line: out->finished(data). This line is not present in the example present in the Connections (issue 12) article. There, the format is slightly different: ice_writeBasicStruct(out,b);
    The call to finished is always necessary, it just wasn't shown in the article for the sake of brevity.

    Forgive me for asking, but you do have a good reason for using streaming, right? :)

    Take care,
    - Mark
  • rdaneelrdaneel Member Iman AwaadOrganization: FH-BRSProject: XPERSIF
    Thank you!

    Well, I think we do.. we tried to do it simply as a vector of ints but we got an error along the lines of protocol limit exceeded.. we need to send an image - around 420x600 pixels x3 channels.. we could only send a quarter of the image.

    We actually have the three apps A B C and we need to get the image from C, through B to A.. where it is redisplayed.. we ask about streaming slice-defined types since we are hoping to change the sequence<int> for a struct with two ints and the same secuence we are currently sending.

    Is there perhaps another method? In several posts, there was something about increasing the message size in the config files.. but I assumed this was only done when actually streaming..
  • marcmarc FloridaAdministrators, ZeroC Staff Marc LaukienOrganization: ZeroC, Inc.Project: The Internet Communications Engine ZeroC Staff
    Using Dynamic Ice does not increase the max. request size.

    Have a look at these two FAQs:

    http://www.zeroc.com/faq/requestSizeLimit.html
    http://www.zeroc.com/faq/fileTransfer.html

    Also, I highly recommend to read the article "Optimizing Performance of File Transfers" in issue 20 of our newsletter "Connections":

    http://www.zeroc.com/newsletter/issue20.pdf
  • rdaneelrdaneel Member Iman AwaadOrganization: FH-BRSProject: XPERSIF
    Thank you very much for the links! I've read the first two.. they were indeed very helpful!! :) I am currently reading the connections article which is also really great! I am just wondering if you still think streaming may not be a good choice for sending back the images..

    Application C is the simulator (A is the client) - and we would have liked to send at least 30frames/s (we get 100 or so on the machine running C). We lose time already by copying the buffer from the GPU to the CPU and although its a lot easier to save the contents to a file and transfer it as presented in the newsletter for example, I am thinking we lose a lot more time in this process of writing and reading to and from disk as opposed to simply copying the data from one buffer in cpu to the vector we wish to stream.. what would be the "best practices" in our case?

    Thanks again! I'm much obliged!
    regards,
    Iman
  • marcmarc FloridaAdministrators, ZeroC Staff Marc LaukienOrganization: ZeroC, Inc.Project: The Internet Communications Engine ZeroC Staff
    The newsletter shows different methods for how to transfer large amounts of data. You can easily adopt the example to transfer something other than files. This newsletter article describes "best practices", so let me suggest that you study this article first.
  • rdaneelrdaneel Member Iman AwaadOrganization: FH-BRSProject: XPERSIF
    Thanks again for the links. We've gone through them and thought we'd at least start by changing the MessageSizMax property, but it seems we must be doing something wrong..

    This is the config file for the two applications + the node:

    IceGrid.InstanceName=XPERO-IceGrid

    #
    # The IceGrid locator proxy.
    #
    Ice.Default.Locator=XPERO-IceGrid/Locator:tcp -h localhost -p 11002

    IceGrid.Node.Name=beaLaptop
    IceGrid.Node.Endpoints=tcp -p 12000
    IceGrid.Node.Data=db/node
    #IceGrid.Node.CollocateRegistry=1
    #IceGrid.Node.Output=db
    #IceGrid.Node.RedirectErrToOut=1
    Ice.MessageSizeMax=16384

    IceGrid.Node.Trace.Activator=1
    IceGrid.Node.Trace.Patch=1
    Ice.Trace.Network=2
    Ice.Trace.Protocol=1
    Ice.ServerIdleTime=3600
    #IceGrid.Node.Trace.Adapter=2
    #IceGrid.Node.Trace.Server=3

    For the client:

    Ice.Default.Locator=XPERO-IceGrid/Locator:default -h localhost -p 11002
    ControlObserverOfAdapter.Endpoints=tcp
    Ice.MessageSizeMax=16384

    The actual size of the data we need to transfer is 3094Kb, so it should be more than enough.. we've restarted the node, even updated the servers in icegridadmin.. have we missed something?

    Many thanks
  • mesmes CaliforniaAdministrators, ZeroC Staff Mark SpruiellOrganization: ZeroC, Inc.Project: Ice Developer ZeroC Staff
    So you are still hitting the maximum message size? Which application raises the exception?

    My guess is that it is a server started by IceGrid; if so, you should add a definition for the Ice.MessageSizeMax property to your IceGrid server descriptors.

    As Marc mentioned, streaming does not allow you to circumvent the maximum message size. Streaming is most useful in situations where messages must be forwarded without knowledge of their contents, and in applications where type information is obtained dynamically.

    Take care,
    - Mark
  • rdaneelrdaneel Member Iman AwaadOrganization: FH-BRSProject: XPERSIF
    Thank you very much Mark! I realize now that dynamic Ice and streaming isn't what we thought it was in fact!

    We've implemented the first example from the newsletter and sent the buffer in parts (and it works!!:) But I am still curious why we hit the maximum size, when trying to send it in one go.

    My question is when you say to add it to the server descriptors, do you mean the XML files? (something like <property name="Ice.MessageSizeMax" value="16384"/> for each server?) Sorry for such a silly q.. but as you've probably gathered, this has all been very new for us! :)
  • mesmes CaliforniaAdministrators, ZeroC Staff Mark SpruiellOrganization: ZeroC, Inc.Project: Ice Developer ZeroC Staff
    Hi,
    rdaneel wrote: »
    My question is when you say to add it to the server descriptors, do you mean the XML files? (something like <property name="Ice.MessageSizeMax" value="16384"/> for each server?)
    Yes, that's what I meant. If you need to increase the maximum message size, you have to make sure that you increase it in every application that requires the larger message size. Setting the property in the IceGrid configuration file likely won't make any difference because your client doesn't need to exchange large messages with IceGrid but with the servers that IceGrid is managing.

    For example:
    <application ...>
        <node name="TheNode">
            <server id="TheServer" exe="./server">
                <properties>
                    <property name="Ice.MessageSizeMax" value="16384"/>
                </properties>
            </server>
        </node>
    </application>
    
    This will add a property definition for Ice.MessageSizeMax to the configuration file that IceGrid generates automatically for the server.

    Take care,
    - Mark
Sign In or Register to comment.