Can ICE support binary data?

I want to send a block of binary data from a client to server through ICE,

does ICE support it ? :confused:

Comments

  • michimichi Member Michi HenningOrganization: Triodia TechnologiesProject: I have a passing interest in Ice :-) ✭✭✭
    Yes, you can do this. Use a sequence of byte:
    sequence<byte> ByteSeq;
    
    interface Transceiver {
        ByteSeq get();
        void put(ByteSeq bs);
    };
    

    Slice byte is an 8-bit byte that is guaranteed not to be tampered with in transit, so it is suitable for sending binary.

    Cheers,

    Michi.
  • GREAT!! Thank you!!
  • nsnsnsns Member ✭✭
    There is a limit on the size of data which can be transfered in this way.
    It equals to 1MB and can be changed to some extent via Ice.MessageSizeMax property.
    To transfer really big amount of binary data (e.g. 1GB) you have to invoke the put() method several times.

    The transfer of files with data is rather typical task and maybe Ice can provide more help,
    e.g. by extending the Slice language with new built-in type 'File' :

    interface Transceiver {
    File get();
    void put(File bs);
    };

    The semantic of File can be similar to 'sequence<byte>' but implementation can be done via memory mapping ('mmap').

    Of course, this approach immediatly rises another question: binary vs text files.

    --
    Cheers, Nikolai
  • soultakersoultaker Member
    How about the performance

    Transfering binary data with sequence would be in bad performance, is it?
    If not, how does ICE solve the issue?
  • marcmarc FloridaAdministrators, ZeroC Staff Marc LaukienOrganization: ZeroC, Inc.Project: The Internet Communications Engine ZeroC Staff
    For some performance figures, please see http://www.zeroc.com/performance/.

    If you are concerned about the overhead for using an std::vector<Ice::Byte>, this really only matters if you have a slow computer with a very fast (gigabit) network. Otherwise the network is the limiting factor, not Ice or std::vector.
  • twofacetwoface Member
    make a bigger data block

    if put Ice::byte to std::vector,the vector will use more memory than our data. We can make a bigger data block,put it into vector.

    #ifndef _ICE_BINBLOCK_ICE
    #define _ICE_BINBLOCK_ICE

    module Trans
    {
    struct BinBlock
    {
    byte b0;
    byte b1;
    byte b2;
    byte b3;
    byte b4;
    byte b5;
    byte b6;
    byte b7;
    byte b8;
    byte b9;
    byte b10;
    byte b11;
    byte b12;
    byte b13;
    byte b14;
    byte b15;
    byte b16;
    byte b17;
    byte b18;
    byte b19;
    byte b20;
    byte b21;
    byte b22;
    byte b23;
    byte b24;
    byte b25;
    byte b26;
    byte b27;
    byte b28;
    byte b29;
    byte b30;
    byte b31;
    };
    sequence<BinBlock> BinBlockList;

    struct BinData
    {
    BinBlockList datalist;
    int size;
    };

    interface Transceiver {
    BinData get();
    void put(BinData data);
    };

    };


    #endif _ICE_BINBLOCK_ICE
  • kinsungkinsung Member kinsungOrganization: AF Engineering UniversityProject: simulation applications remote controler, data dri
    that may be a bad design

    ICE set up many constrains to programmer to keep things simple, that's always OK. But this time, I think this design is bad.

    I developed a applications just like TightVNC, the server can transfer desktop image as JPEG fromat to the client. I encode the image data push_back to a std::vector byte by byte in server side, and decode it to image binary data byte by byte too. When the applications is working, they are not real-time, but feel time delay distinctly.

    Can you leave me a interface to transfer binary data as fast as possible?
  • matthewmatthew NL, CanadaMember Matthew NewhookOrganization: ZeroC, Inc.Project: Internet Communications Engine ✭✭✭
    Perhaps you could explain a little better exactly what you are doing? Did you look at the zero-copy methods that Ice has available for transferring byte sequences? Did you profile your application to find out really where the bottleneck is?

    Also, before we offer you further assistance you must fill in your signature information as detailed in the link contained in my signature.
  • kinsungkinsung Member kinsungOrganization: AF Engineering UniversityProject: simulation applications remote controler, data dri
    I am home, and there is no real code at hand, so the pseudocode can describle it:

    Server side:

    void getDesktpImage(std::vector<BYTE>& imgBuf)
    {
    captuer the desktop image into a buffer[n] //use freeImage library

    imgBuf.reserve(n);
    int i=0;
    for(i=0; i<n; ++i)
    {
    imgBuf.push_back(buffer);
    }
    }


    Client side:

    std::vector<BYTE> imgBuf;
    obj->getDesktpImage(imgBuf);
    int i=0;
    int n=imgBuf.size();
    BYTE* pBuf=new BYTE[n];
    for(i=0; i<n; ++i)
    {
    pBuf=imgBuf);
    }

    use pBuf to display the remote desktop image

    delete []pBuf;


    PS.
    What are the zero-copy methods?
  • matthewmatthew NL, CanadaMember Matthew NewhookOrganization: ZeroC, Inc.Project: Internet Communications Engine ✭✭✭
    Note that vector guarantees that all data is kept in one contiguous segment so you don't need to do the copy one step a time, and on the client side you don't need to copy at all.

    You could do something more like:
    void getDesktpImage(std::vector<BYTE>& imgBuf)
    {
    captuer the desktop image into a buffer[n] //use freeImage library
    
    imgBuf.reserve(n);
    imgBuf.resize(n);
    memcpy(&imgBuf[0], buffer, n);
    }
    

    You can make this a little more efficient because this avoids the initial fill of the vector with zeros.
    vector result(buffer, buffer + n);
    imgBuf.swap(result);
    

    The best solution here is to use AMD and the cpp:range mapping which avoids the additional copy altogether. Something like:
    interface ImageGetter
    {
    ["amd"] void getImageData(["cpp:range"] out ByteSeq data);
    };
    
    void
    ImageGetter::getImageData_async(const AMD_ImageGetter_getImageDataPt& cb, const Ice::Current& current)
    {
       // read n bytes of data into buffer.
       cb->ice_response(pair<const Ice::Byte*, const Ice::Byte*>(buffer, buffer + n));
    }
    

    You an look at the IcePatch2 source for a full example (grep for getFileCompressed_async).

    Note that as the initial article said you probably don't want to send the entire vector all at once either -- assuming the images are big.

    I doubt that Ice has anything to do with your performance problems. A straight copy for all but the most performance and memory intensive applications is no problem -- if your application is implemented correctly. I suggest the use of a good profiler to help isolate the trouble spots.

    You can read up on the zero-copy in the Ice manual, or you can look at my latest article in Connections for an example.
  • kinsungkinsung Member kinsungOrganization: AF Engineering UniversityProject: simulation applications remote controler, data dri
    matthew wrote:
    Note that vector guarantees that all data is kept in one contiguous segment so you don't need to do the copy one step a time, and on the client side you don't need to copy at all.

    I don not know that "vector guarantees that all data is kept in one contiguous segment" at all! :p
    I think my problem has been solved.
    thank you!
Sign In or Register to comment.