Archived
This forum has been archived. Please start a new discussion on GitHub.
File datatype
Hi !
The transfer of files with data is rather typical task and maybe Ice can provide more help,
e.g. by extending the Slice language with new built-in type 'File' :
interface Transceiver {
File get();
void put(File bs);
};
The semantic of 'File' can be similar to 'sequence<byte>' (i.e. Unix model of files) but implementation can be done via memory mapping ('mmap'), i.e. without the 1MB limit.
Of course, this approach immediatly rises another question: binary vs text files.
Nevertheless, what do you think ?
--
Cheers, Nikolai
The transfer of files with data is rather typical task and maybe Ice can provide more help,
e.g. by extending the Slice language with new built-in type 'File' :
interface Transceiver {
File get();
void put(File bs);
};
The semantic of 'File' can be similar to 'sequence<byte>' (i.e. Unix model of files) but implementation can be done via memory mapping ('mmap'), i.e. without the 1MB limit.
Of course, this approach immediatly rises another question: binary vs text files.
Nevertheless, what do you think ?
--
Cheers, Nikolai
0
Comments
-
We rather don't put special-case types like this in Slice. A sequence of bytes works fine. It's not advisable to transfer such large amounts of data in one call anyway, as regardless of the mapping, Ice internally still has to copy the whole file into a buffer together with header and other parameters. (The complexity of a marshaling engine that tries to avoid this, sometimes also called "zero copy", is extreme, and not a good design trade-off -- but this is a different subject.)
Have a look at how IcePatch is implemented, and how it transfers files in small chunks, all represented as a sequence of bytes.
Note that the 1MB request size limit is not a hard limit. You can set it to any value you like by setting Ice.MessageSizeMax.0 -
The transferring of big files by small chunks is exactly what I do in my application.
This requires some coding because I need to maintain a session-id, and the support from
Ice could be useful.
No, if File was a built-in type, Ice could take care to correctly and efficiently transfer the fileIce internally still has to copy the whole file into a buffer
by small chunks.
--
Cheers, Nikolai0 -
Originally posted by nsns
No, if File was a built-in type, Ice could take care to correctly and efficiently transfer the file
by small chunks.
I'm afraid it's not that easy. Without going into detail, this would require extensive changes to the Ice protocol and marshaling engine. The result would be a much more complex Ice core, only for the benefit of one special-purpose use-case, which can also be handled at the application level.0 -
I think it easy to use sequence of bytes to transfer file through programming extra code, and to support new data type may cause many problems.0
