Archived
This forum has been archived. Please start a new discussion on GitHub.
IcePy can not handle buffers with longs (I64)
Hi everyone,
For a project we are working on we wish to send a numpy array as quick as possible. We match the numpy array datatype to the type defined in our Slice definition, and wrap it in a buffer. Now this works fine for doubles and ints, and it is awesomely quick. However, for longs (64bit signed) it gives a "expected sequence value" exception. Now we've traced it down to this point in IcePy (Types.cpp, line: 1066).:
However, we do not wish to keep a separate version of Ice, and besides I do not know if there is a deeper reason why this is not possible. I would not like to bump into odd failures in the future because we didn't know what we where doing .
Regards,
Vincent
For a project we are working on we wish to send a numpy array as quick as possible. We match the numpy array datatype to the type defined in our Slice definition, and wrap it in a buffer. Now this works fine for doubles and ints, and it is awesomely quick. However, for longs (64bit signed) it gives a "expected sequence value" exception. Now we've traced it down to this point in IcePy (Types.cpp, line: 1066).:
void IcePy::SequenceInfo::marshalPrimitiveSequence(const PrimitiveInfoPtr& pi, PyObject* p, const Ice::OutputStreamPtr& os) { // // For most types, we accept an object that implements the buffer protocol // (this includes the array.array type). // const void* buf = 0; Py_ssize_t sz; if(PyObject_AsReadBuffer(p, &buf, &sz) == 0) { const Ice::Byte* b = reinterpret_cast<const Ice::Byte*>(buf); switch(pi>kind) { case PrimitiveInfo::KindBool: { os>writeBoolSeq(reinterpret_cast<const bool*>(b), reinterpret_cast<const bool*>(b + sz)); break; } // [ ... ] case PrimitiveInfo::KindLong: // ??? { PyErr_Format(PyExc_ValueError, STRCAST("expected sequence value")); throw AbortMarshaling(); } case PrimitiveInfo::KindFloat: { os>writeFloatSeq(reinterpret_cast<const Ice::Float*>(b), reinterpret_cast<const Ice::Float*>(b + sz)); break; } case PrimitiveInfo::KindDouble: { os>writeDoubleSeq(reinterpret_cast<const Ice::Double*>(b), reinterpret_cast<const Ice::Double*>(b + sz)); break; } // [ ....]As you can see it accepts every primitive kind, except long. Now we modified this piece of code to accepts longs, exactly the same as with the other primitives, and it seems to work fine.
However, we do not wish to keep a separate version of Ice, and besides I do not know if there is a deeper reason why this is not possible. I would not like to bump into odd failures in the future because we didn't know what we where doing .
Regards,
Vincent
0
Comments

numpy can handle those, yes. If that could be included in the future releases that would be great!
Thanks,
Vincent0 
+1 I'll add a workaround for the moment, but longs are a sizeable portion of what we'd be serializing from numpy.0

Also, if possible supporting byte sequences would be beneficial:
In [80]: numpy.array([1,2,3], dtype=tables.UInt8Atom()) Out[80]: array([1, 2, 3], dtype=uint8) In [81]: numpy.array([1,2,256], dtype=tables.UInt8Atom()) Out[81]: array([1, 2, 0], dtype=uint8) ... In [83]: a = numpy.array([1,2,3], dtype=tables.UInt8Atom()) In [84]: print a [1 2 3] In [85]: omero.grid.MaskColumn(None, None, None, None, None, None, None, None, None, [a]) Out[85]: object #0 (::omero::grid::MaskColumn) { name = None description = None imageId = {} theZ = {} theT = {} x = {} y = {} w = {} h = {} bytes = { [0] = { [0] = <invalid value  expected byte> [1] = <invalid value  expected byte> [2] = <invalid value  expected byte> } } } ... In [90]: a Out[90]: array([ 1, 2, 128], dtype=int8) In [91]: a = numpy.array([1,2,127], dtype=tables.Int8Atom()) In [92]: a Out[92]: array([ 1, 2, 127], dtype=int8) In [93]: omero.grid.MaskColumn(None, None, None, None, None, None, None, None, None, [a]) Out[93]: object #0 (::omero::grid::MaskColumn) { name = None description = None imageId = {} theZ = {} theT = {} x = {} y = {} w = {} h = {} bytes = { [0] = { [0] = <invalid value  expected byte> [1] = <invalid value  expected byte> [2] = <invalid value  expected byte> } } }
Or do the various ranges for Ice::Byte (128,127 or 0, 255) make this difficult/impossible?0 
These issues will be addressed in Ice 3.4
Regards,
Mark0 
Hi Mes, that's great to hear.
Not knowing exactly what the plans are, I'll add more feedback about what I'm running into with 3.3 here, but will retest everything with 3.4.
Most recently we've encountered a 64bit issue. For the following slice:module b { sequence<int> IntArray; class c { IntArray i; }; };
numpy arrays of type 'i4' are considered invalid.import b import os import Ice import numpy os.system("uname a") print "Ice", Ice.stringVersion() def test(name, array): d = b.c() d.i = array s = str(d) i = s.find("invalid") print name, "... ", i<0 and "Ok" or "Failed" test("Python array", [1,2]) arr4 = numpy.array([1,2], dtype='i4') test("numpy array i4", arr4) test("numpy array i4 tolist", arr4.tolist()) arr8 = numpy.array([1,2], dtype='i8') test("numpy array i8", arr8) test("numpy array i8 tolist", arr8.tolist())
works on 32 bit mac:Darwin mac 8.11.1 Darwin Kernel Version 8.11.1: Wed Oct 10 18:23:28 PDT 2007; root:xnu792.25.20~1/RELEASE_I386 i386 i386 Ice 3.3.0 Python array ... Ok numpy array i4 ... Ok numpy array i4 tolist ... Ok numpy array i8 ... Ok numpy array i8 tolist ... Ok
and slightly less so on 32bit Linux:Linux valewalker 2.6.24gentoor2 #2 SMP Sat Feb 23 10:40:02 GMT 2008 i686 Intel(R) Xeon(TM) CPU 1.60GHz GenuineIntel GNU/Linux Ice 3.3.1 Python array ... Ok numpy array i4 ... Ok numpy array i4 tolist ... Ok numpy array i8 ... Failed numpy array i8 tolist ... Ok
but on a 64bit Linux machine:Linux warlock 2.6.29gentoor5 #1 SMP Wed Jul 8 21:59:17 BST 2009 x86_64 AMD Opteron(tm) Processor 252 AuthenticAMD GNU/Linux Ice 3.3.1 Python array ... Ok numpy array i4 ... Failed numpy array i4 tolist ... Ok numpy array i8 ... Ok numpy array i8 tolist ... Ok
If possible, it would be good to not have to manually convert numpy arrays based on whether the local machine is 32 or 64 bit.
Thanks, ~J.0 
Hi Josh,
I ran your test in our Ice 3.4 development tree on Linux (x86 & x64) and it works correctly. Note however that the range of your array elements must still comply with the limits of the Slice data types:>>> d.i = numpy.array([1,2,9223372036854775808], dtype='i8') >>> d.i array([1, 2, 9223372036854775808], dtype=int64) >>> d object #0 (::b::c) { i = { [0] = 1 [1] = 2 [2] = <invalid value  expected int> } }
Regards,
Mark0