Home Help Center

Server truncating a python dictionary

joseangeljoseangel Member jose angelOrganization: PersonalProject: massive execution
Hi I found that servers are truncating python dictionaries. I read a json file, 26 Kb, and then assign it to a python dictionary. The dictionary is truncated. However using the same code in a standalone python script with the same json file all works well.

No exception or error are raised.

Any ideas about this behavior?

There is some server memory parameter that could be responsible of that?

Thanks in advance,

Comments

  • xdmxdm La Coruña, SpainAdministrators, ZeroC Staff Jose Gutierrez de la ConchaOrganization: ZeroC, Inc.Project: Ice Developer ZeroC Staff
    Hi,

    The maximum size of a request is controlled by Ice.MessageSizeMax but in this case you should have received an exception.

    Can you provide a small test case that allow us to reproduce the issue?

    Can you also clarify what Ice version, Python version and operating system are you using?
  • joseangeljoseangel Member jose angelOrganization: PersonalProject: massive execution
    Hi,

    I am using ice 3.5.1 and RHEL 6.7 x86_64 (python 2.6.6).

    The problem is not with the response to the client where applies Ice.MessageSizeMax. By default is 1 MB, if I am not wrong, and the response is 26k more of less.

    The problem occurs in server, before notifying the client, when the json is read.

    The code is:

    with open(locResFile) as fd:
    self.resultados[item] = json.load(fd)
    syslog.syslog("DEBUG " + json.dumps(self.resultados[item]))
    self.resultados[item] = self.node
    fd.close()

    This code is inside a try ... exception and neither exception or error is shown. I have tried to assign the json to the dictionary in several ways:

    self.results[item] = json.loads(fd.read())
    self.results[item] = eval(fd.read())

    and assigning fd.read() to a variable and the using that variable with json.loads and eval. The json is alwais truncated. However, if I write a python script with the same code reading the same json all works well (running it directly in the shell outside ice).

    I have uploaded an example json.

    Thanks in advance,

    P.S. Sorry for the delay. The VMs where I have the app running are in home and I have not got there until now.
  • xdmxdm La Coruña, SpainAdministrators, ZeroC Staff Jose Gutierrez de la ConchaOrganization: ZeroC, Inc.Project: Ice Developer ZeroC Staff
    Hi,

    You wrote that the JSON is always truncated, how do you determine that? if it is because of syslog output it might be because syslog limit the message output. In any case Ice isn't involved in this local variable assignments in anyway.
  • joseangeljoseangel Member jose angelOrganization: PersonalProject: massive execution
    Hi,

    I performed a write of self.results[item] to a file. After closing the current fd:

    fd = open("/tmp/debug.json", "w")
    fd.write(json.dumps(self.results[item]), indent=2)
    fd.close()

    /tmp/debug.json was truncated with the same "string" reported by syslog.

    Thanks in advance,
  • xdmxdm La Coruña, SpainAdministrators, ZeroC Staff Jose Gutierrez de la ConchaOrganization: ZeroC, Inc.Project: Ice Developer ZeroC Staff
    Hi Jose Angel,


    What is self.results and item in your code above? are those types defined in Slice and if so what are the Slice definitions of these types? How are they initialized? if this is in the context on an Ice server method what is the method, and its definition?

    We need many more details in order to determine what is the issue, would be easier if you send us a small test case that we allow us to reproduce the problem.

  • joseangeljoseangel Member jose angelOrganization: PersonalProject: massive execution
    Hi

    they are not defined in Slice, server returns a string, and that's defined in slice.

    I will explain you a bit more.

    I am investigating how to perform massive execution commands so I use zeroc ice to distribute tasks of massive execution.

    A task is a json using several fields, a command to be executed a a list of servers to execute the command.

    When one task gets the server I use ansible to execute the command in all the servers and from that I get a json with the results (stdout, stderr and return code) for each server.

    Item is a hostname of a host in which I executed a command so I am using a loop to create a json with all the information from the execution in servers.

    I am executing commands in one on two servers (VMs in my laptop).

    self.results is initialized as self.results = {}.

    Let me create a small case to reproduce it and send you. Now I am working and I do not have access to my personal laptop where all this is running.

    Thanks in advance,
  • joseangeljoseangel Member jose angelOrganization: PersonalProject: massive execution

    Hi,

    I have the test case, but the issue is not happening, and is doing the same in the same way ¿?. I am not able to upload it, so you can get it from here: https://github.com/jadebustos/cookbooks/tree/master/zerocice/test

  • xdmxdm La Coruña, SpainAdministrators, ZeroC Staff Jose Gutierrez de la ConchaOrganization: ZeroC, Inc.Project: Ice Developer ZeroC Staff

    Hi,

    I looked at your sample and doesn't seems like this issue could be related to Ice. My bet is that there is some issue with json encoding/decoding that truncates the files but is difficult to say without being able to reproduce it.

  • joseangeljoseangel Member jose angelOrganization: PersonalProject: massive execution

    Hi,

    thanks. I think that is due to the size of the string because the trucated string is always similar in size. With fewer sizes it is not truncated. I will try with Ice 3.6.1.

Sign In or Register to comment.