Request For IceGrid/IcePatch2 Demo Example

afcarlafcarl Andy CarlOrganization: Northrop Grumman CorpProject: Distributed analysis of air vehicle subsystemsMember ✭✭
Is there an available example of using IceGrid together w/ IcePatch2 doing an automated distribution of server files? The kind of example that would normally be placed in the "demo/IceGrid" folder, and not just the MP3ripper example from the user's manual? This would be most appreciated!
Thanks,
Andy

Comments

  • matthewmatthew NL, CanadaMatthew NewhookOrganization: ZeroC, Inc.Project: Internet Communications EngineMember ✭✭✭
    There is an an example of this in issue 9 of our newsletter connections.

    See the Distributions section of my IceGrid article for details. The sample code has a full demo, however, since this is rather old now it might not work out of the box any longer.

    If you have any issues please let me know.
  • afcarlafcarl Andy CarlOrganization: Northrop Grumman CorpProject: Distributed analysis of air vehicle subsystemsMember ✭✭
    Frustration

    Matthew,
    Thanks for the pointer to the newsletter article. Unfortunately, the combined effect of no trace output w/ the following settings in the *.cfg file:

    ########################################################
    # Trace properties
    ########################################################

    Ice.Warn.Connections=1

    # Only 0 and 1
    IceGrid.Registry.Trace.Application=1
    IceGrid.Registry.Trace.Adapter=1
    IceGrid.Registry.Trace.Object=1
    IceGrid.Registry.Trace.Patch=1
    IceGrid.Registry.Trace.Replica=1
    IceGrid.Registry.Trace.Server=1

    # Between 0 and 2
    IceGrid.Registry.Trace.Locator=2
    IceGrid.Registry.Trace.Node=2
    IceGrid.Registry.Trace.Session=2

    # Between 0 and 3
    IceGrid.Node.Trace.Activator=3
    IceGrid.Node.Trace.Adapter=3
    IceGrid.Node.Trace.Patch=3
    IceGrid.Node.Trace.Replica=3
    IceGrid.Node.Trace.Server=3

    and the gleened information from your article which I am unable to get to work, contributes to a growing since of frustration.

    The combined features of IceGrid and IcePatch2 are very attractive. But I have as yet not been sucessful to get any automated distribution, though the command-line invocation of icepatch has worked.

    It would seem that a complete working demo example would go a long way to debug the issue, rather than a gleened example from an obsolete version or the immagenary MP3ripper example from the user's manual.

    Given the multiplicity of ICE functionality combinations, an increase in the number of demo examples would go a long way to facilitate utilization of ICE.

    I'm not meaning to be negative, but as stated above, the combined effect of lack of trace output and lack of working IceGrid/IcePatch2 example is frustrating. A working example "out-of-the-box" would be most appreciated!

    Thanks,
    Andy
  • benoitbenoit Rennes, FranceBenoit FoucherOrganization: ZeroC, Inc.Project: IceAdministrators, ZeroC Staff ZeroC Staff
    Hi Andy,

    I'm sorry to hear you're having troubles with IceGrid distributions.

    Until we solve your tracing issue and assuming this is what you're using, I would recommend to disable the stdout/stderr file redirection for IceGrid node servers by not setting the IceGrid.Node.Output property in the configuration file of your IceGrid node. This way the stderr/stdout of your servers will go to the stdout/stderr of your IceGrid node and you will be able to see the traces. If you don't set this property, we would need to know more about the configuration of your IceGrid node and registry.

    That being said, it's not clear to me why the tracing of the server matters here. If I understand it correctly, your problem is with the IceGrid distributions. So what is not working as you expect here? If you tell us a bit more about this problem, I'm sure we can help. A short description of your application and posting the configuration files of your IceGrid node and registry as well as the application descriptor would be good.

    In the meantime, I'll see if I can provide you with a small demo for IceGrid/IcePatch2.

    Cheers,
    Benoit.
  • benoitbenoit Rennes, FranceBenoit FoucherOrganization: ZeroC, Inc.Project: IceAdministrators, ZeroC Staff ZeroC Staff
    Hi,

    I've attached to this post a small demo based on the IceGrid simple demo It's using IcePatch2 and IceGrid distributions. The application descriptor specifies one distribution directory for the application and another one for the SimpleServer server.

    Each distribution directory contains a small dummy text file which is loaded by the server (the server prints the content on stdout). The text files loaded by the server are the ones downloaded from the IcePatch2 server by the IceGrid node (the downloaded files are downloaded to the db/node/distrib and db/node/servers/SimpleServer/distrib directories).

    To try out this demo, you should unzip the icepatch.zip file content in the C:\Ice-3.3.1\demo\IceGrid directory. Then open the C:\Ice-3.3.1\demo.sln solution with Visual Studio and add the projects from the icepatch2 demo directory to the solution and compile both the server and client using the projects you just added. Finally, follow the instructions from the README file to run the demo.

    Let me know if you have any troubles with running this demo.

    Cheers,
    Benoit.
  • afcarlafcarl Andy CarlOrganization: Northrop Grumman CorpProject: Distributed analysis of air vehicle subsystemsMember ✭✭
    Trace Output

    Benoit,
    The problem is three-fold. First, if the IceGrid.Node.Output property is not set, there is no trace output even with the settings previous posted. Second, if the IceGrid.Node.Output property is set, there is trace output on only some servers but not all. Third, if there is some error in the xml deployment discriptor, there is no trace output at all, regardless of the setting of the IceGrid.Node.Output property.

    You are correct that the underlying problem is that of IceGrid/IcePatch2 automated deployment, or more specifically, the lack thereof. The reason for the frustration regarding the lack of trace output is the desire to troubleshoot the IceGrid/IcePatch2 automated deployment failure. This reduces the process to one of trail and error, which has been very time consuming to the tune of three man-days effort so far. Thus my motivation and hope for a working simple example of an IceGrid/IcePatch2 automated deployment.
    Thanks,
    Andy
  • benoitbenoit Rennes, FranceBenoit FoucherOrganization: ZeroC, Inc.Project: IceAdministrators, ZeroC Staff ZeroC Staff
    Hi,

    I'm afraid it's difficult to help if you don't provide more information. Do you use an Ice binary distribution (which one?) or did you build Ice from sources?

    How do you run the IceGrid node and registry, from a command shell or as a windows service? If you run it as a Windows service, can you try to run it with the command shell to see if you're seeing any output? Do you see some tracing when running the IceGrid demos from a command shell (the IceGrid/simple demo for example)?

    Cheers,
    Benoit.
  • afcarlafcarl Andy CarlOrganization: Northrop Grumman CorpProject: Distributed analysis of air vehicle subsystemsMember ✭✭
    Thanks for the IceGrid/IcePatch2 Example

    Benoit,

    Thanks for the IceGrid/IcePatch2 example demo. It really should be put into the demo/IceGrid distribution folder.

    The provided example works as advertised, but the upshot of the matter appears to be that the OMERO ICE implementation has broken some things related to TRACE and ICEPATCH2. I am trying to run it to ground, but at first blush, it appears that your deployment descriptor, when placed in the OMERO master node, does not work. And the apparent broken TRACE functionality has made it appear that ICE was broken. I'll have to get back w/ the OMERO developers on this one.

    Again, Thanks!
    Andy
  • benoitbenoit Rennes, FranceBenoit FoucherOrganization: ZeroC, Inc.Project: IceAdministrators, ZeroC Staff ZeroC Staff
    Hi Andy,

    Yes, we'll include this demo or something along these lines in the next release. Let us know if you need any additional information to help solve this tracing problem.

    Cheers,
    Benoit.
  • afcarlafcarl Andy CarlOrganization: Northrop Grumman CorpProject: Distributed analysis of air vehicle subsystemsMember ✭✭
    The Latest Twist

    Benoit,

    Using the supplied example provided by you as a guide, I adapted the OMERO ICE deployment descriptor to perform as advertised (by Zeroc), as long as the IceGrid invocation and IceGridAdmin deployment were both performed from the command-line.

    The identicle configuration and deployment when invoked as a service, caused both the TRACE and IcePatch2 functionality to fail. Since this is the standard method for the OMERO deployment, any progress has been painfully non-existant, until I received your supplied example demo.

    I am unclear as to why it should make any difference between being deployed as a service or from the command-line interface. Please advise if this is a "ICE" issue, or an "OMERO" issue.

    Thanks,
    Andy
  • matthewmatthew NL, CanadaMatthew NewhookOrganization: ZeroC, Inc.Project: Internet Communications EngineMember ✭✭✭
    If it works when run on the command line, but not when run as a service then its likely that the issue is related to permissions.
  • benoitbenoit Rennes, FranceBenoit FoucherOrganization: ZeroC, Inc.Project: IceAdministrators, ZeroC Staff ZeroC Staff
    Hi Andy,

    I don't know much about Omero but if it installs the IceGrid registry and node as Windows services, you should find the tracing in the Windows Event Viewer like other Windows services. Did you check out Windows event logs to see if there was any tracing?

    Cheers,
    Benoit
  • afcarlafcarl Andy CarlOrganization: Northrop Grumman CorpProject: Distributed analysis of air vehicle subsystemsMember ✭✭
    Permission Related Hypotheses

    I don't believe it is permissions related for the following reasons:
    a) Service runs as "SYSTEM", Command-line runs as "user"
    b) The same identical TRACE output files are created in both instances, but the content is different
    c) There are no read or write errors contained in what small TRACE output from the SYSTEM deployment

    Regards,
    Andy
  • joshmoorejoshmoore GermanyJosh MooreOrganization: Glencoe Software, Inc.Project: OME, http://openmicroscopy.orgMember
    Benoit,

    just wanted to confirm: Omero does start the IceGrid registry as a service using the local system user. iceserviceinstall isn't used since the registry is collocated with the node. Instead, sc.exe is used directly. Regardless, the trace statements I've needed (Trace.Activator, etc.) have shown up in the Event Viewer. In fact, a while ago I uploaded a (very poor quality) movie to youtube to explain just that to a user:

    YouTube - EventViewer

    Cheers,
    ~Josh
  • benoitbenoit Rennes, FranceBenoit FoucherOrganization: ZeroC, Inc.Project: IceAdministrators, ZeroC Staff ZeroC Staff
    Hi,

    Josh, thanks for the additional information. So the IceGrid node here is deployed as a Windows service by the Omero scripts/installers and it collocates the IceGrid registry. Unfortunately, many things can go wrong when using Windows services :( Btw, other than the fact that iceserviceinstall doesn't support registry collocation with the node, are there any other reasons why you're not using it?

    Andy, again, when deployed as Windows services, the IceGrid node & registry tracing should go to the Windows event log and can be accessed with the Event Viewer. Let us know if nothing shows up there.

    For servers spawned by the IceGrid node ran as a Windows services, their output goes nowhere unless their stderr and stdout are redirected to files. This can easily be done by setting the IceGrid.Node.Output property to the path of a directory in the IceGrid node configuration file. You must make sure that this path is accessible by the account running the IceGrid node service. I don't know if this property is set by default by the Omero distribution, if not, I would recommend to set it (and have the Omero installer to ensure that the path has the correct access rights).

    Now on top of this and as mentioned on the other thread, another issue is that python Ice servers currently don't support the redirection of stderr/stdout to files. We're looking into whether or not this can be fixed. But in the meantime you can't rely on being able to checkout your python server's stderr/stdout if you run the IceGrid node as a Windows service (it will work if you run it from the command line though).

    If possible, I would really recommend to setup a command line environment here for development purposes (if Omero doesn't provide scripts to run the services from the command line, it might be good to add this). It's best to use Windows services once everything works because many other things can go wrong when moving to Windows services (access rights for instance) and as you unfortunately discovered, debugging Windows services related problems can be very frustrating.

    Cheers,
    Benoit.
  • joshmoorejoshmoore GermanyJosh MooreOrganization: Glencoe Software, Inc.Project: OME, http://openmicroscopy.orgMember
    benoit wrote: »
    Josh, thanks for the additional information.
    Gladly.
    benoit wrote: »
    So the IceGrid node here is deployed as a Windows service by the Omero scripts/installers and it collocates the IceGrid registry.

    Yep.
    benoit wrote: »
    Unfortunately, many things can go wrong when using Windows services :(

    Agreed. Especially the system user/permissions issues have been, um, interesting to track down.
    benoit wrote: »
    Btw, other than the fact that iceserviceinstall doesn't support registry collocation with the node, are there any other reasons why you're not using it?

    That's pretty much the only reason. For the moment, we're wrapping sc.exe but would definitely use iceserviceinstall if collocation was supported.
    benoit wrote: »
    service. I don't know if this property is set by default by the Omero distribution, if not, I would recommend to set it (and have the Omero installer to ensure that the path has the correct access rights).

    The configuration values in effect are:
    IceGrid.Node.Data=c:\omero.server-build1048\var\master
    Ice.StdOut=c:\omero.server-build1048\var\log\master.out
    Ice.StdErr=c:\omero.server-build1048\var\log\master.err
    

    and Node.Output is unset.
    benoit wrote: »
    Now on top of this and as mentioned on the other thread, another issue is that python Ice servers currently don't support the redirection of stderr/stdout to files. We're looking into whether or not this can be fixed. But in the meantime you can't rely on being able to checkout your python server's stderr/stdout if you run the IceGrid node as a Windows service (it will work if you run it from the command line though).
    Interesting. Didn't realize this. We use the logging module in Python, so hadn't run into this.

    Thanks, Benoit.
    ~J.
  • benoitbenoit Rennes, FranceBenoit FoucherOrganization: ZeroC, Inc.Project: IceAdministrators, ZeroC Staff ZeroC Staff
    Hi John,
    The configuration values in effect are:
    IceGrid.Node.Data=c:\omero.server-build1048\var\master
    Ice.StdOut=c:\omero.server-build1048\var\log\master.out
    Ice.StdErr=c:\omero.server-build1048\var\log\master.err
    

    and Node.Output is unset.

    I would recommend to not set Ice.StdOut/Ice.StdErr when running the IceGrid node as a service since nothing is logged there and this can be a source of confusion. A comment saying that the IceGrid node logging goes to the event log would be better :)

    You should also consider setting IceGrid.Node.Output to the Omero var/log directory to enable stderr/stdout file redirection for servers spawned by the IceGrid node (another option is to do this on a per-server basis by setting the Ice.StdOut/Ice.StdErr properties in the server configuration properties if you don't want to keep the stderr/stdout of all the servers).

    Cheers,
    Benoit.
  • joshmoorejoshmoore GermanyJosh MooreOrganization: Glencoe Software, Inc.Project: OME, http://openmicroscopy.orgMember
    Benoit,

    The Ice.Std{Out,Err} are set primarily to prevent confusion, since Omero uses property overriding as a work around for:

    http://www.zeroc.com/forums/comments/3774-minor-rfe-property-expansion.html

    I.e. there's a global config (master.cfg) and then on Windows "windows.cfg" is appended to the Ice.Config list. If the properties are no use on Windows, I could alternatively set them to empty. However, I assumed that with "Node.Output" unset, the output from the servers in the node would still get combined to the nodes "Ice.StdOut/Ice.StdErr"

    Is this not the case? (Our servers use explicit logging and not stdout/err, so this isn't currently an issue, but might help prevent future confusion)
    Thanks,
    ~Josh.
  • benoitbenoit Rennes, FranceBenoit FoucherOrganization: ZeroC, Inc.Project: IceAdministrators, ZeroC Staff ZeroC Staff
    Hi John,

    The stderr/stdout redirection of servers spawned by the IceGrid node to the node stderr/stdout files unfortunately doesn't work when the IceGrid node is started as a service (I believe this is because Windows services don't have a console). It only works if the IceGrid node is started interactively from a console. So instead of setting Ice.StdOut/Ice.StdErr in the node configuration file (when the node is started as a Windows service at least), you should consider using IceGrid.Node.Output to redirect the stderr/stdout of each server to individual files.

    Cheers,
    Benoit.
  • joshmoorejoshmoore GermanyJosh MooreOrganization: Glencoe Software, Inc.Project: OME, http://openmicroscopy.orgMember
    Benoit,

    thanks for the info. I've add a ticket on our side to adjust this. Did I miss the documentation for this somewhere? If not, it'd be good to have it in the Windows Service section and/or as a demo. I'm finally getting the hang of the Windows deploy, but it is certainly a special beast.

    Cheers,
    ~Josh.
  • benoitbenoit Rennes, FranceBenoit FoucherOrganization: ZeroC, Inc.Project: IceAdministrators, ZeroC Staff ZeroC Staff
    Hi Josh,

    This isn't very well documented at present, we'll fix this for the next release!

    Cheers,
    Benoit.
Sign In or Register to comment.