Archived

This forum has been archived. Please start a new discussion on GitHub.

php5-Ice on FreeBSD 7.1

Hi,

One facet of our small company is providing commercial hosting for Mumble, and one of the things we've been working on is integrating a Murmur control panel via Ice.

At this point all of our nodes are on either budget servers or VPSes running Linux, but no one in our shop is entirely confident administrating linux machines so we're trying to migrate everything to some nice FreeBSD servers as we outgrow the others.

I'm having some bother using php5-Ice on FreeBSD. Long story short, from my observation, a php5-Ice client that's on FreeBSD will hang in this state when trying to connect to an Ice server built on FreeBSD:

12401 fwaggle 1 44 0 85276K 16064K umtxn 0 0:00 0.29% php

This only occurs when both the client and server are running FreeBSD (we're running amd64 if that matters). I can connect from our FreeBSD host (where the prototype panel is running) to Linux Ice endpoints no problem - so great, I thought, the Ice server's busted on FreeBSD.

Except that on further testing, I can run an Ice php script on a Linux machine and it connects to a FreeBSD endpoint no problems. It goes without saying of course I've had no troubles linux->linux.

The problem appears to be isolated to the PHP bindings for Ice, because a simple client I knocked together in C++ to stop/start the first Mumble instance works fine. I later installed the python bindings and python works fine too.

We were planning on re-building our panel in python anyway, but I don't want to do it yet - but I have no idea where to start looking at php5-ice... anyone have any ideas?

Comments

  • Which versions of ice and ice-php are installed? Are they installed from the ports tree? Last time I checked ice itself was 3.3.1 in ports, while ice-php was 3.3.0, so maybe this is a binary compatibility problem. Are the server and client are running on the same host?

    I would be very interested in your observations, because we're planning to roll out a very similar setup (FreeBSD 7.1, jailed apache+icephp, servers running within a different jail, amd64)
  • Hi Michael,

    devel/ice is as 3.3.1 now, but I've tried pkg_adding v3.3.0 as well and it makes no difference. devel/php5-ice registers itself as v3.3.0 so perhaps it needs updating, but I'm not sure. It appears to build from the Ice-3.3.1 tree so I think it just registers itself in portsdb incorrectly.

    I have two FreeBSD/amd64 machines and three CentOS Linux machines - I've tried every possible connection direction, and achieve the following results every time regardless of which respective machines I use:

    FreeBSD->FreeBSD = fail
    FreeBSD->Linux = works
    Linux->FreeBSD = works
    Linux->Linux = works

    devel/php5-ice appears broken on my 7.0/i386 machine, it drops core every time I add it to PHP. :(
  • Hmm thats's actually too bad.

    Right now I've got no amd64 machine to play with (my dev machine runs CURRENT i386 from December last year). Maybe there's somebody out there using the same config. I will try to get this stuff up and running on new servers within the next few weeks, but maybe somebody else already solved this in the past and could assist in the meantime? I'm currently trying to establish Ice as a CORBA replacement in our company, so having that stuff work is very important to me. Maybe you could give some more details (network, php configuration etc.) to give people an idea of what your setup looks like.

    I've got +10 years experience using FreeBSD and +5 years using CORBA but practically zero experience using ICE, so getting some ideas from the ICE developers might be helpful.
  • grembo wrote: »
    my dev machine runs CURRENT i386 from December last year

    Do you use devel/php5-ice on that machine? Every i386 FreeBSD machine I'm running does this:
    terminate called after throwing an instance of 'IceUtil::ThreadSyscallException'
    what(): RecMutex.cpp:96: IceUtil::ThreadSyscallException:
    syscall exception: Invalid argument
    Abort (core dumped)

    That's just on executing /usr/local/bin/php :( Only way to get it to not do that is to remove the IcePHP library.

    It's not Ice-related, it's purely isolated to the bindings in php5-ice - py-ice works fine.
    I've got +10 years experience using FreeBSD

    Same here, I've been using it in shells since about 3.4, and installing it on my own machines since 4.0 was brand new. I try every now and then to get into Linux, and I quite like CentOS - but FreeBSD just feels like home, I know where everything is. It just sucks that it seems like for running many different kinds of things such as gameservers, Ice, things of that nature, FreeBSD just isn't as easy to work with as Linux (and, for example, ZeroC don't seem to support FreeBSD at all unless you have a support contract with them).
  • I get the same abort when calling command line php. Maybe the standalone version isn't compiled using -pthread?

    When using the apache module it seems to work ok, I tested this using the minimal hello example from the ice distribution. My setup is:

    ice 3.3.0 (not 3.3.1)
    ice-php 3.3.0
    (my ports tree dates back before the upgrade to 3.3.1)

    Python server listening on IP 192.168.250.33 port 10000, client running on apache within a jail (192.168.250.34).

    test.php script:
    <?php
    
    Ice_loadProfile();
    try
    {
        $base = $ICE->stringToProxy("hello:default -h 192.168.250.33 -p 10000");
        $hello = $base->ice_checkedCast("Demo::Hello");
        $hello->sayHello();
    }
    catch(Ice_LocalException $ex)
    {
        print_r($ex);
    }
    

    On the server side I run Server.py from the minimal demo, but I had to change it to force it to listen to 192.168.250.33:
    adapter = communicator.createObjectAdapterWithEndpoints("Hello", "tcp -h 192.168.250.33 -p 10000")
    

    Without doing so, the script will die raising a DNSException (host =9, whatever =), which makes we wonder why Ice is attempting anything like a dns lookup when listening to all IP addresses?!?

    Maybe you can verify, if that simple example works in your environment and give more information about what you're doing to provoke hangs.
  • Additional information:

    /var/db/ports/php5/options:
    _OPTIONS_READ=php5-5.2.8
    WITH_CLI=true
    WITH_CGI=true
    WITH_APACHE=true
    WITHOUT_DEBUG=true
    WITHOUT_SUHOSIN=true
    WITHOUT_MULTIBYTE=true
    WITH_IPV6=true
    WITHOUT_MAILHEAD=true
    WITHOUT_REDIRECT=true
    WITHOUT_DISCARD=true
    WITH_FASTCGI=true
    WITH_PATHINFO=true
    

    /usr/local/etc/php.ini (default plus the followinf two lines):
    ice.options="--Ice.Trace.Network=1 --Ice.Warn.Connections=1"
    ice.slice="-I/tmp /tmp/Hello.ice"
    
  • I think PHP cli is in fact crashing because it's not linked properly, since preloading libthread seems to fix the problem:
    [root@devjail /tmp]# export LD_PRELOAD=/usr/lib/libthr.so
    [root@devjail /tmp]# php
    <?php
    Ice_loadProfile();
    $base = $ICE->stringToProxy("hello:default -h 192.168.250.33 -p 10000");
    $hello = $base->ice_checkedCast("Demo::Hello");
    $hello->sayHello();
    ?>
    ^D
    [ 04/14/09 10:57:19.851 Network: tcp connection established
      local address = 192.168.250.34:32192
      remote address = 192.168.250.33:10000 ]
    [ 04/14/09 10:57:19.908 Network: closing tcp connection
      local address = 192.168.250.34:32192
      remote address = 192.168.250.33:10000 ]
    [root@devjail /tmp]# 
    
  • grembo wrote: »
    I think PHP cli is in fact crashing because it's not linked properly, since preloading libthread seems to fix the problem:

    I can confirm this on FreeBSD 7.1/i386. If I compile the port, stop it right after configure is finished, then go into the work directory and edit the Makefile, adding -lpthread to both the cli and the cgi build lines, php5-ice works on i386 (I'm using fastcgi, not mod_php, so I never had it working on i386). I more or less have everything I want from Mumble/Ice on i386 - thank you very much for your help.

    If I'm not mistaken, umtxn is a thread-related lock - so perhaps the solution for amd64 is somewhat simple too? I'm not really very familiar with threads at all. :(

    I reinstalled my production box to i386 to address some issues with the software I'm connecting Ice to - so unfortunately I can't really hunt for a solution for amd64 on that. My other amd64 environment is an amd64 installation inside Xen that I'm going to be getting rid of pretty quick, but I will try whatever ideas you have while I have it.
  • Maybe you can send me as many details about your setup as possible, since I might have a chance to play with new servers next week. E.g.:
    - Which version of Apache are you using, is it forking our multithreaded (MTM). I read about mod_php5 on FreeBSD with apache 2.2 MTM having the issues you described (even without ice)
    - Do you use mod_php on the amd64 box, or is this fastcgi as well?
    - Does the problem happen on every call or just over time
    - Is it possible to simulate the problem with an example as simple as Hello.ice?
    - php.ini
    etc.pp.

    I would like to setup an identical system here to see if I can reproduce the problem.

    Note that I'm also in touch with the maintainer of the php-ice and py-ice ports, he just sent me port sceletons for testing, I expect the ports tree to be updated to 3.3.1 within the next few days.
  • Oh and please send me also all information about how php and ice have been built. Is there any chance you're using openssl from ports instead of the one shipped with FreeBSD? Could you also do a procstat -k $pid of the hanging process?
  • grembo wrote: »
    Maybe you can send me as many details about your setup as possible, since I might have a chance to play with new servers next week.

    Sure, I'll try. I'm up to my arse in this server migration at the moment but I'll try help you out since you helped me. :D

    Wow, I seem to have been all confused, so I just re-ran all my tests. Apache 2.2 mpm-prefork, mod_php5 seems to work as expected. The CLI version will hang the client in umtxn. I have tried Ice and php5-ice from packages (3.3.0 and 3.3.0 both I believe) as well as Ice-3.3.1 from ports and php5-ice 3.3.0 from ports, none of those seems to help it. I had erroneously thought it happened inside Apache as well, but on my new machine I was using mpm-worker and fcgi, so that's why (mod_php5 seems to work fine on amd64).
    Note that I'm also in touch with the maintainer of the php-ice and py-ice ports, he just sent me port sceletons for testing, I expect the ports tree to be updated to 3.3.1 within the next few days.

    It'd be nice if that's all it will take to fix it.

    Oh and here:

    31626 fwaggle 1 99 0 79564K 14436K umtxn 2 0:01 0.00% php

    xatu# procstat -k 31626
    PID TID COMM TDNAME KSTACK
    31626 100231 php initial thread tdq_cpu

    I hope I've included everything you asked for, I'm running on empty a bit at the moment. :(
  • That seems to make sense. I just tried installing 3.3.1, both ports appear to work okay, but I don't think they'll fix any of your issues.

    Good to hear that the apache module works as expected -phew!

    Did you try to set LD_PRELOAD on amd64 like you did on i386 before using the CLI version - maybe it is basically the same problem (it seems like it's hanging in thread creation) but happening earlier because of the different implementation in the 64bit kernel?
  • grembo wrote: »
    Did you try to set LD_PRELOAD on amd64 like you did on i386 before using the CLI version - maybe it is basically the same problem (it seems like it's hanging in thread creation) but happening earlier because of the different implementation in the 64bit kernel?

    I didn't, and now that I think about it, I'm ashamed. It does seem to fix it. Perhaps you can pass that information on to the port maintainer? Not sure if he can do anything about it, considering the "fix" is to link cli/cgi versions of php with -lpthread. :\

    I guess that's somewhat good news though - it is odd however that a php client will connect to a Linux Ice endpoint without issue, isn't it?

    Cancel that, I must have been imagining that too.. It doesn't even work to Linux servers.

    I guess this is what I get for working on 6 different things at once.
  • Don't work too hard, live's short :)

    Anyway, I passed the info on to the maintainer, I'll let you know when I get feedback. IMHO the php5 port needs a configuration switch for passing -pthread and the php ice port needs either something to verify that or at least have a pkg-message warning about it. Stay tuned...
  • Update on this issue

    This problem still persists (8.1-RELEASE amd64), it seems like the php5-Ice port is orphaned at this point. I might consider taking it over, for the time being the advice given in this thread is still valid.