Home Help Center

error: filedescriptor 1257 not registered with `Ice.ThreadPool.Server'

kentkent Member quhaoOrganization: OPIProject: Testing
Hi,
I have an Im Front Server that is to accept and manage client's connections, and communicates with Main Server and Session Server to maintain online stats, etc. The Im Front Server has 2 Ice::Communicator, one is to response the Main Server's request and the other is to request needed information from the Third Server. When Im Front Server accepts more than 1000 clients(we have reset the connection limit to 60,000), it gets the following error (error: filedescriptor 1257 not registered with `Ice.ThreadPool.Server'), in which, the fd (1257) could ranges from 1025 to any other fd bigger (for example 1257). What's the problem?

Comments

  • matthewmatthew NL, CanadaMember Matthew NewhookOrganization: ZeroC, Inc.Project: Internet Communications Engine ✭✭✭
    What operating system and version of Ice are you using?
  • kentkent Member quhaoOrganization: OPIProject: Testing
    OS:Red Hat Enterprise Linux AS release 4 (Nahant)
    Ice: 3.0.1

    I do a little test as following:

    Server:
    single thread
    use epoll to dispatch events from client and provide a simple Ice service.

    Client:
    multi thread for each socket connection and Ice service calling
    connect to Server and send a message every 1 second, call Ice service every 1 second.

    if Client make more than 1024 connection first and then call Ice service, crash after soon,
    if Client call Ice service first and then make more than 1024 connection, it works,
    if Client make less than 1024, it works in spite of startup order

    ;)
  • matthewmatthew NL, CanadaMember Matthew NewhookOrganization: ZeroC, Inc.Project: Internet Communications Engine ✭✭✭
    Sorry, I don't understand your setup. What exactly are you doing with epoll, and what does that have to do with Ice? Ice internally uses select, not epoll. There is no hard limit at 1024 -- if there was you would reach this before anyway due to some of the other fd's that your process opens (stdin, stdout, etc). How are you forcing so many connections to be established? What exactly are you doing? Posting an example that duplicates this problem would help.
  • kentkent Member quhaoOrganization: OPIProject: Testing
    The server is a Instance Messenger Front Server, so it has to accept many connections, receive messages from each connection and post it to another Ice Server. The Front Server also include a Ice Adapter to response other server's request(message routing, etc).

    We find that the problem occurs only when the max fd number is larger than 1024(or other) of Ice's connection. we fcntl accepted connection's fd larger than 1000, and reserve fd less than 1000, there's no problem even accepts more than 2000 connections.

    so, it is like that fd number is the key.
  • kentkent Member quhaoOrganization: OPIProject: Testing
    #include <sys/types.h>
    #include <sys/stat.h>
    #include <sys/queue.h>
    #include <unistd.h>
    #include <sys/time.h>
    #include <fcntl.h>
    #include <stdlib.h>
    #include <stdio.h>
    #include <string.h>
    #include <errno.h>
    #include <sys/types.h>
    #include <sys/socket.h>
    #include <netinet/in.h>
    #include <netdb.h>
    #include <unistd.h>
    #include <string>
    #include <arpa/inet.h>
    #include <errno.h>
    #include <netinet/in.h>
    #include <limits.h>
    #include <fcntl.h>
    #include <event.h>
    #include <sys/time.h>
    #include <sys/resource.h>
    #include <unistd.h>

    #include <iostream>
    #include <Hello.h>
    #include "Channel.h"

    using namespace std;
    using namespace MyUtil;

    class HelloI : virtual public im::Hello
    {
    public:
    virtual void say(const string& message, const Ice::Current&) {
    //cout << message << endl;
    }
    };

    void
    socket_read(int fd, short event, void *arg)
    {
    char buf[255];
    int len;
    struct event *ev = (struct event*)arg;

    //
    // fprintf(stderr, "socket_read called with fd: %d, event: %d, arg: %p\n",
    // fd, event, arg);
    len = read(fd, buf, sizeof(buf) - 1);

    if (len == -1) {
    delete ev;
    perror("read");
    close(fd);
    return;
    } else if (len == 0) {
    fprintf(stderr, "Connection closed\n");
    delete ev;
    close(fd);
    return;
    }

    buf[len] = '\0';
    //fprintf(stdout, "Read: %s\n", buf);

    /* Reschedule this event */
    event_add(ev, NULL);
    }

    void
    socket_accept(int socket, short event, void* arg)
    {
    struct event *ev = (struct event*)arg;
    /* Reschedule this event */
    event_add(ev, NULL);

    int fd = accept(socket, NULL, NULL);
    if (fd < 0) {
    cout << "accept: " << strerror(errno) << endl;
    return;
    }
    struct event* evsocket = new struct event;
    event_set(evsocket, fd, EV_READ, socket_read, evsocket);

    /* Add it to the active events, without a timeout */
    event_add(evsocket, NULL);
    }

    int
    setlimit(uint32_t filemax)
    {
    int ret;
    struct rlimit rl;

    ret = getrlimit(RLIMIT_NOFILE, &rl);
    if (ret != 0) {
    fprintf(stderr, "Unable to read open file limit: (%d, %s)\n",
    errno, strerror(errno));
    return 1;
    }

    fprintf(stderr, "Limit was %d (max %d), ",
    (int) rl.rlim_cur, (int) rl.rlim_max);

    if (rl.rlim_cur >= filemax) {
    fprintf(stderr, "unchanged\n");
    return 0;
    } else {
    fprintf(stderr, "setting to %d\n", filemax);
    }

    rl.rlim_cur = rl.rlim_max = filemax;
    ret = setrlimit(RLIMIT_NOFILE, &rl);
    if (ret != 0) {
    fprintf(stderr, "Unable to set open file limit: (%d, %s)\n",
    errno, strerror(errno));
    return 1;
    }

    ret = getrlimit(RLIMIT_NOFILE, &rl);
    if (ret != 0) {
    fprintf(stderr, "Unable to read _NEW_ open file limit: (%d, %s)\n",
    errno, strerror(errno));
    return 1;
    }

    if (rl.rlim_cur < filemax) {
    fprintf(stderr, "Failed to set open file limit: "
    "limit is %d, expected %d\n", (int) rl.rlim_cur, filemax);
    return 1;
    }

    return 0;
    }

    int
    main (int argc, char **argv)
    {
    struct event* evsocket = new struct event;
    int socket;

    setlimit(60000);

    ChannelPtr channel = new Channel(argc, argv);
    Ice::ObjectAdapterPtr adapter = channel->getCommunicator()
    ->createObjectAdapterWithEndpoints("Server", ":tcp -p 8889");
    adapter->add(new HelloI, Ice::stringToIdentity("Hello"));
    adapter->activate();

    cout << "Adapter initialized" << endl;
    /* step1: create a socket */
    if ((socket = ::socket(AF_INET, SOCK_STREAM, 0)) < 0) {
    cout << "socket: " << strerror(errno) << endl;
    cout.flush();
    return -1;
    }
    cout << "Socket created" << endl;
    int value = 1;
    socklen_t size = sizeof(value);
    if (setsockopt(socket, SOL_SOCKET, SO_REUSEADDR, &value, size) < 0) {
    cout << "setsockopt: " << strerror(errno) << endl;
    return -1;
    }

    struct sockaddr_in addr;
    addr.sin_addr.s_addr = INADDR_ANY;
    addr.sin_family = AF_INET;
    addr.sin_port = htons(8888);
    if (bind(socket, (struct sockaddr*)&addr, sizeof(struct sockaddr)) < 0) {
    cout << "bind: " << strerror(errno) << endl;
    return -1;
    }

    if (listen (socket, 2000) < 0) {
    cout << "listen: " << strerror(errno) << endl;
    return -1;
    }

    /* Initalize the event library */
    event_init();

    event_set(evsocket, socket, EV_READ, socket_accept, evsocket);

    /* Add it to the active events, without a timeout */
    event_add(evsocket, NULL);

    event_dispatch();

    return (0);
    }
  • matthewmatthew NL, CanadaMember Matthew NewhookOrganization: ZeroC, Inc.Project: Internet Communications Engine ✭✭✭
    Without detailed knowledge of your application its basically impossible to determine what exactly is happening. This is certainly nothing that is Ice specific. If you require more help with this problem please contact [email protected] to enquire about our consulting services.
  • kentkent Member quhaoOrganization: OPIProject: Testing
    The above is Server.cpp, it uses libevent to dispatch events.
    If i change socket_accept like this:

    void
    socket_accept(int socket, short event, void* arg)
    {
    struct event *ev = (struct event*)arg;
    /* Reschedule this event */
    event_add(ev, NULL);

    int fd = accept(socket, NULL, NULL);
    if (fd < 0) {
    cout << "accept: " << strerror(errno) << endl;
    return;
    }
    int newfd = fcntl(fd, F_DUPFD, 1000);
    if (newfd <= 0) {
    cout << "cannot move fd: " << fd << endl;
    continue;
    } else {
    close(fd);
    fd = newfd;
    }

    struct event* evsocket = new struct event;
    event_set(evsocket, fd, EV_READ, socket_read, evsocket);

    /* Add it to the active events, without a timeout */
    event_add(evsocket, NULL);
    }


    Force the new incoming connection's fd > 1000 to reserve the lower fd for Ice, there's no problem
  • matthewmatthew NL, CanadaMember Matthew NewhookOrganization: ZeroC, Inc.Project: Internet Communications Engine ✭✭✭
    You may be running into a kernel limitation on the number of file descriptors that select will support. You should look at your kernel configuration to determine what the hard limit is. At any rate, if you opt to use our consulting services we can look into this in more detail.
  • kentkent Member quhaoOrganization: OPIProject: Testing
    I think that is not the reason, because Ice only establishes 1 connection, and select a few file descriptors in Ice.ThreadPool. The other lots of connection established by epoll have no limits, and we have set max connection limit to 60000. The problem occurs only if Ice's connection's fd number is larger than 1000(or other number). Like the message I have post above, we have tested it is the key.

    I suggest that you will make a same test.
  • kentkent Member quhaoOrganization: OPIProject: Testing
    I look into ice/ThreadPool.cpp, the following codes are the problem source that puzzle me, please give some suggestion.

    //
    // Round robin for the filedescriptors.
    //
    if(_lastFd < _minFd - 1 || _lastFd == INVALID_SOCKET)
    {
    _lastFd = _minFd - 1;
    }

    int loops = 0;
    do
    {
    if(++_lastFd > _maxFd)
    {
    ++loops;
    _lastFd = _minFd;
    }
    }
    while(!FD_ISSET(_lastFd, &fdSet) && loops <= 1);

    if(loops > 1)
    {
    Error out(_instance->logger());
    out << "select() in `" << _prefix << "' returned " << ret
    << " but no filedescriptor is readable";
    continue;
    }
  • marcmarc FloridaAdministrators, ZeroC Staff Marc LaukienOrganization: ZeroC, Inc.Project: The Internet Communications Engine ZeroC Staff
    I'm sorry, but explaining Ice internals is beyond the free support we can provide here in these forums, and so is helping to debug applications that do their own non-Ice network communications. As Matthew said, we can provide you with consulting services to help you solve your problem. Please contact us at [email protected] if you are interested.
Sign In or Register to comment.