socket resource exhaust

fengshengheli · October 2019

Hello,
I write a client and a server with Ice 3.6.1. The slice is very sample and few interfaces.
But, the client will send a large number of calls to server. A few days later, the sock of server
exhaust. Is it a bug of ice? What can i do? Please help me. Thanks.

file850 18349 root 12u sock 0,8 0t0 940597383 can't identify protocol
file850 18349 root 13u sock 0,8 0t0 940593371 can't identify protocol
file850 18349 root 14u sock 0,8 0t0 940820216 can't identify protocol
file850 18349 root 15u sock 0,8 0t0 940663999 can't identify protocol
file850 18349 root 16u sock 0,8 0t0 940871873 can't identify protocol
file850 18349 root 17u sock 0,8 0t0 940729868 can't identify protocol
file850 18349 root 18u sock 0,8 0t0 940594653 can't identify protocol
file850 18349 root 19u sock 0,8 0t0 940878789 can't identify protocol

the client props:
//设置Ice属性
//Get the initialized propert set
Ice::PropertiesPtr props = Ice::createProperties(argc, argv);
//Set the size of the server's thread pool
props->setProperty("Ice.ThreadPool.Client.Size", "5");
props->setProperty("Ice.ThreadPool.Client.SizeMax", "1000");
props->setProperty("Ice.ThreadPool.Server.Size", "5");
props->setProperty("Ice.ThreadPool.Server.SizeMax", "1000");
props->setProperty("Ice.ThreadPool.Server.ThreadIdleTime", "10");
props->setProperty("Ice.MessageSizeMax", "2048000");
props->setProperty("Ice.Override.ConnectTimeout", "10000");// 连接建立的超时时间是10秒
props->setProperty("Ice.Override.Timeout", "30000"); // 连接的超时时间ms，含建立连接、读、写、关闭等操作
props->setProperty("Ice.RetryIntervals", "-1"); // 禁止重试
props->setProperty("Ice.Default.EncodingVersion", "1.0");
//Initialize a communicator with these properties
Ice::InitializationData id;
id.properties = props;

the server props:
//设置Ice属性
//Get the initialized propert set
Ice::PropertiesPtr props = Ice::createProperties(argc, argv);
//Set the size of the server's thread pool
props->setProperty("Ice.ThreadPool.Client.Size", "5");
props->setProperty("Ice.ThreadPool.Client.SizeMax", "1000");
props->setProperty("Ice.ThreadPool.Server.Size", "5");
props->setProperty("Ice.ThreadPool.Server.SizeMax", "1000");
props->setProperty("Ice.ThreadPool.Server.ThreadIdleTime", "10");
props->setProperty("Ice.MessageSizeMax", "2048000");
props->setProperty("Ice.Override.ConnectTimeout", "10000"); // 连接建立的超时时间是10秒
props->setProperty("Ice.Override.Timeout", "30000"); // 连接超时时间是30秒，含建立连接、读、写、关闭连接等操作
props->setProperty("Ice.RetryIntervals", "-1"); // 禁止重试
//Initialize a communicator with these properties
Ice::InitializationData id;
id.properties = props;

benoit · October 2019

Hi,

From your description, it sounds like there's a file descriptor leak. It's impossible to say however what could be the cause without additional information.

Could you provide a little more information on your application? For instance:

Is your application composed of a single client and server?
Is this client or are these clients connecting/disconnecting often from the server over the several days before the failure?
Which operating system do you use?
If you enable network tracing with Ice.Trace.Network=1 on the server, can you verify that network connections for clients are correctly closed?

Also, could you upgrade to the latest Ice 3.7.3 version or at least Ice 3.6.5?

Cheers,
Benoit

fengshengheli · October 2019

The slice：
module ied
{
struct GETFILE
{
string remoteFile;
string localFile;
};
sequence GetFileSeq;

    struct FILEATTR
    {
        string fileName;
        string filePath;
        int    fileSize;
        long   fileLastModified;
    };
    sequence<FILEATTR> FileAttrSeq;

    interface iedFile
    {
        bool connected(string ip, int port);
        bool readFile(string ip, int port, string remotefile, string localfile);
        bool readFiles(string ip, int port, GetFileSeq files, out string failedfile);
        bool readComtradeDirectory(string ip, int port, string dirname, string caname, out FileAttrSeq seq);
        bool readDirectory(string ip, int port, string dirname, string caname, out FileAttrSeq seq);
    };
};

The ice server app run function:
int CFile850App::run(int argc, char* argv[])
{
//This function creates an IceUtil::CtrlCHandler that shuts down the
//communicator when one of the monitored signals is raised.
shutdownOnInterrupt();

    m_communicator = communicator();
    m_adapterIedFile = m_communicator->createObjectAdapterWithEndpoints((string)("IedFile"), (string)("default -p 10012"));
    CIedFileIPtr iedfileiptr = new CIedFileI();
    m_adapterIedFile->add(iedfileiptr, communicator()->stringToIdentity("IedFile"));
    m_adapterIedFile->activate();

    try
    {
        m_communicator->waitForShutdown();
    }
    catch (const Ice::Exception& e)
    {
        cerr << e << endl;
    }

    if (communicator())
    {
        try
        {
            communicator()->destroy();
        }
        catch (const Ice::Exception& e)
        {
            cerr << e << endl;
        }
    }

    return EXIT_SUCCESS;
}

The ice server side interface implementation:
bool CIedFileI::connected(const ::std::string& ip, ::Ice::Int port, const ::Ice::Current& cur /* = Ice::Current() */)
{
SdlIec61850Client client(ip, port);
bool ret = client.initialize();
return ret;
}

bool CIedFileI::readFile(const ::std::string& ip, ::Ice::Int port, const ::std::string& remoteFile, const ::std::string& localFile, const ::Ice::Current& cur /* = Ice::Current() */)
{
    SdlIec61850Client client(ip, port);
    bool ret = client.initialize();
    if (!ret)
        return ret;

    ret = client.get_file(remoteFile, localFile);
    return ret;
}

bool CIedFileI::readFiles(const ::std::string& ip, ::Ice::Int port, const ::ied::GetFileSeq& files, ::std::string& failedfile, const ::Ice::Current& cur /* = Ice::Current() */)
{
    SdlIec61850Client client(ip, port);
    bool ret = client.initialize();
    if (!ret)
        return false;

    ::ied::GetFileSeq::const_iterator iter;
    for (iter = files.begin(); iter != files.end(); ++iter)
    {
        ret = client.get_file(iter->remoteFile, iter->localFile);
        if (!ret)
        {
            failedfile = iter->remoteFile;
            return false;
        }
    }
    return true;
}

SdlIec61850Client is a client for getting data from IEC61850 server. It is written using libiec61850 library. I use libiec61850 in other programs. It works ok. So the socket leak can not caused by libiec61850. In my program file850, only ice and libiec61850 use socket, I didn't write any socket code.

The ice client side code:
iedFilePrx CShandong850I::get_iedfile_proxy()
{
try
{
string strUri = "IedFile:default -h 127.0.0.1 -p 10012";
Ice::CommunicatorPtr communicatorPtr = Ice::Application::communicator();
Ice::ObjectPrx objPrx = communicatorPtr->stringToProxy(strUri);
iedFilePrx filePrx = iedFilePrx::checkedCast(objPrx);
return filePrx;
}
catch (Ice::LocalException& e)
{
if (strstr(e.what(), "ConnectTimeoutException") != NULL)
{
kill_file850_program();
}
return NULL;
}
catch (Ice::Exception& e)
{
m_log.WriteALog(e.what());
return NULL;
}
catch (...)
{
m_log.WriteALog("readDirectory: unknown exception");
return NULL;
}

    return NULL;
}

bool CShandong850I::is_connected()
{
    try
    {
        iedFilePrx filePrx= get_iedfile_proxy();
        if(filePrx == NULL)
            return false;

        string ip = (m_ied.valid_ip_no == 1)? m_ied.ip1 : m_ied.ip2;
        int32_t port = (m_ied.valid_ip_no == 1)? m_ied.port1 : m_ied.port2;

        iedFilePrx timeoutFilePrx = filePrx->ice_invocationTimeout(600000);         
        bool ret = timeoutFilePrx->connected(ip, port);
        return ret;
    }
    catch (Ice::InvocationTimeoutException& e)
    {
        return false;
    }
    catch(Ice::LocalException& e)
    {
        return false;
    }
    catch(Ice::Exception& e)
    {
        return false;
    }
    catch(...)
    {
        return false;
    }
    return false;
}

bool CShandong850I::get_comtrade_files_async(COMTRADE_FILE_ATTRIBUTE* comtrade_file_attribute, char* dst_file_name, string& failed_file, CLog* log/*=NULL*/)
{
    failed_file = "";
    string srcFileName;
    if (comtrade_file_attribute->file_name[0] == '/')
        srcFileName = comtrade_file_attribute->file_name;
    else
    {
        srcFileName = "/COMTRADE/";
        srcFileName += comtrade_file_attribute->file_name;
    }
    string dstFileName = dst_file_name;
    string srcDatFile = srcFileName + comtrade_file_attribute->dat_suffix;
    string srcCfgFile = srcFileName + comtrade_file_attribute->cfg_suffix;
    string srcHdrFile = srcFileName + comtrade_file_attribute->hdr_suffix;
    string srcDmfFile = srcFileName + comtrade_file_attribute->dmf_suffix;
    string dstDatFile = dstFileName + ".dat";
    string dstCfgFile = dstFileName + ".cfg";
    string dstHdrFile = dstFileName + ".hdr";
    string dstDmfFile = dstFileName + ".dmf";

    GetFileSeq files;
    ::ied::GETFILE get_file;
    get_file.remoteFile = srcCfgFile;
    get_file.localFile = dstCfgFile;
    files.push_back(get_file);
    get_file.remoteFile = srcDatFile;
    get_file.localFile = dstDatFile;
    files.push_back(get_file);
    if (strlen(comtrade_file_attribute->hdr_suffix) > 0)
    {
        get_file.remoteFile = srcHdrFile;
        get_file.localFile = dstHdrFile;
        files.push_back(get_file);
    }
    if (strlen(comtrade_file_attribute->dmf_suffix) > 0)
    {
        get_file.remoteFile = srcDmfFile;
        get_file.localFile = dstDmfFile;
        files.push_back(get_file);
    }

    int32_t ice_override_timeout = CEnvironment::get_env()->get_ice_override_timeout();
    int32_t loop_times = ice_override_timeout / 50 + 1;
    iedFilePrx filePrx = get_iedfile_proxy();
    if (filePrx == NULL)
    {
        failed_file = "connect to file850 failed";
        return false;
    }

    string ip = (m_ied.valid_ip_no == 1) ? m_ied.ip1 : m_ied.ip2;
    int32_t port = (m_ied.valid_ip_no == 1) ? m_ied.port1 : m_ied.port2;

    try
    {
        iedFilePrx timeoutFilePrx = filePrx->ice_invocationTimeout(ice_override_timeout);

        Ice::AsyncResultPtr resultPtr = timeoutFilePrx->begin_readFiles(ip, port, files);
        resultPtr->waitForSent();

        for (int i = 0; i < loop_times; ++i)
        {
            if (!resultPtr->isCompleted())
            {
                MySleep(50000); // sleep 50 ms
                continue;
            }

            bool ret = timeoutFilePrx->end_readFiles(failed_file, resultPtr);
            if (!ret)
            {
                MyDeleteFile(dstDatFile.c_str());
                MyDeleteFile(dstCfgFile.c_str());
                MyDeleteFile(dstHdrFile.c_str());
                MyDeleteFile(dstDmfFile.c_str());
            }
            return ret;
        }

        //调用ice接口超时
        MyDeleteFile(dstDatFile.c_str());
        MyDeleteFile(dstCfgFile.c_str());
        MyDeleteFile(dstHdrFile.c_str());
        MyDeleteFile(dstDmfFile.c_str());
    }
    catch (Ice::InvocationTimeoutException& e)
    {
        return false;
    }
    catch (Ice::LocalException& e)
    {
        return false;
    }
    catch (Ice::Exception& e)
    {
        return false;
    }
    catch (...)
    {
        return false;
    }
    return false;
}

The ice client side program, having more than 100 threads. These threads mutual independence. All threads will send ice call to ice server from code above.

benoit · October 2019

Hi,

Could you provide the information I asked in my previous post?

Also, did you try to run your server under a memory leak checker (e.g.: valgrind if using Linux) to see if it pointed out any obvious leaks?

We check each Ice release for memory leaks and there are no known leaks in Ice 3.6 or 3.7.

One thing you could also try to is to comment out the code that downloads the file with the SdlIec61850Client object and let run the client / server for a while while monitoring the FD usage.

Cheers,
Benoit

fengshengheli · November 2019

Hi,
Thanks very mush.
I write a test server and client. It works well. It perhaps libIEC61850's problem.

socket resource exhaust

Answers