Bug in ThreadPool in Ice 3.2.1

A java servant in an IceBox instance in our IceGrid deployment encountered a java.lang.OutOfMemoryError. Thereafter, our services were not available because an assert in IceInternal.ThreadPool kept popping. Here is the assert:

Exception in thread "HaloCentralGrid-Services-01-DatabaseServices-Ice.ThreadPool.Server-106" java.lang.AssertionError
at IceInternal.ThreadPool.run(ThreadPool.java:820)
at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:1242)

It appears that this assertion pops when the ThreadPool believes that the number of running Threads exceeds the total number of threads hosted by the ThreadPool instance. I believe that the root-cause is in the run method of the EventHandlerThread class: it appears that when it runs a servant invocation, it catches only java.lang.Exception and Ice.LocalException - it does NOT catch java.lang.Error, which is the superclass of a java.lang.OutOfMemoryError. It needs a catch block for java.lang.Error, or any java.lang.Error subclass encountered in the servant invocation will render the entire ThreadPool useless thereafter.

Here is the trace for the OutOfMemoryError:

Exception in thread "HaloCentralGrid-Services-01-DatabaseServices-Ice.ThreadPool.Server-82" java.lang.OutOfMemoryError: Java heap space
at com.microsoft.sqlserver.jdbc.DTV.setFromTDS(Unknown Source)
at com.microsoft.sqlserver.jdbc.Column.setFromTDS(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.indexRow(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.countRowsInFetchBuffer(Unknown Source)
at com.microsoft.sqlserver.jdbc.ClientFetchBuffer.getRowCount(Unknown Source)
at com.microsoft.sqlserver.jdbc.ClientFetchBuffer.relative(Unknown Source)
at com.microsoft.sqlserver.jdbc.PositionedCursor.positionedRelative(Unknown Source)
at com.microsoft.sqlserver.jdbc.PositionedCursor.next(Unknown Source)
at com.microsoft.sqlserver.jdbc.SQLServerResultSet.next(Unknown Source)
at com.mchange.v2.c3p0.impl.NewProxyResultSet.next(NewProxyResultSet.java:2859)
at org.hibernate.loader.Loader.doQuery(Loader.java:685)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:224)
at org.hibernate.loader.Loader.loadCollection(Loader.java:1919)
at org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:36)
at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:520)
at org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:60)
at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1676)
at org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:344)
at org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:86)
at org.hibernate.collection.PersistentSet.iterator(PersistentSet.java:138)
at com.hp.halo.databaseservice.dtmUtils.DtmUtils.DTMMakeAudioListFromDbAudioListCapability(DtmUtils.java:564)
at com.hp.halo.databaseservice.dtmUtils.DtmUtils.DTMMakeNodeBaseFromDbNode(DtmUtils.java:393)
at com.hp.halo.databaseservice.node.Node.getAllNodes(Node.java:1689)
at com.hp.halo.databaseservice.DatabaseServicesImpl.getAllNodes(DatabaseServicesImpl.java:1366)
at com.hp.halo.HaloICE._DatabaseServicesDisp.___getAllNodes(Unknown Source)
at com.hp.halo.HaloICE._DatabaseServicesDisp.__dispatch(Unknown Source)
at IceInternal.Incoming.invoke(Incoming.java:147)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2249)
at Ice.ConnectionI.message(ConnectionI.java:1362)
at IceInternal.ThreadPool.run(ThreadPool.java:782)
at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:1242)


Dirk