System.StackOverflowException when using ZeroC Ice 3.4.2 on Mono 2.10.x

miltondp

I have reported this bug in the Mono bug tracking system here:

Bug 2449 – System.StackOverflowException when using ZeroC Ice 3.4.2 on Mono 2.10.x

However, one of the developers suggested to file a bug in these forums as well, as you likely have more information to resolve it.

Take a look please at the bug link for the complete bug report. I've also attached a file with a trace of the exception. The problem seems to reside in this method:

IceInternal.AssemblyUtil.loadReferencedAssemblies (System.Reflection.Assembly)

mes

Hi Milton,

Thanks for reporting this. If it is a Mono bug, we'll try to find a workaround that we can include in the next release.

Regards,
Mark

mes

Milton,

I've been doing some testing with Mono 2.10.8 on SLES 11 but haven't been able to reproduce the problem you described. Of course, the problem may be dependent on your application structure, such as how many assemblies are used in your server, where the Slice-generated code is located, and so on.

Would you be able to provide a small test case that we can use to reproduce this problem?

If not, can you describe in detail the assembly structure used by your server?

Thanks,
Mark

miltondp

Hi Mark, thanks for your reply.

I just installed Mono 2.10.8 on Fedora and I was able to reproduce the problem. I'm planning to upload the source code of my app soon so you can play with it. Will let you know when it's ready.

Thank you!
Milton.

mathgl

Hi

Have you prepared your codes yet? If so, i would like to run on my machine and try to find out what happens.

Thanks..

miltondp

Sorry for the long delay guys!

You can download the source code from this URL. These are the steps to reproduce the problem:

1. Navigate to the src/Stratos folder.
2. Run the "make" command (it uses xbuild).
3. Configure Stratos Server to run against SQLite (so you don't have to have a database server to test it): move to src/Stratos/StratosServer/build and run: cp ActiveRecord.config.sqlite.xml ActiveRecord.config.xml
4. Start the Stratos Server: go to src/Stratos/StratosServer/build and run the command: mono StratosServer.Startup.exe. Now the server is listening for incoming connections.
5. Start the Stratos Client (Gtk app): go to src/Stratos/StratosClient/build and run the command: mono StratosClient.Startup.exe
6. When starting the Client, you'll see it freezes at startup. Take a look at the Server console, you'll see messages like this:

   Stack overflow in unmanaged: IP: 0x82266c9, fault addr: 0xb3f07fbc
   Stack overflow in unmanaged: IP: 0x82266c9, fault addr: 0xb3f07f0c
   Stack overflow in unmanaged: IP: 0x82266c9, fault addr: 0xb3f07eec
   

Let me know if you have any problem. And sorry again for the long delay.

Thank you.

miltondp

Is there any news on this bug?

miltondp

It seems that the bug is only reproducible when you use the 4.0 profile when compiling with Mono.

miltondp

Is it possible to test the code I published before and verify if this is a problem with Ice or with Mono runtime? If the latter, I'd like to update the mono devs with that.

miltondp

Guys, sorry for the insistence, but I'd need to know if you are planning to see what's happening with this bug. I already published the code, in the link above, along with the instructions to compile and run it.

Maybe for you this is low priority, what I understand, but I only need to know if this is going to be tested or if it was marked as low priority. This issue will be blocking me soon from releasing my application, and I need to evaluate other alternatives if necessary.

I'm available to provide more information if needed, or to debug Ice internally, but I need some clue.

Thank you.

mes

Hi,

I'll admit this hasn't been a high priority for us, for several reasons:

• You're using an unsupported version of Mono, on an unsupported platform.
• The example you provided is large and complex.
• We already tried to reproduce the problem using Mono 2.10.x on a supported platform, but couldn't duplicate it.

The quickest way to get this resolved is for you to provide a small example that we can use to reproduce the problem. Please also tell us the exact versions of Mono and operating system that you are using.

Regards,
Mark

miltondp

Thanks for the quick reply Mark,

I'll try to trace down the problem and make a small test for you.

One more thing: in my first post in this thread, I put a link to the Mono's Bugzilla bug I reported. In that bug there is a log file attached, where you can see why the Stackoverflow exception is thrown. Maybe you can take a quick look at it and give me a clue about where could be the problem. This is the stack trace:

(lots of calls to IceInternal.AssemblyUtil.loadReferencedAssemblies here)

  at IceInternal.AssemblyUtil.loadReferencedAssemblies (System.Reflection.Assembly) <0x000eb>
  at IceInternal.AssemblyUtil.loadReferencedAssemblies (System.Reflection.Assembly) <0x000eb>
  at IceInternal.AssemblyUtil.loadReferencedAssemblies (System.Reflection.Assembly) <0x000eb>
  at IceInternal.AssemblyUtil.loadReferencedAssemblies (System.Reflection.Assembly) <0x000eb>
  at IceInternal.AssemblyUtil.loadAssemblies () <0x000c7>
  at IceInternal.AssemblyUtil.findType (IceInternal.Instance,string) <0x00023>
  at IceInternal.BasicStream.loadObjectFactory (string) <0x00053>
  at IceInternal.BasicStream.readObject (IceInternal.IPatcher) <0x00317>
  at IceInternal.BasicStream.readPendingObjects () <0x0004d>
  at StratosServer.Controladores.ControladorBaseDisp_.Refrescar___ (StratosServer.Controladores.ControladorBase,IceInternal.Incoming,Ice.Current) <0x00088>
  at StratosServer.Controladores.ControladorFacturacionDisp_.dispatch__ (IceInternal.Incoming,Ice.Current) <0x001ab>
  at IceInternal.Incoming.invoke (IceInternal.ServantManager) <0x003dc>
  at Ice.ConnectionI.invokeAll (IceInternal.BasicStream,int,int,byte,IceInternal.ServantManager,Ice.ObjectAdapter) <0x000f7>
  at Ice.ConnectionI.dispatch (Ice.ConnectionI/StartCallback,System.Collections.Generic.Queue`1<Ice.ConnectionI/OutgoingMessage>,Ice.ConnectionI/MessageInfo) <0x0013f>
  at Ice.ConnectionI/<message>c__AnonStorey5.<>m__4 () <0x00177>
  at IceInternal.ThreadPool.run (IceInternal.ThreadPool/WorkerThread) <0x0080f>
  at IceInternal.ThreadPool/WorkerThread.Run () <0x00267>
  at System.Threading.Thread.StartUnsafe () <0x00057>
  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) <0xffffffff>
[0x5264b40:] EXCEPTION handling: System.StackOverflowException: (No exception message for: StackOverflowException)

"StratosServer.Startup.exe-Ice.ThreadPool.Server-0" tid=0x0x5264b40 this=0x0x32bf150 thread handle 0x437 state : not waiting owns ()
  at (wrapper managed-to-native) string.InternalSplit (string,char[],int,int) <0xffffffff>
  at string.Split (char[],int) <0x0008f>
  at string.Split (char[]) <0x0001b>
  at System.IO.Path.CanonicalizePath (string) <0x0008b>
  at System.IO.Path.InsecureGetFullPath (string) <0x001d7>
  at System.IO.Path.GetFullPath (string) <0x00013>
  at System.IO.FileStream..ctor (string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare,int,bool,System.IO.FileOptions) <0x0011b>
  at System.IO.FileStream..ctor (string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare) <0x00023>
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor (string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare) <0xffffffff>
  at Mono.Addins.Database.FileDatabase.FileLock (System.IO.FileAccess,int) <0x0012b>
  at Mono.Addins.Database.FileDatabase.LockRead () <0x00017>
  at Mono.Addins.Database.AddinDatabase.InternalCheck (string) <0x0002f>
  at Mono.Addins.Database.AddinDatabase.GetAddinForHostAssembly (string,string) <0x0002f>
  at Mono.Addins.AddinRegistry.GetAddinForHostAssembly (string) <0x0003f>
  at Mono.Addins.AddinEngine.CheckHostAssembly (System.Reflection.Assembly) <0x000ef>
  at Mono.Addins.AddinEngine.OnAssemblyLoaded (object,System.AssemblyLoadEventArgs) <0x0001f>
  at System.AppDomain.DoAssemblyLoad (System.Reflection.Assembly) <0x00043>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) <0xffffffff>
  at (wrapper managed-to-native) System.AppDomain.LoadAssembly (System.AppDomain,string,System.Security.Policy.Evidence,bool) <0xffffffff>
  at System.AppDomain.Load (System.Reflection.AssemblyName,System.Security.Policy.Evidence) <0x000cf>
  at (wrapper remoting-invoke-with-check) System.AppDomain.Load (System.Reflection.AssemblyName,System.Security.Policy.Evidence) <0xffffffff>
  at System.AppDomain.Load (System.Reflection.AssemblyName) <0x0001b>
  at (wrapper remoting-invoke-with-check) System.AppDomain.Load (System.Reflection.AssemblyName) <0xffffffff>
  at System.Reflection.Assembly.Load (System.Reflection.AssemblyName) <0x0001b>
  at IceInternal.AssemblyUtil.loadReferencedAssemblies (System.Reflection.Assembly) <0x000a3>
  at IceInternal.AssemblyUtil.loadReferencedAssemblies (System.Reflection.Assembly) <0x000eb>

(lots of calls to IceInternal.AssemblyUtil.loadReferencedAssemblies here)

  at IceInternal.AssemblyUtil.loadReferencedAssemblies (System.Reflection.Assembly) <0x000eb>
  at IceInternal.AssemblyUtil.loadReferencedAssemblies (System.Reflection.Assembly) <0x000eb>
  at IceInternal.AssemblyUtil.loadReferencedAssemblies (System.Reflection.Assembly) <0x000eb>
  at IceInternal.AssemblyUtil.loadReferencedAssemblies (System.Reflection.Assembly) <0x000eb>
  at IceInternal.AssemblyUtil.loadReferencedAssemblies (System.Reflection.Assembly) <0x000eb>
  at IceInternal.AssemblyUtil.loadReferencedAssemblies (System.Reflection.Assembly) <0x000eb>
  at IceInternal.AssemblyUtil.loadReferencedAssemblies (System.Reflection.Assembly) <0x000eb>
  at IceInternal.AssemblyUtil.loadAssemblies () <0x000c7>
  at IceInternal.AssemblyUtil.findType (IceInternal.Instance,string) <0x00023>
  at IceInternal.BasicStream.loadObjectFactory (string) <0x00053>
  at IceInternal.BasicStream.readObject (IceInternal.IPatcher) <0x00317>
  at IceInternal.BasicStream.readPendingObjects () <0x0004d>
  at StratosServer.Controladores.ControladorBaseDisp_.Refrescar___ (StratosServer.Controladores.ControladorBase,IceInternal.Incoming,Ice.Current) <0x00088>
  at StratosServer.Controladores.ControladorFacturacionDisp_.dispatch__ (IceInternal.Incoming,Ice.Current) <0x001ab>
  at IceInternal.Incoming.invoke (IceInternal.ServantManager) <0x003dc>
  at Ice.ConnectionI.invokeAll (IceInternal.BasicStream,int,int,byte,IceInternal.ServantManager,Ice.ObjectAdapter) <0x000f7>
  at Ice.ConnectionI.dispatch (Ice.ConnectionI/StartCallback,System.Collections.Generic.Queue`1<Ice.ConnectionI/OutgoingMessage>,Ice.ConnectionI/MessageInfo) <0x0013f>
  at Ice.ConnectionI/<message>c__AnonStorey5.<>m__4 () <0x00177>
  at IceInternal.ThreadPool.run (IceInternal.ThreadPool/WorkerThread) <0x0080f>
  at IceInternal.ThreadPool/WorkerThread.Run () <0x00267>
  at System.Threading.Thread.StartUnsafe () <0x00057>
  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) <0xffffffff>

mes

Ok, let's take a look at the loadReferencedAssemblies method, which ensures that all assemblies are (recursively) pre-loaded. The method uses a dictionary to keep track of the assemblies it has already seen, in order to avoid an infinite recursion. I've marked two lines below with comments:

private static void loadReferencedAssemblies(Assembly a)
{
    AssemblyName[] names = a.GetReferencedAssemblies();
    foreach(AssemblyName name in names)
    {
        if(!_loadedAssemblies.ContainsKey(name.FullName)) // 1
        {
            try
            {
                Assembly ra = Assembly.Load(name);
                _loadedAssemblies[ra.FullName] = ra; // 2
                loadReferencedAssemblies(ra);
            }
            catch(System.Exception)
            {
                // Ignore assemblies that cannot be loaded.
            }
        }
    }
}

In line 1, the code tests to see if the dictionary contains an entry for the value of the AssemblyName.FullName property. Later, on line 2, the code adds an entry to this dictionary using the Assembly.FullName property. The code assumes that these two properties are identical. If they are not identical, we could be in trouble.

First, I would add a couple of trace statements to the method, such as:

private static void loadReferencedAssemblies(Assembly a)
{
    AssemblyName[] names = a.GetReferencedAssemblies();
    foreach(AssemblyName name in names)
    {
        Console.WriteLine("Checking to see if we have already loaded " + name.FullName);
        if(!_loadedAssemblies.ContainsKey(name.FullName))
        {
            try
            {
                Assembly ra = Assembly.Load(name);
                Console.WriteLine("Adding an entry for " + ra.FullName);
                _loadedAssemblies[ra.FullName] = ra;
                loadReferencedAssemblies(ra);
            }
            catch(System.Exception)
            {
                // Ignore assemblies that cannot be loaded.
            }
        }
    }
}

If it turns out that the two values of FullName are different, I would probably consider that to be a bug in Mono (since it's not a problem in .NET for Windows).

Let us know what you find out.

Regards,
Mark

miltondp

Mark, thank you very much! That's exactly what I did today afternoon, and I fixed the problem! (sorry, but I'm very happy). I was going to post a comment here later, but just got the mail notification about your post.

This was the problem: my application uses Npgsql.dll (PostgreSQL provider for .NET), which depends on Mono.Security.dll. The latter was copied to my application output directory, and it's version is 2.0.0.0. However, Assembly.Load was loading Mono.Security version 4.0.0.0. And here you have different values for AssemblyName.FullName and Assembly.FullName.

I guess this is not a problem with Mono. I bet my application was working on Windows because there is no Mono.Security installed in the GAC there, as it is in Ubuntu (where the problem was reproducible).

If you are going to fix this problem by making that code more robust, I can provide more details and maybe a small test case.

Regards,
Milton

mes

Milton,

Glad to hear that you solved the problem. I would be interested in seeing your solution.

Regards,
Mark