IceStorm behind a private net

fgarcia · March 2006

Hi, I have been searching the forums and Glacier2, but I have not find a solution for my problem:

I run IceStorm in a pc with a main application that periodicaly publishes some information about different topics. On the other side I have a Siemens modem TC65 with a loaded application that tries to subscribe to any topic. The modem uses the private net of vodafone to connect to internet through an access point. The subscription is successful, but when IceStorm tries to send the published information to our modem, it is unreacheable.

How can I fix it?
Is Glacier2 the solution? If is it, I don't know the way of ussing it in these environment.

Thank you for your help in advance...

marc · March 2006

Yes, you can use Glacier2 for this. There is really nothing special about IceStorm and Glacier2. For Glacier2, IceStorm is just another backend server. I recommend to read the Glacier2 section in our Ice manual, and Matthew's article series about Glacier2 in Connections (starting with issue 1).

fgarcia · March 2006

I have readed the Glacier2 section in the Ice manual and the articles in Connections, but I still have a problem becuase I can't configure the firewalls of the private network of VODAFONE. This is the main problem that I have not access to this firewall, so I can't configure it.

What can I do?
Could I fix it using bi-directional connections? I say this because the "demo/Ice/bidir/README" show this:
"This demo shows how to use bi-directional connections for callbacks. This is typically used if the server cannot open a connection to the client to send callbacks, for example, because firewalls block incoming connections to the client".

Thanks

marc · March 2006

If your server-side firewall completely disallows incoming connections, then there is not much you can do. If it blocks everything, then you can't run any kind of service behind the firewall, and expect clients to access it from outside the firewall.

If, however, your firewall allows a connection to Glacier2, then you're good. You can then run Glacier2 anywhere in your private network, and set the published endpoints to the firewall's IP address/port.

The client-side firewall doesn't matter, as long as you're allowed to make outgoing connections. Of course, if your client-side firewall doesn't allow outgoing connections either, then you're out of luck, too--regardless of whether you use Ice or some other middleware.

fgarcia · March 2006

I am going to show you a diagram and describe the situation.

Attachment not found.

When the subscriber (modem) wants to subscribe to a topic at the IceStorm running in our computer, it uses the private network of vodafone with a private assigned address to connect to internet. At this moment moment, we can see that the subscription has finished successfully, but when the publisher sends new information, IceStorm is unable to find the endpoints of the subscriber because the private address assigned by Vodafone at the subscription time does not exist now because the connection is not persistent.

Is there any way to create a persistent connection or a "tunnel" in this situation?

Thanks for your time.

benoit · March 2006

I believe the problem is that the IceStorm service fails to establish a connection to your subcriber. To solve this, your client should use a bi-directional connection. This way, the IceStorm service would use this bi-directional connection to send updates to your subscriber instead of trying to establish a new connection to your subscriber.

You can't directly use Ice bi-directional connections as shown in the demo/Ice/bidir example with IceStorm. Instead, you should use Glacier2. The Glacier2 process would sit between the Internet and the IceStorm service on your diagram. Your client would connect to Glacier2 and create a session. The connection between your client and Glacier2 would automatically be a bi-directional connection.

I would recommend to try the demo/Glacier2/callback or demo/Glacier2/chat demo. If possible, you could try to deploy this demo in the scenario you describe on your diagram.

The Issue 3 of the newsletter, "An Introduction to IceStorm", also has an example very similar to what you're trying to do with IceStorm, I recommend you to take a look at it and eventually try it.

Let us know if you need more information!

Cheers,
Benoit.

fgarcia · April 2006

Usually when we use Glacier2 (at least in the demos chat and callback), we have a server with an adapter which allocate two servants, a PermissionVerifier and a SessionManager, and a client which starts a session with the glacier router; and finally the client invokes an operations on a proxy's server.
But in my situations I don't know how to do it, because the client is the publisher and the server is the subscriber, so who has to establishes the session with the router?, who has to allocate the SessionManager and the PermissionVerifier?
What should be the steps for the subscriber and the publisher to use icestorm and glacier2 in the previous diagram, supposing that Glacier2 is sat between internet and the IceStorm service?

Thanks for your help

matthew · April 2006

According to the network diagram you posted above since the publishers live within the firewall they do not need to establish glacier2 sessions. However, the subscribers do since they want to cross the firewall boundary.

fgarcia · April 2006

Ok, then I have to establish a glacier2 session with the subscriber. But after reading the documentation of IceStorm and Glacier2, the demos and the Issues, I really do not know how to use IceStorm with Glacier2. Usually the client establishes a glacier2 session, create an adapter with an identity (with the respective category obtained for the session) and invokes operations over the server; but now it is the subscriber who establishes the session (server), and the publisher (client) who invokes the operations...so how can I do it?

Thanks

matthew · April 2006

Usually the client establishes a glacier2 session, create an adapter with an identity (with the respective category obtained for the session) and invokes operations over the server; but now it is the subscriber who establishes the session (server), and the publisher (client) who invokes the operations...so how can I do it?

Don't get confused by the client/server in this case. These are only roles that an application plays at a given time. In this case your subscriber is both a client & a server.

The subscriber creates a glacier2 session. Then you can talk directly to IceStorm and register the object that you have created on the subscribers object adapter to receive callbacks from IceStorm when events arrive. You need to ensure the category is correct on these objects. Other than that, you don't have to do anything special.

fgarcia · April 2006

Now, the subscriber subscribes right to the topic in IceStorm, but when I try to send a message from the publisher, I get this exception in IceStorm:

[ icebox-IceStorm: Topic: Subscribe: U;O.]M\\OAU-qx&34(+^L/monitor ]
[ icebox-IceStorm: Flush: 1 object(s) ]
[ icebox-IceStorm: Flush: 1 object(s) ]
[ icebox-IceStorm: Flush: 1 object(s) ]
[ icebox-IceStorm: Flush: 1 object(s) ]
[ icebox-IceStorm: Flush: 1 object(s) ]
[ icebox-IceStorm: Flush: 1 object(s) ]
[ icebox-IceStorm: Flush: 1 object(s) ]
[ icebox-IceStorm: Flush: 1 object(s) ]
[ icebox-IceStorm: Flush: 1 object(s) ]
[ icebox-IceStorm: Flush: 1 object(s) ]
[ icebox-IceStorm: Flush: 1 object(s) ]
[ icebox-IceStorm: Flush: 1 object(s) ]
icebox-IceStorm: warning: connection exception:
SslTransceiver.cpp:336: Ice::ConnectionLostException:
connection lost: recv() returned zero
local address = 138.4.9.76:48743
remote address = 138.4.9.76:10000
[ icebox-IceStorm: Subscriber: U;O.]M\\OAU-qx&34(+^L/monitor: publish failed: SslTransceiver.cpp:336: Ice::ConnectionLostException:
connection lost: recv() returned zero ]
[ icebox-IceStorm: Flush: 0 object(s) ]

Also when I close the subscriber with C^c I obtain this exception in the subscriber terminal:

subscriber: ObjectAdapterI.cpp:767: IceInternal::ServantManagerPtr Ice::ObjectAdapterI::getServantManager() const: Assertion `_instance' failed.
Abortado

and this one in the glacier2router:

glacier2router: warning: connection exception:
SslTransceiver.cpp:336: Ice::ConnectionLostException:
connection lost: recv() returned zero
local address = 138.4.9.76:10000
remote address = 138.4.9.76:48736

Do you know what do they mean?

Thanks

benoit · April 2006

Hi,

The "Assertion `_instance' failed." error from your subscriber is an Ice bug which has been fixed for the next release.

I'm afraid I don't know why the publish of the event failed with the ConnectionLostException. It sounds like the IceStorm service is trying to connect to the wrong endpoints. The best way to figure this out would be to send us a small self contained example that demonstrates the problem. You could also send us the code of the client showing how you establish the connection with Glacier2 and how you subscribe to the IceStorm topic. Seeing the configuration files of your subscriber, publisher, IceStorm and Glacier2 would also be helpful.

Cheers,
Benoit.

fgarcia · April 2006

Ok, I am going to post the important parts of the code.

* First I execute the icebox service:

$ icebox --Ice.Config=config.service

that launches IceStorm with the following configuration files ("config"), that is also used by the subscriber (I have omitted here some properties of tracing and similars):

# The proxy to the Glacier2 router for all outgoing connections. This
# must match the value of Glacier2.Client.Endpoints in config.glacier2.
#
Ice.Default.Router=DemoGlacier2/router:ssl -p 10000 -h 127.0.0.1

# The proxy for the Glacier2 router, installed in the client's
# object adapter named.... This router proxy must
# match the value of Glacier2.Client.Endpoints.
#
Monitor.Subscriber.Router=DemoGlacier2/router:ssl -p 10000 -h 127.0.0.1

# This property is used to configure the endpoints of the monitor
# subscriber adapter.
#
Monitor.Subscriber.Endpoints=tcp

Ice.ACM.Client=0
Ice.ACM.Server=0

Ice.MonitorConnections=60

#Connections must remain established.
#
Ice.RetryIntervals=-1

Ice.Warn.Connections=1

Ice.Plugin.IceSSL=IceSSL:create
IceSSL.Client.CertPath=../../../certs
IceSSL.Client.Config=sslconfig.xml
IceSSL.Server.CertPath=../../../certs
IceSSL.Server.Config=sslconfig.xml

# This property is used by the subscriber to connect to IceStorm.
#
IceStorm.TopicManager.Proxy=DemoIceStorm/TopicManager:default -p 10005

# This property defines the endpoints on which the IceStorm
# TopicManager listens.
#
IceStorm.TopicManager.Endpoints=default -p 10005

#
# This property defines the endpoints on which the topic
# publisher objects listen.
#
IceStorm.Publish.Endpoints=default

* Then I execute the sessionserver and glacier2router with this configuration file ("config.glacier2"):

Glacier2.InstanceName=DemoGlacier2
Ice.ThreadPerConnection.StackSize=262144

# The client-visible endpoint of Glacier2.
#
Glacier2.Client.Endpoints=ssl -p 10000 -h 127.0.0.1

# The server-visible endpoint of Glacier2.
#
Glacier2.Server.Endpoints=tcp -h 127.0.0.1

# The configures the session manager.
#
Glacier2.SessionManager=sessionmanager:tcp -h 127.0.0.1 -p 10011
Glacier2.PermissionsVerifier=verifier:tcp -h 127.0.0.1 -p 10011

Glacier2.Client.Buffered=1
Glacier2.Server.Buffered=1

Glacier2.Client.ForwardContext=1
Glacier2.Server.ForwardContext=1

Ice.Warn.Connections=1

Ice.Plugin.IceSSL=IceSSL:create
IceSSL.Client.CertPath=../../../certs
IceSSL.Client.Config=sslconfig.xml
IceSSL.Server.CertPath=../../../certs
IceSSL.Server.Config=sslconfig.xml

* The next step I do is to launch the publisher application. The code related with glacier2 in this application is the next:

topic = _manager->retrieve(sTopic);
assert (topic);
Ice::Context context;
context["_fwd"] = "Oz";
Ice::ObjectPrx obj = topic->getPublisher()->ice_newContext(context);
if (!obj->ice_isDatagram()){
obj = obj->ice_oneway();
}
monitor = MonitorPrx::uncheckedCast (obj);

The configuration file that this application use is the following ("config.core"):

#
# This property is used by the clients to connect to IceStorm.
#
IceStorm.TopicManager.Proxy=DemoIceStorm/TopicManager:default -p 10005

* Finally I execute the subscriber, with the next important code (I only post the code different from other subscriber that does not use glacier2):

Ice::RouterPrx defaultRouter = communicator()->getDefaultRouter();
if (!defaultRouter){
cerr << argv[0] << ": no default router set" << endl;
return EXIT_FAILURE;
}

Glacier2::RouterPrx router = Glacier2::RouterPrx::checkedCast(defaultRouter);
if (!router){
cerr << argv[0] << ": configured router is not a Glacier2 router" << endl;
return EXIT_FAILURE;
}

while(true){
cout << "This demo accepts any user-id / password combination.\n";

string id;
cout << "user id: " << flush;
cin >> id;

string pw;
cout << "password: " << flush;
cin >> pw;

try{
router->createSession(id, pw);
break;
}
catch(const Glacier2::PermissionDeniedException& ex){
cout << "permission denied:\n" << ex.reason << endl;
}
}
...
string category = router->getServerProxy()->ice_getIdentity().category;
Ice::Identity monitorIdent;
monitorIdent.name = "monitor";
monitorIdent.category = category;
...
Ice::ObjectAdapterPtr adapter = communicator()->createObjectAdapter("Monitor.Subscriber");
Ice::ObjectPtr monitor = new MonitorI();
...
Ice::ObjectPrx object = adapter->add(monitor,monitorIdent);
try{
IceStorm::TopicPrx topic = manager->retrieve(*p);
topic->subscribe(qos, object);
}
catch(const IceStorm::NoSuchTopic& e){
cerr << appName() << ": " << e << " name: " << e.name << endl;
break;
}
...

The subscriber uses the configuration file "config" that has been show before.

Thanks for all your help, and sorry for this huge post

benoit · April 2006

Hi,

The subscriber and the IceStorm service shouldn't share the same configuration file because right now the IceStorm service is configured to use the Glacier2 router client endpoints to route its outgoing requests (and this fails with an Ice::ConnectionLostException because IceStorm doesn't have any session established with Glacier2). IceStorm should transparently route requests through the Glacier2 server endpoints, without any extra configuration needed, if the proxy used by the client to subscribe is correctly created.

Only Glacier2 clients should have a router configured -- check out the configuration of the demo/Glacier2/callback or chat demos for example.

Let us know if this works better once you split the client and IceStorm configuration in 2 separate files!

Cheers,
Benoit.

fgarcia · April 2006

Thank you very much, I have split the configuration file in two, and it works!!!!
Now I have to replace the local subscriber by a remote subscriber implemented with Ice-E, I will report you the results

Thank you another time

fgarcia · April 2006

Another question about IceStorm and Glacier2. Is it necessary to create the context in the subscriber, like follows?

string category = router->getServerProxy()->ice_getIdentity().category;
Ice::Identity monitorIdent;
monitorIdent.name = "monitor";
monitorIdent.category = category;

Ice::ObjectPrx object = adapter->add(monitor,monitorIdent);
try{
IceStorm::TopicPrx topic = manager->retrieve(*p);
topic->subscribe(qos, object);
}

If I do not create the context and add the servant "monitor" with UUID, it does not give any exception, but the subscriber does not receive the information published by the publisher; but if I create the context, when I try to subscribe to serveral topics with the same subscriber I get this exception:

./subscriber: ServantManager.cpp:47: Ice::AlreadyRegisteredException:
servant with id `Z^v)Qt\/E]F0_o6Z5t58t/monitor' is already registered

The "monitorIdent" is the same in a subscriber for all its ObjectPrx, so what can I do?

Thanks in advance

benoit · April 2006

I assume that by "context" you actually mean "identity". Yes, you need to set the category for your subscriber object identity to the category of the router server proxy identity. This is used by the Glacier2 router to correctly route the requests from the IceStorm service to your subscriber. However, you don't have to use a fixed identity name. For example, you could use the following identity to register your subscriber with a unique identity:

   #include <IceUtil/UUID.h>
   ...

   Ice::Identity id;
   id.category = router->getServerProxy()->ice_getIdentity().category;
   id.name = IceUtil::generateUUID();

Cheers,
Benoit.

fgarcia · April 2006

Hi,
Now all is working, thanks; but the exception

subscriber: ObjectAdapterI.cpp:767: IceInternal::ServantManagerPtr Ice::ObjectAdapterI::getServantManager() const: Assertion `_instance' failed.
Abortado

still appears although now I am ussing Ice-3.0.1. Is it a bug or is an error of my application?

Thanks for all your help

benoit · April 2006

Hi,

Yes, I believe this is a known Ice bug (bug #535 in our internal database). Just to make sure, could you get the stack trace of the assert with the debugger and post it here on the forums?

Cheers,
Benoit.

fgarcia · April 2006

Ok, the traces generated are:

subscriber: ObjectAdapterI.cpp:767: IceInternal::ServantManagerPtr Ice::ObjectAdapterI::getServantManager() const: Assertion `_instance' failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread -1483573024 (LWP 11993)]
0xa79637c7 in raise () from /lib/tls/libc.so.6
(gdb) bt
#0 0xa79637c7 in raise () from /lib/tls/libc.so.6
#1 0xa796506b in abort () from /lib/tls/libc.so.6
#2 0xa795c735 in __assert_fail () from /lib/tls/libc.so.6
#3 0xa7e04a43 in Ice::ObjectAdapterI::getServantManager (this=0x808ec80) at ObjectAdapterI.cpp:767
#4 0xa7d921d4 in Ice::ConnectionI::setAdapter (this=0x807a160, adapter=@0xaf88d43c) at ConnectionI.cpp:1161
#5 0xa7e3fa59 in IceInternal:: DirectReference::getConnection (this=0x808f288, comp=@0x808fa7c) at Reference.cpp:903
#6 0xa7e30bfb in IceDelegateM::Ice::Object::setup (this=0x808fa70, ref=@0x808f258) at Proxy.cpp:1066
#7 0xa7e30eb1 in IceProxy::Ice::Object::__getDelegate (this=0x808f234) at Proxy.cpp:812
#8 0xa7f16e27 in IceProxy::IceStorm::Topic::unsubscribe (this=0x808f230, subscriber=@0x808f73c, __ctx=@0x808f2a8)
at IceStorm.cpp:818
#9 0xa7f16fe0 in IceProxy::IceStorm::Topic::unsubscribe (this=0x808f230, subscriber=@0x808f73c) at IceStorm.cpp:807
#10 0x08059ac0 in Subscriber::run (this=0xaf88d7c8, argc=2, argv=0xaf88d854) at Subscriber.cpp:162
#11 0xa7d5e611 in Ice::Application::main (this=0xaf88d7c8, argc=2, argv=0xaf88d854,
configFile=0x805ec7a "config.subscriber", logger=@0xaf88d7cc) at Application.cpp:266
#12 0x08058b2d in main (argc=2, argv=0xaf88d854) at Subscriber.cpp:30

And the trace of the assert is:

(gdb) up
#3 0xa7e04a43 in Ice::ObjectAdapterI::getServantManager (this=0x808ec80) at ObjectAdapterI.cpp:767
767 assert(_instance); // Must not be called after waitForDeactivate().
(gdb) list
762 // immutable after creation until they are removed in
763 // waitForDeactivate().
764
765 // Not check for deactivation here!
766
767 assert(_instance); // Must not be called after waitForDeactivate().
768
769 return _servantManager;
770 }
771

benoit · April 2006

Thanks, this is indeed the same bug. I suspect that you're calling unsubscribe() on the IceStorm topic after calling destroySession() on the router (from the trace, Ice tries to establish a new connection). You should call it before destroying the session otherwise Glacier2 will reject the unsusbscribe() request. I believe this should also workaround this bug. Let us know if it doesn't though (please also post the shutdown code of your subscriber).

In any case, this bug will be fixed in the next Ice release!

Cheers,
Benoit.

fgarcia · April 2006

This is the shutdown code of the subscriber

...
if(subscribers.size() == topics.size()){
adapter->activate();
shutdownOnInterrupt();
communicator()->waitForShutdown();
}

for(map<string,Ice::ObjectPrx>::const_iterator q = subscribers.begin(); q != subscribers.end(); ++q){
try{
IceStorm::TopicPrx topic = manager->retrieve(q->first);
topic->unsubscribe(q->second);
}
catch(const IceStorm::NoSuchTopic& e){
cerr << appName() << ": " << e << " name: " << e.name << endl;
}
}

return EXIT_SUCCESS;

I don't call destroySession() on the router, but I suppose that the session is destroied when I puss C^c due to the shutdownOnInterrupt() command.

Thanks

benoit · April 2006

Hi,

The communicator shutdown (initiated by Ctrl-C) should only deactivate the object adapter of your client. Outgoing connections shouldn't be closed -- these are closed when the communicator is destroyed.

So in theory, the session and the connection to the router shouldn't be closed after waitForShutdown() returns and you should be able to invoke unsubscribe() on the IceStorm topics. Did you try to run your client with --Ice.Trace.Network to see when the connection to the router is closed? Are you correctly keeping alive the session with a separate thread (see the SessionPingThread class in the demo/Glacier2/chat/Client.cpp for an example on how to do this)?

In any case, cleaning up server resources (here the IceStorm subscribers) in the client isn't very reliable. The connection between the client and the router could be lost and the subscribers would never be unsubscribed from the IceStorm topic and this would eventually lead to resource leaks (see a recent discussion about this [post=9930]here[/post]). The right approach to cleanup server resources allocated by a Glacier2 client is to implement a session manager and clean up the resources allocated by the client in the session implementation. I recommend you to take a look at the Issue 2 and 3 of the Ice newsletter for more information and an example on how to do this.

Cheers,
Benoit.

fgarcia · May 2006

Running the client with --Ice.Trace.Network I get the following result:

[ Network: tcp connection established
local address = 138.4.9.76:38795
remote address = 138.4.9.76:10000 ]
Step 0
[ Network: closing tcp connection
local address = 138.4.9.76:38795
remote address = 138.4.9.76:10000 ]
Step 1
./subscriber: Instance.cpp:199: Ice::CommunicatorDestroyedException:
communicator object destroyed

These traces are related with this code of the client:

adapter->activate();
cout << "Step 0" << endl;
communicator()->waitForShutdown();
cout << "Step 1" << endl;

//
// Unsubscribe all objects.
//
for(map<string,Ice::ObjectPrx>::const_iterator q = subscribers.begin(); q != subscribers.end(); ++q){
try{
IceStorm::TopicPrx topic = manager->retrieve(q->first);
cout << "Step 2" << endl;
topic->unsubscribe(q->second);
}
catch(const IceStorm::NoSuchTopic& e){
cerr << appName() << ": " << e << " name: " << e.name << endl;
}
}

So, I think that the communicator is destroyed after I push Ctrl-C.

Also I have do the changes for keeping the session alive with a separate thread, but I suppose that firstly I have to fix the problem I just posted.

marc · May 2006

Are you sure you don't use destroyOnInterrupt() instead of shutdownOnInterrupt()? The former instructs the Ctrl-C handler used by the Application class instance to destroy the communicator on interrupt, while the latter will only cause the communicator to be shut down.

fgarcia · May 2006

Yes, I'm sure...I do not know why the communicator is destroyed.

marc · May 2006

Does the example in demo/IceStorm/clock work for you? It uses the exact same technique. See demo/IceStorm/clock/Subscriber.cpp.

fgarcia · May 2006

Yes, it works perfectly, so I suppose that the problem is the combination of IceStorm and Glacier2, but I am not sure, and of course I do not know why.

Thanks

marc · May 2006

We will investigate if or why the behavior is different if a router (Glacier2 in this case) is being used.

fgarcia · May 2006

Thanks for your interest

dwayne · May 2006

Hi,

So far we have been unable to reproduce this problem. It would help if you could send us an example (including config files) that reproduces the problem.

Regards,
Dwayne

IceStorm behind a private net

Comments