Why am I getting SpaceProxyImpl handleInactiveSpaceException ???

edit

I've been using a primary-backup cluster and I find that writing an Entry to it right after the network cable has been removed from the primary machine, I will get a pause (now down to about 20 seconds) and then the Entry will get written to the backup machine once it has failed over successfully and becomes the primary.

However, suddenly today, I am getting SpaceProxyImpl handleInactiveSpaceExceptions .... Why is this happening? What am I doing wrong to cause this? The write behavior does not seem robust now with this error - it seems like the write does not always happen successfully now. What could I have done to make this happen and how do I stop it?

thanks,

-dave

Client-side logging:

UOW: Dirty Element to update: 3.3.3.5 / 255.255.255.0 : -- 6f1f0462-9c7a-4296-8f89-e54e814cf0aa
Updating Entry: com.mystuff.persistence.NetworkEntry@e66ae7 : name=3.3.3.5 -- UID = -1878825686³⁰6f1f0462-9c7a-4296-8f89-e54e814cf0aa⁰0
Feb 9, 2010 11:26:07 AM com.gigaspaces.internal.client.spaceproxy.SpaceProxyImpl handleInactiveSpaceException
INFO: Accessed an inactive backup space MySpacecontainer11:MySpace, retrying 1/20
Feb 9, 2010 11:26:08 AM com.gigaspaces.internal.client.spaceproxy.SpaceProxyImpl handleInactiveSpaceException
INFO: Accessed an inactive backup space MySpacecontainer11:MySpace, retrying 2/20
Feb 9, 2010 11:26:09 AM com.gigaspaces.internal.client.spaceproxy.SpaceProxyImpl handleInactiveSpaceException
INFO: Accessed an inactive backup space MySpacecontainer11:MySpace, retrying 3/20
Feb 9, 2010 11:26:10 AM com.gigaspaces.internal.client.spaceproxy.SpaceProxyImpl handleInactiveSpaceException
INFO: Accessed an inactive backup space MySpacecontainer11:MySpace, retrying 4/20
Feb 9, 2010 11:26:11 AM com.gigaspaces.internal.client.spaceproxy.SpaceProxyImpl handleInactiveSpaceException
INFO: Accessed an inactive backup space MySpacecontainer11:MySpace, retrying 5/20
Feb 9, 2010 11:26:12 AM com.gigaspaces.internal.client.spaceproxy.SpaceProxyImpl handleInactiveSpaceException
INFO: Accessed an inactive backup space MySpacecontainer11:MySpace, retrying 6/20
Feb 9, 2010 11:26:13 AM com.gigaspaces.internal.client.spaceproxy.SpaceProxyImpl handleInactiveSpaceException
INFO: Accessed an inactive backup space MySpacecontainer11:MySpace, retrying 7/20
Feb 9, 2010 11:26:14 AM com.gigaspaces.internal.client.spaceproxy.SpaceProxyImpl handleInactiveSpaceException
INFO: Accessed an inactive backup space MySpacecontainer11:MySpace, retrying 8/20
Feb 9, 2010 11:26:15 AM com.gigaspaces.internal.client.spaceproxy.SpaceProxyImpl handleInactiveSpaceException
INFO: Accessed an inactive backup space MySpacecontainer11:MySpace, retrying 9/20
Feb 9, 2010 11:26:16 AM com.gigaspaces.internal.client.spaceproxy.SpaceProxyImpl handleInactiveSpaceException
INFO: Accessed an inactive backup space MySpacecontainer11:MySpace, retrying 10/20
Feb 9, 2010 11:26:16 AM net.jini.lookup.ServiceDiscoveryManager lookup
INFO: Exception occurred during query, discarding reggie com.sun.jini.reggie.RegistrarProxy[registrar=dd635398-2d70-4cf7-b2bd-fd343d377114 DynamicSmartStub [ImplObjClass: com.sun.jini.reggie.GigaRegistrar, ConnectionURL: NIO://192.168.2.172:40001/pid[20393/12927536744366941-4172152096200269046, MaxConnPool: 1024 ]]]
java.rmi.ConnectIOException: LRMI transport protocol over NIO broken connection with ServerEndPoint: NIO://192.168.2.172:40001/pid[20393/12927536744366941-4172152096200269046]; nested exception is:
    java.net.SocketTimeoutException: connect timed out
    at com.gigaspaces.lrmi.nio.CPeer.invoke(CPeer.java:501)
    at com.gigaspaces.lrmi.ConnPoolInvocationHandler.invoke(ConnPoolInvocationHandler.java:57)
    at com.gigaspaces.lrmi.DynamicSmartStub.invokeRemote(DynamicSmartStub.java:366)
    at com.gigaspaces.lrmi.DynamicSmartStub.invoke(DynamicSmartStub.java:354)
    at $Proxy0.lookup(Unknown Source)
    at com.sun.jini.reggie.RegistrarProxy.lookup(RegistrarProxy.java:134)
    at net.jini.lookup.ServiceDiscoveryManager.lookup(ServiceDiscoveryManager.java:2890)
    at com.j_spaces.core.client.LookupFinder.find(LookupFinder.java:362)
    at com.j_spaces.core.client.SpaceFinder.findJiniService(SpaceFinder.java:1045)
    at com.j_spaces.core.client.SpaceFinder.findService(SpaceFinder.java:1023)
    at com.j_spaces.core.client.SpaceFinder.findService(SpaceFinder.java:592)
    at com.j_spaces.core.client.SpaceFinder.find(SpaceFinder.java:413)
    at com.j_spaces.core.client.SpaceFinder.find(SpaceFinder.java:357)
    at com.j_spaces.core.cluster.connectivity.SpacesConnectionManager.findRemoteSpace(SpacesConnectionManager.java:424)
    at com.j_spaces.core.cluster.connectivity.SpacesConnectionManager.connect(SpacesConnectionManager.java:363)
    at com.j_spaces.core.cluster.connectivity.SpacesConnectionManager$LivenessDetector.run(SpacesConnectionManager.java:710)
    at java.lang.Thread.run(Thread.java:619)
    at com.j_spaces.kernel.ScheduledRunner.run(ScheduledRunner.java:85)
Caused by: java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
    at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:519)
    at com.gigaspaces.lrmi.nio.watchdog.RequestTimeoutObserver.timeoutOccured(RequestTimeoutObserver.java:62)
    at com.gigaspaces.lrmi.nio.watchdog.Watchdog$WatchdogGroup.fireTimeoutOccured(Watchdog.java:406)
    at com.gigaspaces.lrmi.nio.watchdog.Watchdog$WatchdogGroup.timeout(Watchdog.java:383)
    at com.gigaspaces.lrmi.nio.watchdog.Watchdog$WatchdogGroup.access$100(Watchdog.java:270)
    at com.gigaspaces.lrmi.nio.watchdog.Watchdog.run(Watchdog.java:180)
Feb 9, 2010 11:26:17 AM com.gigaspaces.internal.client.spaceproxy.SpaceProxyImpl handleInactiveSpaceException
INFO: Accessed an inactive backup space MySpacecontainer11:MySpace, retrying 11/20
Feb 9, 2010 11:26:17 AM net.jini.lookup.ServiceDiscoveryManager lookup
INFO: Exception occurred during query, discarding reggie com.sun.jini.reggie.RegistrarProxy[registrar=dd635398-2d70-4cf7-b2bd-fd343d377114 DynamicSmartStub [ImplObjClass: com.sun.jini.reggie.GigaRegistrar, ConnectionURL: NIO://192.168.2.172:40001/pid[20393/12927536744366941-4172152096200269046, MaxConnPool: 1024 ]]]
java.rmi.ConnectException: Connect Failed to NIO://192.168.2.172:40001/pid[20393/12927536744366941-4172152096200269046]; nested exception is:
    java.net.SocketTimeoutException
    at com.gigaspaces.lrmi.nio.CPeer.connect(CPeer.java:183)
    at com.gigaspaces.lrmi.ConnectionPool.getConnection(ConnectionPool.java:56)
    at com.gigaspaces.lrmi.ConnPoolInvocationHandler.invoke(ConnPoolInvocationHandler.java:40)
    at com.gigaspaces.lrmi.DynamicSmartStub.invokeRemote(DynamicSmartStub.java:366)
    at com.gigaspaces.lrmi.DynamicSmartStub.invoke(DynamicSmartStub.java:354)
    at $Proxy0.lookup(Unknown Source)
    at com.sun.jini.reggie.RegistrarProxy.lookup(RegistrarProxy.java:134)
    at net.jini.lookup.ServiceDiscoveryManager.lookup(ServiceDiscoveryManager.java:2890)
    at com.j_spaces.core.client.LookupFinder.find(LookupFinder.java:362)
    at com.j_spaces.core.client.SpaceFinder.findJiniService(SpaceFinder.java:1045)
    at com.j_spaces.core.client.SpaceFinder.findService(SpaceFinder.java:1023)
    at com.j_spaces.core.client.SpaceFinder.findService(SpaceFinder.java:592)
    at com.j_spaces.core.client ...