Welcome to the new Gigaspaces XAP forum. To recover your account, please follow these instructions.

Ask Your Question
0

Space goes into infinite loop when coming up

I have a space with 4 primaries and 4 backups. When I deploy the space, one of the primary/backup combinations goes into an infinite loop after first displaying this error message in the gsc log file:

Failed to init [attrade-space] space.; Caused by: com.j_spaces.core.sadapter.SAException: org.hibernate.exception.SQLGrammarException: could not advance using next()

After this error message, and a few others, the following error message appears in the gsc log file over and over, for hours. The space cannot be undeployed from the Management Center when it gets in this state:

2017-03-30 08:48:08,087 AtTradeSpace.4 [2] WARNING [com.gigaspaces.space.active-election.attrade-space.4_1] - Cannot advance from [PENDING] to [PREPARE] - found better candidate; force change state to [PENDING]

We are converting from Gigaspaces 9 to 10, so not sure if this is a problem with one of the jar files we had to upgrade.

Sometimes when we get the "could not advance using next()" error message, instead of an infinite loop we get this block of error messages: Caused by: com.sybase.jdbc3.jdbc.SybSQLException: SQL Anywhere Error -210: User 'another user' has the row in 'AtTradeBrokerClientAggregate' locked at com.sybase.jdbc3.tds.Tds.a(Unknown Source) at com.sybase.jdbc3.tds.Tds.nextResult(Unknown Source) at com.sybase.jdbc3.tds.TdsCursor.fetch(Unknown Source) at com.sybase.jdbc3.tds.TdsCursor.fetch(Unknown Source) at com.sybase.jdbc3.jdbc.SybCursorResultSet.next(Unknown Source) at org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207) at org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207) at org.hibernate.internal.ScrollableResultsImpl.next(ScrollableResultsImpl.java:120) ... 5 more So we either get the infinite loop, or we get the inability to access tables because another user has them locked.

Here is the entire list of error messages that appears in the infinite loop case:

2017-03-29 15:40:00,118 AtTradeSpace.4 [2] SEVERE [com.gigaspaces.space.engine.attrade-space.4_1] - Failed to init [attrade-space] space.; Caused by: com.j_spaces.core.sadapter.SAException: org.hibernate.exception.SQLGrammarException: could not advance using next() at com.j_spaces.core.cache.CacheManager.initCache(CacheManager.java:581) at com.gigaspaces.internal.server.space.SpaceEngine.init(SpaceEngine.java:563) at com.gigaspaces.internal.server.space.SpaceImpl.initAndRecoverFromDataStorage(SpaceImpl.java:1471) at com.gigaspaces.internal.server.space.recovery.strategy.PrimarySpaceRecovery.recoverFromDB(PrimarySpaceRecovery.java:65) at com.gigaspaces.internal.server.space.recovery.strategy.PrimarySpaceRecovery.recover(PrimarySpaceRecovery.java:56) at com.gigaspaces.internal.server.space.SpaceImpl.initAndStartPrimaryBackupSpace(SpaceImpl.java:1374) at com.gigaspaces.internal.server.space.SpaceImpl.recover(SpaceImpl.java:3316) at com.gigaspaces.internal.server.space.SpaceImpl.startInternal(SpaceImpl.java:3268) at com.gigaspaces.internal.server.space.SpaceImpl.<init>(SpaceImpl.java:261) at com.j_spaces.core.JSpaceContainerImpl.createSpaceImpl(JSpaceContainerImpl.java:1747) at com.j_spaces.core.JSpaceContainerImpl.loadSpace(JSpaceContainerImpl.java:1716) at com.j_spaces.core.JSpaceContainerImpl.xmlInit(JSpaceContainerImpl.java:701) at com.j_spaces.core.JSpaceContainerImpl.initContainer(JSpaceContainerImpl.java:518) at com.j_spaces.core.JSpaceContainerImpl.<init>(JSpaceContainerImpl.java:393) at com.j_spaces.core.client.SpaceFinder.createEmbeddedSpace(SpaceFinder.java:641) at com.j_spaces.core.client.SpaceFinder.findEmbeddedSpace(SpaceFinder.java:677) at com.j_spaces.core.client.SpaceFinder.findService(SpaceFinder.java:501) at com.j_spaces.core.client.SpaceFinder.findService(SpaceFinder.java:456) at com.j_spaces.core.client.SpaceFinder.find(SpaceFinder.java:345) at com.gigaspaces.client.SpaceProxyFactory.createSpaceProxy(SpaceProxyFactory.java:77) at org.openspaces.core.space.InternalSpaceFactory.create(InternalSpaceFactory.java:104) at org.openspaces.core.space.UrlSpaceFactoryBean.doCreateSpace(UrlSpaceFactoryBean.java:99) at org.openspaces.core.space.AbstractSpaceFactoryBean.afterPropertiesSet(AbstractSpaceFactoryBean.java:186) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$6.run(AbstractAutowireCapableBeanFactory.java:1617) at java.security.AccessController.doPrivileged(Native Method) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1614) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1564) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:540) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:229) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1471) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1216) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:538) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:229) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:956) at org.openspaces.core.context.GigaSpaceContextBeanPostProcessor.initMapsIfNecessary(GigaSpaceContextBeanPostProcessor.java:73) at org.openspaces.core.context.GigaSpaceContextBeanPostProcessor.findGigaSpaceByName(GigaSpaceContextBeanPostProcessor.java:95) at org.openspaces.core.context.GigaSpaceContextBeanPostProcessor$AnnotatedMember.resolve(GigaSpaceContextBeanPostProcessor.java:218) at org.openspaces.core.context.GigaSpaceContextBeanPostProcessor$AnnotatedMember.inject(GigaSpaceContextBeanPostProcessor.java:173) at org.openspaces.core.context.GigaSpaceContextBeanPostProcessor.postProcessAfterInstantiation(GigaSpaceContextBeanPostProcessor.java:117) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1166) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:538) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:229) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:725) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480) at org.openspaces.pu.container.integrated.IntegratedProcessingUnitContainerProvider.createContainer(IntegratedProcessingUnitContainerProvider.java:248) at org.openspaces.pu.container.servicegrid.PUServiceBeanImpl.startPU(PUServiceBeanImpl.java:768) at org.openspaces.pu.container.servicegrid.PUServiceBeanImpl.doStart(PUServiceBeanImpl ... (more)

asked 2017-03-30 08:03:17 -0500

LexLuthor gravatar image

updated 2017-03-30 08:55:09 -0500

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

You are getting the following JDBC error: org.openspaces.persistency.hibernate.iterator.AbstractScrollableDataIterator.hasNext(AbstractScrollableDataIterator.java:156) at org.openspaces.persistency.support.ConcurrentMultiDataIterator$DataIteratorRunnable.run(ConcurrentMultiDataIterator.java:154) ... 3 more Caused by: com.sybase.jdbc3.jdbc.SybSQLException: SQL Anywhere Error -210: User 'another user' has the row in 'AtTradeBrokerClientAggregate' locked at com.sybase.jdbc3.tds.Tds.a(Unknown Source) at com.sybase.jdbc3.tds.Tds.nextResult(Unknown Source) at com.sybase.jdbc3.tds.TdsCursor.fetch(Unknown Source) at com.sybase.jdbc3.tds.TdsCursor.fetch(Unknown Source) at com.sybase.jdbc3.jdbc.SybCursorResultSet.next(Unknown Source) at

Can you please investigate this and why they have this row locked? Thanks, Dixson

answered 2017-03-30 08:52:55 -0500

Dixson Huie gravatar image
edit flag offensive delete link more

Comments

We haven't been able to determine why the row is locked, but our theory is that the 4 partitions are getting in each other's way. They appear to be locking the database at the same time, so that this error message appears. I tested this theory by only having 1 partition, and this error message does not occur. I did this by changing this line in the deploy script:
-max-instances-per-vm 1 -cluster schema=partitioned-sync2backup total_members=4,1
to this line:
-max-instances-per-vm 1 -cluster schema=partitioned-sync2backup total_members=1,1
But now we have a new problem. Our code is searching for values in the non-existent partitions 2 through 4. I wonder if fixing that requires modifying @SpaceRouting or @SpaceIndex on some of our classes?

2017-04-03 11:02:00,347 [ajp-0.0.0.0-8501-46] ERROR STDERR - Apr 03, 2017 11:02:00 AM com.gigaspaces.internal.remoting.routing.clustered.SpaceProxyLoadBalancingStrategy findActiveMember SEVERE: Failed to find an active server for partition #2 to execute ExecuteTaskSpaceOperationRequesttask=org.openspaces.core.executor.internal.InternalSpaceTaskWrapper@3947cbb6 2017-04-03 11:02:00,348 [ajp-0.0.0.0-8501-46] ERROR com.itgssi.springmvc.PnLMonitorController - SDA request failed ! java.lang.reflect.UndeclaredThrowableException

Caused by: java.rmi.RemoteException: Failed to find an active server for partition #2 to execute ExecuteTaskSpaceOperationRequesttask=org.openspaces.core.executor.internal.InternalSpaceTaskWrapper@3947cbb6 at com.gigaspaces.internal.remoting.routing.clustered.SpaceProxyLoadBalancingStrategy.findActiveMember(SpaceProxyLoadBalancingStrategy.java:69) at com.gigaspaces.internal.remoting.routing.clustered.SpaceProxyLoadBalancingStrategy.findActiveMemberUninterruptibly(SpaceProxyLoadBalancingStrategy.java:85) at com.gigaspaces.internal.remoting.routing.clustered.ClusterRemoteOperationRouter$AsyncOperationExecutor.executeAsync(ClusterRemoteOperationRouter.java:243) at com.gigaspaces.internal.remoting.routing.clustered.ClusterRemoteOperationRouter.executeAsync(ClusterRemoteOperationRouter.java:181) at com.gigaspaces.internal.remoting.routing.partitioned.PartitionedClusterRemoteOperationRouter.executeSingleAsync(PartitionedClusterRemoteOperationRouter.java:226) at com.gigaspaces.internal.remoting.routing.partitioned.PartitionedClusterRemoteOperationRouter.executeAsync(PartitionedClusterRemoteOperationRouter.java:153) at com.gigaspaces.internal.client.spaceproxy.router.SpaceProxyRouter.executeAsync(SpaceProxyRouter.java:238) at com.gigaspaces.internal.client.spaceproxy.actions.SpaceProxyImplAdminAction.execute(SpaceProxyImplAdminAction.java:67) at com.gigaspaces.internal.client.spaceproxy.actions.SpaceProxyImplAdminAction.execute(SpaceProxyImplAdminAction.java:29) at com.gigaspaces.internal.client.spaceproxy.actions.AbstractSpaceProxyActionManager.executeTask(AbstractSpaceProxyActionManager.java:147) at com.gigaspaces.internal.client.spaceproxy.AbstractSpaceProxy.execute(AbstractSpaceProxy.java:199) at org.openspaces.core.DefaultGigaSpace.execute(DefaultGigaSpace.java:1622) at org.openspaces.core.DefaultGigaSpace.execute(DefaultGigaSpace.java:1612) at org.openspaces.remoting.ExecutorSpaceRemotingProxyFactoryBean.invokeRemote(ExecutorSpaceRemotingProxyFactoryBean.java:313) at org.openspaces.remoting.ExecutorSpaceRemotingProxyFactoryBean.invoke(ExecutorSpaceRemotingProxyFactoryBean.java:231) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) ... 55 more

LexLuthor gravatar imageLexLuthor ( 2017-04-03 12:58:15 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2017-03-30 08:03:17 -0500

Seen: 32 times

Last updated: Mar 30