geode-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GEODE-774) WAN Cluster outgage due to DiskAccessException
Date Mon, 29 Feb 2016 20:51:18 GMT

    [ https://issues.apache.org/jira/browse/GEODE-774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172587#comment-15172587
] 

ASF subversion and git services commented on GEODE-774:
-------------------------------------------------------

Commit 1e31fa646f414aeea28c7fa28096b70dc2ccd120 in incubator-geode's branch refs/heads/feature/GEODE-953
from zhouxh
[ https://git-wip-us.apache.org/repos/asf?p=incubator-geode.git;h=1e31fa6 ]

GEODE-774

Add and enhance junit tests for previous fix.


> WAN Cluster outgage due to DiskAccessException
> ----------------------------------------------
>
>                 Key: GEODE-774
>                 URL: https://issues.apache.org/jira/browse/GEODE-774
>             Project: Geode
>          Issue Type: Bug
>            Reporter: xiaojian zhou
>            Assignee: xiaojian zhou
>             Fix For: 1.0.0-incubating, 1.0.0-incubating.M1
>
>
> {noformat}
> This is the 52550 issue we simulated and reproduced in house. 
> CLIENT vm_4_thr_9_bridge_1_2_w2-2013-lin-04_6672
> TASK[0] newWan.WANTest.HydraTask_doOps
> ERROR util.TestException: com.gemstone.gemfire.cache.CacheClosedException: For DiskStore:
disk3: Fatal error from asynchronous flusher thread, caused by com.gemstone.gemfire.cache.DiskAccessException:
For DiskStore: disk3: Fatal error from asynchronous flusher thread, caused by java.lang.AssertionError:
Version stamp should have a member at this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a
(key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; time=1452091668017};member=null)
>         at com.gemstone.gemfire.internal.cache.GemFireCacheImpl$Stopper.generateCancelledException(GemFireCacheImpl.java:1464)
>         at com.gemstone.gemfire.CancelCriterion.checkCancelInProgress(CancelCriterion.java:91)
>         at com.gemstone.gemfire.internal.cache.LocalRegion.checkRegionDestroyed(LocalRegion.java:8118)
>         at com.gemstone.gemfire.internal.cache.LocalRegion.checkReadiness(LocalRegion.java:2994)
>         at com.gemstone.gemfire.internal.cache.PartitionedRegion$KeysSet$KeysSetIterator.next(PartitionedRegion.java:6544)
>         at com.gemstone.gemfire.internal.cache.PartitionedRegion$KeysSet.toArray(PartitionedRegion.java:6638)
>         at com.gemstone.gemfire.internal.cache.PartitionedRegion$KeysSet.toArray(PartitionedRegion.java:6631)
>         at java.util.Collections$UnmodifiableCollection.toArray(Collections.java:1033)
>         at java.util.ArrayList.addAll(ArrayList.java:577)
>         at newWan.WANOperationsClient.getExistingKeys(WANOperationsClient.java:792)
>         at newWan.WANOperationsClient.putAll(WANOperationsClient.java:463)
>         at newWan.WANOperationsClient.doOps(WANOperationsClient.java:1419)
>         at newWan.WANTest.doOps(WANTest.java:1650)
>         at newWan.WANTest.HydraTask_doOps(WANTest.java:1639)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:497)
>         at hydra.MethExecutor.execute(MethExecutor.java:198)
>         at hydra.MethExecutor.execute(MethExecutor.java:162)
>         at hydra.TestTask.execute(TestTask.java:195)
>         at hydra.RemoteTestModule$1.run(RemoteTestModule.java:216)
> Caused by: com.gemstone.gemfire.cache.DiskAccessException: For DiskStore: disk3: Fatal
error from asynchronous flusher thread, caused by java.lang.AssertionError: Version stamp
should have a member at this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a
(key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; time=1452091668017};member=null)
>         at com.gemstone.gemfire.internal.cache.DiskStoreImpl$FlusherThread.run(DiskStoreImpl.java:1812)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.AssertionError: Version stamp should have a member at this point
for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a (key=Object_36851; rawValue=VMCachedDeserializable@454500130;
version={v0; rv0; ds=0; time=1452091668017};member=null)
>         at com.gemstone.gemfire.internal.cache.Oplog.create(Oplog.java:3553)
>         at com.gemstone.gemfire.internal.cache.PersistentOplogSet.create(PersistentOplogSet.java:189)
>         at com.gemstone.gemfire.internal.cache.DiskStoreImpl.put(DiskStoreImpl.java:751)
>         at com.gemstone.gemfire.internal.cache.DiskRegion.put(DiskRegion.java:381)
>         at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeBytesToDisk(DiskEntry.java:874)
>         at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeToDisk(DiskEntry.java:867)
>         at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeToDisk(DiskEntry.java:857)
>         at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.doAsyncFlush(DiskEntry.java:1742)
>         at com.gemstone.gemfire.internal.cache.DiskStoreImpl$FlusherThread.run(DiskStoreImpl.java:1768)
>         ... 1 more
>         [CIRCULAR REFERENCE:com.gemstone.gemfire.cache.DiskAccessException: For DiskStore:
disk3: Fatal error from asynchronous flusher thread, caused by java.lang.AssertionError: Version
stamp should have a member at this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a
(key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; time=1452091668017};member=null)]
> To reproduce it, run following bt test:
> rollingupgrade/newWan/parallel/garminHCT.conf
> bridgeHostsPerSite=2
> bridgeThreadsPerVM=2
> bridgeVMsPerHost=4
> clientMem=2g
> edgeHostsPerSite=2
> edgeThreadsPerVM=5
> edgeVMsPerHost=5
> locatorHostsPerSite=2
> locatorThreadsPerVM=1
> locatorVMsPerHost=1
> resultWaitSec=1200
> serverMem=12g
> wanSites=2
> with local.conf:
> newWan.WANOperationsClientPrms-randModValueForDebug = true;
> hydra.HostPrms-hostNames =
>   fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 w2-2013-lin-14 w2-2013-lin-15\",
 1, 4)" ncf
>   fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 w2-2013-lin-14 w2-2013-lin-15\",
 1, 4)" ncf
>   fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 w2-2013-lin-14 w2-2013-lin-15\",
 1, 4)" ncf
>   ;
> hydra.HostPrms-resourceDirBaseMapFileName = /home/xzhou/bin/hostdirs.prop;
> hydra.RegionPrms-cacheListeners = none;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message