hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1073) HStore out of sync with what is on filesystem
Date Sun, 21 Dec 2008 03:22:44 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658348#action_12658348
] 

stack commented on HBASE-1073:
------------------------------

It looks like it though its hard to tell with these binary keys.  At the time that problematic
region is first registered, there is shutdown processing going on for another server altogether,
server 214.  This seems to mess things up.   Regions that were just written into the meta
as being successfully opened on server 218 are then assigned out to server 215 and 210.

Here is illustration of above (grep on 1229470182375):
{code}
2008-12-16 15:31:20,375 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: streamitems,^@^@^@^@^@�^D;,1229470182375
open on 72.34.249.218:60020
2008-12-16 15:31:20,375 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row streamitems,^@^@^@^@^@�^D;,1229470182375 in region .META.,,1 with startcode 1229369663539
and server 72.34.249.218:60020
2008-12-16 15:31:20,376 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop:
PendingOpenOperation from 72.34.249.218:60020
2008-12-16 15:31:20,376 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: streamitems,^@^@^@^@^@�^V�,1229470182375
open on 72.34.249.218:60020
2008-12-16 15:31:20,376 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row streamitems,^@^@^@^@^@�^V�,1229470182375 in region .META.,,1 with startcode 1229369663539
and server 72.34.249.218:60020
2008-12-16 15:31:21,745 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^B�|s,1229459878511 to server 72.34.249.212:60020
2008-12-16 15:31:21,746 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^B�^Us,1229459878511 to server 72.34.249.212:60020
2008-12-16 15:31:21,747 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
commentchunks,,1229364208367 to server 72.34.249.212:60020
2008-12-16 15:31:21,747 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^B�B�,1229459764918 to server 72.34.249.212:60020
2008-12-16 15:31:21,747 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^@xƙ,1229460120667 to server 72.34.249.212:60020
2008-12-16 15:31:21,748 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^@r��,1229459815399 to server 72.34.249.212:60020
2008-12-16 15:31:21,748 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^B7Oi,1229374367471 to server 72.34.249.212:60020
2008-12-16 15:31:21,749 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^C|��,1229375922174 to server 72.34.249.212:60020
2008-12-16 15:31:21,749 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^C�n�,1229459949444 to server 72.34.249.212:60020
2008-12-16 15:31:21,749 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^@^Z�q,1229375049350 to server 72.34.249.212:60020
2008-12-16 15:31:22,101 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^@�P-,1229460052735 to server 72.34.249.215:60020
2008-12-16 15:31:22,101 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^A�n/,1229373923594 to server 72.34.249.215:60020
2008-12-16 15:31:22,102 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^@��[,1229460052735 to server 72.34.249.215:60020
2008-12-16 15:31:22,102 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^@uU1,1229460120667 to server 72.34.249.215:60020
2008-12-16 15:31:22,103 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^A��,1229373923594 to server 72.34.249.215:60020
2008-12-16 15:31:22,103 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^B:7�,1229374367471 to server 72.34.249.215:60020
2008-12-16 15:31:22,103 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^C���,1229375603887 to server 72.34.249.215:60020
2008-12-16 15:31:22,104 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^@.@3,1229374602767 to server 72.34.249.215:60020
2008-12-16 15:31:22,104 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^@�^V�,1229470182375 to server 72.34.249.215:60020
2008-12-16 15:31:22,105 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^C^E(!,1229459984826 to server 72.34.249.215:60020
2008-12-16 15:31:22,119 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^C^Bi�,1229459984826 to server 72.34.249.219:60020
2008-12-16 15:31:22,119 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^@eϭ,1229374034359 to server 72.34.249.219:60020
2008-12-16 15:31:22,120 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^B^Y/a,1229374146691 to server 72.34.249.219:60020
2008-12-16 15:31:22,120 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^B��,1229459764918 to server 72.34.249.219:60020
2008-12-16 15:31:22,120 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^@pZ�,1229459815399 to server 72.34.249.219:60020
2008-12-16 15:31:22,121 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^C�hg,1229375603887 to server 72.34.249.219:60020
2008-12-16 15:31:22,121 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^@k^G/,1229369754111 to server 72.34.249.219:60020
2008-12-16 15:31:22,122 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^AfB^Q,1229459739597 to server 72.34.249.219:60020
2008-12-16 15:31:22,122 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^C^?�+,1229375922174 to server 72.34.249.219:60020
2008-12-16 15:31:22,123 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
sourceurls,,1229364124122 to server 72.34.249.219:60020
2008-12-16 15:31:22,239 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {regionname: .META.,,1, startKey: <>, server: 72.34.249.216:60020}
2008-12-16 15:31:22,300 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
streamitems,^@^@^@^@^@�^D;,1229470182375 to server 72.34.249.210:60020
{code}

Here is lifetime of the regions of rid 1229470182375
{code}
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:29:45,601 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_SPLIT: streamitems,^@^@^@^@^@�^V�,1229374149929: streamitems,^@^@^@^@^@�^V�,1229374149929
split; daughters: streamitems,^@^@^@^@^@�^V�,1229470182375, streamitems,^@^@^@^@^@�^D;,1229470182375
from XX.XX.XX.217:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:29:51,115 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^@^@^@^@^@�^D;,1229470182375 to server XX.XX.XX.218:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:29:51,119 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^@^@^@^@^@�^V�,1229470182375 to server XX.XX.XX.218:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:29:54,126 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_PROCESS_OPEN: streamitems,^@^@^@^@^@�^V�,1229470182375 from XX.XX.XX.218:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:29:54,126 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_OPEN: streamitems,^@^@^@^@^@�^D;,1229470182375 from XX.XX.XX.218:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:29:54,126 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_OPEN: streamitems,^@^@^@^@^@�^V�,1229470182375 from XX.XX.XX.218:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:30:22,421 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
streamitems,^@^@^@^@^@�^V�,1229470182375 no longer has references to streamitems,^@^@^@^@^@�^V�,1229374149929
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:30:22,423 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
streamitems,^@^@^@^@^@�^D;,1229470182375 no longer has references to streamitems,^@^@^@^@^@�^V�,1229374149929
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:20,288 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation:
shutdown scanner looking at streamitems,^@^@^@^@^@�^V�,1229470182375
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:20,289 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation:
shutdown scanner looking at streamitems,^@^@^@^@^@�^D;,1229470182375
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:20,375 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
streamitems,^@^@^@^@^@�^D;,1229470182375 open on XX.XX.XX.218:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:20,375 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
updating row streamitems,^@^@^@^@^@�^D;,1229470182375 in region .META.,,1 with startcode
1229369663539 and server XX.XX.XX.218:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:20,376 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
streamitems,^@^@^@^@^@�^V�,1229470182375 open on XX.XX.XX.218:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:20,376 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
updating row streamitems,^@^@^@^@^@�^V�,1229470182375 in region .META.,,1 with startcode
1229369663539 and server XX.XX.XX.218:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:22,104 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^@^@^@^@^@�^V�,1229470182375 to server XX.XX.XX.215:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:22,300 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region streamitems,^@^@^@^@^@�^D;,1229470182375 to server XX.XX.XX.210:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:25,113 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_PROCESS_OPEN: streamitems,^@^@^@^@^@�^V�,1229470182375 from XX.XX.XX.215:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:25,334 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_OPEN: streamitems,^@^@^@^@^@�^D;,1229470182375 from XX.XX.XX.210:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:25,335 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
streamitems,^@^@^@^@^@�^D;,1229470182375 open on XX.XX.XX.210:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:25,335 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
updating row streamitems,^@^@^@^@^@�^D;,1229470182375 in region .META.,,1 with startcode
1229369663563 and server XX.XX.XX.210:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:28,129 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_PROCESS_OPEN: streamitems,^@^@^@^@^@�^V�,1229470182375 from XX.XX.XX.215:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:31,142 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_PROCESS_OPEN: streamitems,^@^@^@^@^@�^V�,1229470182375 from XX.XX.XX.215:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:31,143 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_OPEN: streamitems,^@^@^@^@^@�^V�,1229470182375 from XX.XX.XX.215:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:31,145 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
streamitems,^@^@^@^@^@�^V�,1229470182375 open on XX.XX.XX.215:60020
hbase-hbase-master-mb0.streamy.com.log.2008-12-16:2008-12-16 15:31:31,145 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
updating row streamitems,^@^@^@^@^@�^V�,1229470182375 in region .META.,,1 with startcode
1229369663604 and server XX.XX.XX.215:60020
{code}



> HStore out of sync with what is on filesystem
> ---------------------------------------------
>
>                 Key: HBASE-1073
>                 URL: https://issues.apache.org/jira/browse/HBASE-1073
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.19.0
>
>
> On streamy cluster, TRUNK, I saw this when we went to major compact:
> {code}
> 2008-12-17 16:21:48,725 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Major compaction
triggered on store: 657002096/content. Time since last major compaction: 89517 seconds
> 2008-12-17 16:21:48,842 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Started compaction
of 1 file(s)  into /hbase/streamitems/compaction.dir/657002096/content/mapfiles/362900034457176172
> 2008-12-17 16:21:56,354 INFO org.apache.hadoop.hdfs.DFSClient: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /hbase/streamitems/compaction.dir/657002096/content/mapfiles/362900034457176172/data
File does not exist. [Lease.  Holder: DFSClient_-425857031, pendingcreates: 1]
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1331)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1322)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1250)
>         at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>         at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)
>         at org.apache.hadoop.ipc.Client.call(Client.java:696)
>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
>         at $Proxy1.addBlock(Unknown Source)
>         at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>         at $Proxy1.addBlock(Unknown Source)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2815)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2697)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
> 2008-12-17 16:21:56,354 WARN org.apache.hadoop.hdfs.DFSClient: NotReplicatedYetException
sleeping /hbase/streamitems/compaction.dir/657002096/content/mapfiles/362900034457176172/data
retries left 4
> 2008-12-17 16:21:56,757 INFO org.apache.hadoop.hdfs.DFSClient: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /hbase/streamitems/compaction.dir/657002096/content/mapfiles/362900034457176172/data
File does not exist. [Lease.  Holder: DFSClient_-425857031, pendingcreates: 1]
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1331)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1322)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1250)
>         at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>         at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)
>         at org.apache.hadoop.ipc.Client.call(Client.java:696)
>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
>         at $Proxy1.addBlock(Unknown Source)
>         at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>         at $Proxy1.addBlock(Unknown Source)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2815)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2697)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
> 2008-12-17 16:21:56,757 WARN org.apache.hadoop.hdfs.DFSClient: NotReplicatedYetException
sleeping /hbase/streamitems/compaction.dir/657002096/content/mapfiles/362900034457176172/data
retries left 3
> 2008-12-17 16:21:57,561 INFO org.apache.hadoop.hdfs.DFSClient: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /hbase/streamitems/compaction.dir/657002096/content/mapfiles/362900034457176172/data
File does not exist. [Lease.  Holder: DFSClient_-425857031, pendingcreates: 1]
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1331)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1322)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1250)
>         at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>         at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)
>         at org.apache.hadoop.ipc.Client.call(Client.java:696)
>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
>         at $Proxy1.addBlock(Unknown Source)
>         at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>         at $Proxy1.addBlock(Unknown Source)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2815)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2697)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
> 2008-12-17 16:21:57,561 WARN org.apache.hadoop.hdfs.DFSClient: NotReplicatedYetException
sleeping /hbase/streamitems/compaction.dir/657002096/content/mapfiles/362900034457176172/data
retries left 2
> 2008-12-17 16:21:59,162 INFO org.apache.hadoop.hdfs.DFSClient: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /hbase/streamitems/compaction.dir/657002096/content/mapfiles/362900034457176172/data
File does not exist. [Lease.  Holder: DFSClient_-425857031, pendingcreates: 1]
> ...
> {code}
> Thereafter, an attempt to open a scanner on region fails...
> {code}
> 2008-12-18 00:41:03,964 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
completed on region streamitems,^@^@^@^@^Bv�^K,1229373844647 in 0sec
> 2008-12-18 01:08:50,274 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Failed
openScanner
> java.io.IOException: HStoreScanner failed construction
>         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:70)
>         at org.apache.hadoop.hbase.regionserver.HStoreScanner.<init>(HStoreScanner.java:84)
>         at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2120)
>         at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.<init>(HRegion.java:1978)
>         at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1160)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1581)
>         at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:892)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://mb0:9000/hbase/streamitems/657002096/content/mapfiles/2509136481693763805/data
>         at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
>         at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:679)
>         at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
>         at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
>         at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:302)
>         at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
>         at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:284)
>         at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.<init>(HBaseMapFile.java:81)
>         at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.<init>(BloomFilterMapFile.java:66)
>         at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:443)
>         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
>         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:67)
>         ... 10 more
> ...
> {code}
> The first error seems to mess up our view of whats on the filesystem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message