hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cosmin Lehene <cleh...@adobe.com>
Subject Re: Loosing an entire table after restarting the region server hosting it
Date Tue, 09 Dec 2008 17:01:26 GMT
This also reproduces on HBase-0.18.1 and HBase-0.18.2. Apparently its org.apache.hadoop.hbase.DroppedSnapshotException
that is causing the cache data loss.

If between step 4 and step 5 the cache is flushed (e.g. Hbase cluster restart) no data is
loss. However it means that if you stop a region server with dirty cache you will loose some
data.

Step 5 is hbase/bin/hbase-daemon.sh stop regionserver

I raised https://issues.apache.org/jira/browse/HBASE-1052

On 12/9/08 4:53 PM, "Cosmin Lehene" <clehene@adobe.com> wrote:

Hi,
It seems https://issues.apache.org/jira/browse/HBASE-1010 has some ugly effects: data from
entire table is missing after stopping the HRegionServer

Hadoop 0.18.2
Hbase 0.18.0

Reproducing it is simple:

 1.  Start a Hbase cluster
 2.  Create a table t1: create 't1', {NAME => 'f1'}
 3.  Put a cell in the table: put 't1', 'r1', 'f1:', 'value'
 4.  Scan it, see it's fine
 5.  Stop the HRegionSever hosting the t1 region. Watch the region being reassigned from the
original HRegionServer
 6.  Scan the t1 table again. It's empty now.


The problem we initially found was that a few hours after restarting a HREgionServer we couldn't
instantiate any scanners on the affected table. The problem could have been solved with a
HRegionServer restart. However when trying to reproduce it the entire table content disappeared
.

Thanks,
Cosmin


Original HRegionServer:

java version "1.6.0_06"
Java(TM) SE Runtime Environment (build 1.6.0_06-b02)
Java HotSpot(TM) 64-Bit Server VM (build 10.0-b22, mixed mode)
ulimit -n 32768
2008-12-09 06:35:26,761 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics
with hostName=HRegionServer, port=60020
2008-12-09 06:35:26,862 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Telling
master at h2:60000 that we are up
2008-12-09 06:35:27,559 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from
master: hbase.regionserver.address=10.131.237.51
2008-12-09 06:35:27,559 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from
master: fs.default.name=hdfs://h1:54310/hbase
2008-12-09 06:35:27,559 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from
master: hbase.rootdir=hdfs://h1:54310/hbase
2008-12-09 06:35:27,731 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Log dir
hdfs://h1:54310/hbase/log_10.131.237.51_1228833326838_60020
2008-12-09 06:35:27,835 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer created
at /hbase/log_10.131.237.51_1228833326838_60020/hlog.dat.1228833327777
2008-12-09 06:35:27,908 INFO org.mortbay.util.Credential: Checking Resource aliases
2008-12-09 06:35:27,956 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4
2008-12-09 06:35:27,957 INFO org.mortbay.util.Container: Started HttpContext[/static,/static]
2008-12-09 06:35:27,957 INFO org.mortbay.util.Container: Started HttpContext[/logs,/logs]
2008-12-09 06:35:28,510 INFO org.mortbay.util.Container: Started org.mortbay.jetty.servlet.WebApplicationHandler@60ded0f0
2008-12-09 06:35:28,565 INFO org.mortbay.util.Container: Started WebApplicationContext[/,/]
2008-12-09 06:35:28,568 INFO org.mortbay.http.SocketListener: Started SocketListener on 0.0.0.0:60030
2008-12-09 06:35:28,568 INFO org.mortbay.util.Container: Started org.mortbay.jetty.Server@6490832e
2008-12-09 06:35:28,568 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2008-12-09 06:35:28,570 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 60020: starting
2008-12-09 06:35:28,570 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020:
starting
2008-12-09 06:35:28,577 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020:
starting
2008-12-09 06:35:28,578 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020:
starting
2008-12-09 06:35:28,578 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020:
starting
2008-12-09 06:35:28,578 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020:
starting
2008-12-09 06:35:28,578 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020:
starting
2008-12-09 06:35:28,579 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020:
starting
2008-12-09 06:35:28,579 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020:
starting
2008-12-09 06:35:28,579 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020:
starting
2008-12-09 06:35:28,579 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: HRegionServer
started at: 10.131.237.51:60020
2008-12-09 06:35:28,580 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020:
starting
2008-12-09 06:36:04,729 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
t1,,1228833363456
2008-12-09 06:36:04,730 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
t1,,1228833363456
2008-12-09 06:36:04,815 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED
=> 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true',
FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS
=> '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}
2008-12-09 06:36:04,840 DEBUG org.apache.hadoop.hbase.RegionHistorian: Onlined
2008-12-09 06:36:04,851 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region
t1,,1228833363456/2133923429
2008-12-09 06:36:04,881 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence
id for region t1,,1228833363456 is 0
2008-12-09 06:36:04,883 INFO org.apache.hadoop.hbase.regionserver.HRegion: region t1,,1228833363456/2133923429
available
2008-12-09 06:36:04,884 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction
requested for region: t1,,1228833363456
2008-12-09 06:36:04,884 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction
on region t1,,1228833363456
2008-12-09 06:36:04,886 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed
on region t1,,1228833363456 in 0sec
Tue Dec  9 16:37:46 EET 2008 Stopping regionserver
Tue Dec  9 16:37:46 EET 2008 Killing regionserver
2008-12-09 06:37:46,119 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Starting
shutdown thread.
2008-12-09 06:37:46,871 DEBUG org.apache.hadoop.hbase.RegionHistorian: Offlined
2008-12-09 06:37:46,871 INFO org.apache.hadoop.ipc.Server: Stopping server on 60020
2008-12-09 06:37:46,872 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on
60020
2008-12-09 06:37:46,872 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020:
exiting
2008-12-09 06:37:46,872 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020:
exiting
2008-12-09 06:37:46,872 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020:
exiting
2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020:
exiting
2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020:
exiting
2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020:
exiting
2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020:
exiting
2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020:
exiting
2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020:
exiting
2008-12-09 06:37:46,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020:
exiting
2008-12-09 06:37:46,874 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping
infoServer
2008-12-09 06:37:46,874 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2008-12-09 06:37:46,874 INFO org.mortbay.util.ThreadedServer: Stopping Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=60030]
2008-12-09 06:37:46,886 INFO org.mortbay.http.SocketListener: Stopped SocketListener on 0.0.0.0:60030
2008-12-09 06:37:46,948 INFO org.mortbay.util.Container: Stopped HttpContext[/static,/static]
2008-12-09 06:37:47,007 INFO org.mortbay.util.Container: Stopped HttpContext[/logs,/logs]
2008-12-09 06:37:47,007 INFO org.mortbay.util.Container: Stopped org.mortbay.jetty.servlet.WebApplicationHandler@60ded0f0
2008-12-09 06:37:47,094 INFO org.mortbay.util.Container: Stopped WebApplicationContext[/,/]
2008-12-09 06:37:47,094 INFO org.mortbay.util.Container: Stopped org.mortbay.jetty.Server@6490832e
2008-12-09 06:37:47,094 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: closing
region t1,,1228833363456
2008-12-09 06:37:47,094 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and
cache flushes disabled for region t1,,1228833363456
2008-12-09 06:37:47,094 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled
for region t1,,1228833363456
2008-12-09 06:37:47,094 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active
scanners for region t1,,1228833363456
2008-12-09 06:37:47,095 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled
for region t1,,1228833363456
2008-12-09 06:37:47,095 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks
outstanding on region t1,,1228833363456
2008-12-09 06:37:47,095 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache
flush for region t1,,1228833363456. Current region memcache size 18.0
2008-12-09 06:37:47,095 INFO org.apache.hadoop.hbase.regionserver.Flusher: regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher
exiting
2008-12-09 06:37:47,096 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
2008-12-09 06:37:47,096 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: regionserver/0:0:0:0:0:0:0:0:60020.compactor
exiting
2008-12-09 06:37:47,099 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: error closing
region t1,,1228833363456
org.apache.hadoop.hbase.DroppedSnapshotException: region: t1,,1228833363456
    at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1071)
    at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:619)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.closeAllRegions(HRegionServer.java:951)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:459)
    at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Filesystem closed
    at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:196)
    at org.apache.hadoop.dfs.DFSClient.getFileInfo(DFSClient.java:564)
    at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:390)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:667)
    at org.apache.hadoop.hbase.regionserver.HStoreFile.<init>(HStoreFile.java:152)
    at org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:599)
    at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:577)
    at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1058)
    ... 4 more
2008-12-09 06:37:47,100 DEBUG org.apache.hadoop.hbase.regionserver.HLog: closing log writer
in hdfs://h1:54310/hbase/log_10.131.237.51_1228833326838_60020
2008-12-09 06:37:47,101 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Close and
delete failed
java.io.IOException: Filesystem closed
    at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:196)
    at org.apache.hadoop.dfs.DFSClient.access$600(DFSClient.java:59)
    at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:2689)
    at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:2655)
    at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:59)
    at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:79)
    at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:962)
    at org.apache.hadoop.hbase.regionserver.HLog.close(HLog.java:349)
    at org.apache.hadoop.hbase.regionserver.HLog.closeAndDelete(HLog.java:333)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:461)
    at java.lang.Thread.run(Thread.java:619)
2008-12-09 06:37:47,102 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: telling master
that region server is shutting down at: 10.131.237.51:60020
2008-12-09 06:37:47,104 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: stopping
server at: 10.131.237.51:60020
2008-12-09 06:37:47,882 INFO org.apache.hadoop.hbase.Leases: regionserver/0:0:0:0:0:0:0:0:60020.leaseChecker
closing leases
2008-12-09 06:37:47,882 INFO org.apache.hadoop.hbase.Leases: regionserver/0:0:0:0:0:0:0:0:60020.leaseChecker
closed leases
2008-12-09 06:37:54,919 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread
exiting
2008-12-09 06:37:54,920 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/0:0:0:0:0:0:0:0:60020
exiting
2008-12-09 06:37:54,920 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown
thread complete










Second HRegion Server:

2008-12-09 06:35:28,710 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: HRegionServer
started at: 10.131.237.52:60020
2008-12-09 06:35:28,711 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020:
starting
2008-12-09 06:37:49,965 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
t1,,1228833363456
2008-12-09 06:37:49,966 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
t1,,1228833363456
2008-12-09 06:37:50,055 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED
=> 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true',
FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS
=> '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}
2008-12-09 06:37:50,069 DEBUG org.apache.hadoop.hbase.RegionHistorian: Onlined
2008-12-09 06:37:50,080 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region
t1,,1228833363456/2133923429
2008-12-09 06:37:50,110 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence
id for region t1,,1228833363456 is 0
2008-12-09 06:37:50,112 INFO org.apache.hadoop.hbase.regionserver.HRegion: region t1,,1228833363456/2133923429
available
2008-12-09 06:37:50,112 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction
requested for region: t1,,1228833363456
2008-12-09 06:37:50,114 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction
on region t1,,1228833363456
2008-12-09 06:37:50,118 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed
on region t1,,1228833363456 in 0sec












HMaster Server:

java version "1.6.0_06"
Java(TM) SE Runtime Environment (build 1.6.0_06-b02)
Java HotSpot(TM) 64-Bit Server VM (build 10.0-b22, mixed mode)
ulimit -n 32768
2008-12-09 06:35:24,944 INFO org.apache.hadoop.hbase.master.HMaster: Root region dir: hdfs://h1:54310/hbase/-ROOT-/70236052
2008-12-09 06:35:25,109 INFO org.apache.hadoop.hbase.master.HMaster: BOOTSTRAP: creating ROOT
and first META regions
2008-12-09 06:35:25,156 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer created
at /hbase/-ROOT-/70236052/log/hlog.dat.1228833325121
2008-12-09 06:35:25,160 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region
-ROOT-,,0/70236052
2008-12-09 06:35:25,197 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence
id for region -ROOT-,,0 is 0
2008-12-09 06:35:25,200 INFO org.apache.hadoop.hbase.regionserver.HRegion: region -ROOT-,,0/70236052
available
2008-12-09 06:35:25,219 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer created
at /hbase/.META./1028785192/log/hlog.dat.1228833325214
2008-12-09 06:35:25,220 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region
.META.,,1/1028785192
2008-12-09 06:35:25,252 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence
id for region .META.,,1 is 0
2008-12-09 06:35:25,256 INFO org.apache.hadoop.hbase.regionserver.HRegion: region .META.,,1/1028785192
available
2008-12-09 06:35:25,260 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and
cache flushes disabled for region -ROOT-,,0
2008-12-09 06:35:25,260 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled
for region -ROOT-,,0
2008-12-09 06:35:25,260 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active
scanners for region -ROOT-,,0
2008-12-09 06:35:25,260 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled
for region -ROOT-,,0
2008-12-09 06:35:25,260 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks
outstanding on region -ROOT-,,0
2008-12-09 06:35:25,261 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache
flush for region -ROOT-,,0. Current region memcache size 445.0
2008-12-09 06:35:25,297 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop
library
2008-12-09 06:35:25,299 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully
loaded & initialized native-zlib library
2008-12-09 06:35:25,301 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor
2008-12-09 06:35:25,555 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
2008-12-09 06:35:25,556 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
2008-12-09 06:35:25,556 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
2008-12-09 06:35:25,556 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
2008-12-09 06:35:25,570 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/-ROOT-/70236052/info/mapfiles/2955877428110101962
with 1 entries, sequence id 1, data size 445.0, file size 569.0
2008-12-09 06:35:25,570 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache
flush for region -ROOT-,,0 in 310ms, sequence id=1, compaction requested=false
2008-12-09 06:35:25,570 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 70236052/info
2008-12-09 06:35:25,570 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed -ROOT-,,0
2008-12-09 06:35:25,570 DEBUG org.apache.hadoop.hbase.regionserver.HLog: closing log writer
in hdfs://h1:54310/hbase/-ROOT-/70236052/log
2008-12-09 06:35:25,590 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and
cache flushes disabled for region .META.,,1
2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled
for region .META.,,1
2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active
scanners for region .META.,,1
2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled
for region .META.,,1
2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks
outstanding on region .META.,,1
2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1028785192/historian
2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1028785192/info
2008-12-09 06:35:25,591 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed .META.,,1
2008-12-09 06:35:25,591 DEBUG org.apache.hadoop.hbase.regionserver.HLog: closing log writer
in hdfs://h1:54310/hbase/.META./1028785192/log
2008-12-09 06:35:25,917 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics
with hostName=HMaster, port=60000
2008-12-09 06:35:26,088 INFO org.apache.hadoop.hbase.master.HMaster: HMaster initialized on
10.131.237.52:60000
2008-12-09 06:35:26,152 INFO org.mortbay.util.Credential: Checking Resource aliases
2008-12-09 06:35:26,208 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4
2008-12-09 06:35:26,209 INFO org.mortbay.util.Container: Started HttpContext[/static,/static]
2008-12-09 06:35:26,209 INFO org.mortbay.util.Container: Started HttpContext[/logs,/logs]
2008-12-09 06:35:27,051 INFO org.mortbay.util.Container: Started org.mortbay.jetty.servlet.WebApplicationHandler@741827d1
2008-12-09 06:35:27,133 INFO org.mortbay.util.Container: Started WebApplicationContext[/,/]
2008-12-09 06:35:27,493 INFO org.mortbay.util.Container: Started org.mortbay.jetty.servlet.WebApplicationHandler@25e222e
2008-12-09 06:35:27,495 INFO org.mortbay.util.Container: Started WebApplicationContext[/api,rest]
2008-12-09 06:35:27,498 INFO org.mortbay.http.SocketListener: Started SocketListener on 0.0.0.0:60010
2008-12-09 06:35:27,498 INFO org.mortbay.util.Container: Started org.mortbay.jetty.Server@6632060c
2008-12-09 06:35:27,498 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2008-12-09 06:35:27,499 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 60000: starting
2008-12-09 06:35:27,501 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60000:
starting
2008-12-09 06:35:27,501 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60000:
starting
2008-12-09 06:35:27,501 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60000:
starting
2008-12-09 06:35:27,501 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000:
starting
2008-12-09 06:35:27,501 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60000:
starting
2008-12-09 06:35:27,502 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60000:
starting
2008-12-09 06:35:27,505 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60000:
starting
2008-12-09 06:35:27,507 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60000:
starting
2008-12-09 06:35:27,522 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60000:
starting
2008-12-09 06:35:27,522 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60000:
starting
2008-12-09 06:35:27,523 DEBUG org.apache.hadoop.hbase.master.HMaster: Started service threads
2008-12-09 06:35:27,529 INFO org.apache.hadoop.hbase.master.ServerManager: Received start
message from: 10.131.237.54:60020
2008-12-09 06:35:27,535 INFO org.apache.hadoop.hbase.master.ServerManager: Received start
message from: 10.131.237.51:60020
2008-12-09 06:35:27,538 INFO org.apache.hadoop.hbase.master.ServerManager: Received start
message from: 10.131.237.53:60020
2008-12-09 06:35:27,705 INFO org.apache.hadoop.hbase.master.ServerManager: Received start
message from: 10.131.237.52:60020
2008-12-09 06:35:28,648 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
-ROOT-,,0 to server 10.131.237.54:60020
2008-12-09 06:35:28,651 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 0,
Num Servers: 4, Avg Load: 0.0
2008-12-09 06:35:31,673 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
-ROOT-,,0 from 10.131.237.54:60020
2008-12-09 06:35:31,675 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020}
2008-12-09 06:35:31,769 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,
TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES =>
[{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE',
LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'},
{NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION
=> 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}}, SERVER => '', STARTCODE => 0
2008-12-09 06:35:31,769 DEBUG org.apache.hadoop.hbase.master.BaseScanner: Current assignment
of .META.,,1 is not valid: serverInfo: null, passed startCode: 0, storedInfo.startCode: -1,
unassignedRegions: false, pendingRegions: false
2008-12-09 06:35:31,772 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020}
complete
2008-12-09 06:35:34,661 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
.META.,,1 to server 10.131.237.53:60020
2008-12-09 06:35:36,721 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED
=> 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true',
FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS
=> '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}
2008-12-09 06:35:36,730 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException:
No server address listed in -ROOT- for region .META.,,1>. Retrying after sleep of 2000
2008-12-09 06:35:36,734 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED
=> 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true',
FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS
=> '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}
2008-12-09 06:35:37,685 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
.META.,,1 from 10.131.237.53:60020
2008-12-09 06:35:37,685 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
.META.,,1 from 10.131.237.53:60020
2008-12-09 06:35:37,689 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop:
PendingOpenOperation from 10.131.237.53:60020
2008-12-09 06:35:37,691 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: .META.,,1
open on 10.131.237.53:60020
2008-12-09 06:35:37,691 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row .META.,,1 in region -ROOT-,,0 with startcode 1228833326870 and server 10.131.237.53:60020
2008-12-09 06:35:38,744 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED
=> 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true',
FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS
=> '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}
2008-12-09 06:35:38,754 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit in table locations for row <> and tableName .META.: location server 10.131.237.53:60020,
location region name .META.,,1
2008-12-09 06:35:38,754 DEBUG org.apache.hadoop.hbase.RegionHistorian: Onlined
2008-12-09 06:35:38,755 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit in table locations for row <> and tableName .META.: location server 10.131.237.53:60020,
location region name .META.,,1
2008-12-09 06:35:38,755 DEBUG org.apache.hadoop.hbase.master.ProcessRegionOpen$1: Adding to
onlineMetaRegions: {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020}
2008-12-09 06:35:38,758 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020}
2008-12-09 06:35:38,831 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scan of meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020}
complete
2008-12-09 06:35:38,831 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions
scanned
2008-12-09 06:35:38,931 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit in table locations for row <> and tableName .META.: location server 10.131.237.53:60020,
location region name .META.,,1
2008-12-09 06:35:39,697 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit in table locations for row <> and tableName .META.: location server 10.131.237.53:60020,
location region name .META.,,1
2008-12-09 06:35:40,435 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit in table locations for row <> and tableName .META.: location server 10.131.237.53:60020,
location region name .META.,,1
2008-12-09 06:35:41,223 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit in table locations for row <> and tableName .META.: location server 10.131.237.53:60020,
location region name .META.,,1
2008-12-09 06:35:43,673 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 2,
Num Servers: 4, Avg Load: 1.0
2008-12-09 06:35:58,697 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 2,
Num Servers: 4, Avg Load: 1.0
2008-12-09 06:36:03,477 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer created
at /hbase/t1/2133923429/log/hlog.dat.1228833363473
2008-12-09 06:36:03,478 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region
t1,,1228833363456/2133923429
2008-12-09 06:36:03,489 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence
id for region t1,,1228833363456 is 0
2008-12-09 06:36:03,491 INFO org.apache.hadoop.hbase.regionserver.HRegion: region t1,,1228833363456/2133923429
available
2008-12-09 06:36:03,494 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Compactions and
cache flushes disabled for region t1,,1228833363456
2008-12-09 06:36:03,494 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Scanners disabled
for region t1,,1228833363456
2008-12-09 06:36:03,494 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more active
scanners for region t1,,1228833363456
2008-12-09 06:36:03,494 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled
for region t1,,1228833363456
2008-12-09 06:36:03,494 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks
outstanding on region t1,,1228833363456
2008-12-09 06:36:03,494 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 2133923429/f1
2008-12-09 06:36:03,495 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed t1,,1228833363456
2008-12-09 06:36:03,495 DEBUG org.apache.hadoop.hbase.regionserver.HLog: closing log writer
in hdfs://h1:54310/hbase/t1/2133923429/log
2008-12-09 06:36:03,521 INFO org.apache.hadoop.hbase.master.HMaster: created table t1
2008-12-09 06:36:04,705 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
t1,,1228833363456 to server 10.131.237.51:60020
2008-12-09 06:36:07,726 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
t1,,1228833363456 from 10.131.237.51:60020
2008-12-09 06:36:07,727 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop:
PendingOpenOperation from 10.131.237.51:60020
2008-12-09 06:36:07,727 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: t1,,1228833363456
open on 10.131.237.51:60020
2008-12-09 06:36:07,727 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions:
1, onlineMetaRegions.size(): 1
2008-12-09 06:36:07,727 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row t1,,1228833363456 in region .META.,,1 with startcode 1228833326838 and server 10.131.237.51:60020
2008-12-09 06:36:11,335 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit in table locations for row <> and tableName .META.: location server 10.131.237.53:60020,
location region name .META.,,1
2008-12-09 06:36:13,735 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 4, Avg Load: 1.0
2008-12-09 06:36:24,695 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit in table locations for row <> and tableName .META.: location server 10.131.237.53:60020,
location region name .META.,,1
2008-12-09 06:36:28,756 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 4, Avg Load: 1.0
2008-12-09 06:36:31,782 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020}
2008-12-09 06:36:31,815 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,
TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES =>
[{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE',
LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'},
{NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION
=> 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}}, SERVER => '10.131.237.53:60020', STARTCODE => 1228833326870
2008-12-09 06:36:31,817 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020}
complete
2008-12-09 06:36:38,766 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020}
2008-12-09 06:36:38,773 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
REGION => {NAME => 't1,,1228833363456', STARTKEY => '', ENDKEY => '', ENCODED
=> 2133923429, TABLE => {{NAME => 't1', IS_ROOT => 'false', IS_META => 'false',
FAMILIES => [{NAME => 'f1', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS
=> '3', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}}, SERVER => '10.131.237.51:60020', STARTCODE => 1228833326838
2008-12-09 06:36:38,775 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scan of meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020}
complete
2008-12-09 06:36:38,775 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions
scanned
2008-12-09 06:36:43,777 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 4, Avg Load: 1.0
2008-12-09 06:36:56,635 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020,
location region name t1,,1228833363456
2008-12-09 06:36:58,801 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 4, Avg Load: 1.0
2008-12-09 06:37:13,820 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 4, Avg Load: 1.0
2008-12-09 06:37:26,769 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020,
location region name t1,,1228833363456
2008-12-09 06:37:28,841 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 4, Avg Load: 1.0
2008-12-09 06:37:31,792 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020}
2008-12-09 06:37:31,835 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,
TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES =>
[{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE',
LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'},
{NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION
=> 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}}, SERVER => '10.131.237.53:60020', STARTCODE => 1228833326870
2008-12-09 06:37:31,837 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020}
complete
2008-12-09 06:37:38,776 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020}
2008-12-09 06:37:38,786 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
REGION => {NAME => 't1,,1228833363456', STARTKEY => '', ENDKEY => '', ENCODED
=> 2133923429, TABLE => {{NAME => 't1', IS_ROOT => 'false', IS_META => 'false',
FAMILIES => [{NAME => 'f1', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS
=> '3', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}}, SERVER => '10.131.237.51:60020', STARTCODE => 1228833326838
2008-12-09 06:37:38,789 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scan of meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020}
complete
2008-12-09 06:37:38,789 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions
scanned
2008-12-09 06:37:43,860 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 4, Avg Load: 1.0
2008-12-09 06:37:47,095 INFO org.apache.hadoop.hbase.master.ServerManager: Cancelling lease
for 10.131.237.51:60020
2008-12-09 06:37:47,095 INFO org.apache.hadoop.hbase.master.ServerManager: Region server 10.131.237.51:60020:
MSG_REPORT_EXITING -- lease cancelled
2008-12-09 06:37:47,096 INFO org.apache.hadoop.hbase.master.ServerManager: Processing MSG_REPORT_CLOSE:
t1,,1228833363456 from 10.131.237.51:60020
2008-12-09 06:37:49,949 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
t1,,1228833363456 to server 10.131.237.52:60020
2008-12-09 06:37:52,974 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
t1,,1228833363456 from 10.131.237.52:60020
2008-12-09 06:37:52,974 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
t1,,1228833363456 from 10.131.237.52:60020
2008-12-09 06:37:52,975 DEBUG org.apache.hadoop.hbase.master.HMaster: Main processing loop:
PendingOpenOperation from 10.131.237.52:60020
2008-12-09 06:37:52,975 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: t1,,1228833363456
open on 10.131.237.52:60020
2008-12-09 06:37:52,975 DEBUG org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions:
1, onlineMetaRegions.size(): 1
2008-12-09 06:37:52,975 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row t1,,1228833363456 in region .META.,,1 with startcode 1228833327601 and server 10.131.237.52:60020
2008-12-09 06:37:56,912 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020,
location region name t1,,1228833363456
2008-12-09 06:37:58,877 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 3, Avg Load: 1.0
2008-12-09 06:38:13,894 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 3, Avg Load: 1.0
2008-12-09 06:38:27,035 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020,
location region name t1,,1228833363456
2008-12-09 06:38:28,914 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 3, Avg Load: 1.0
2008-12-09 06:38:31,802 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020}
2008-12-09 06:38:31,833 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,
TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES =>
[{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE',
LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'},
{NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION
=> 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}}, SERVER => '10.131.237.53:60020', STARTCODE => 1228833326870
2008-12-09 06:38:31,835 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020}
complete
2008-12-09 06:38:38,787 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020}
2008-12-09 06:38:38,792 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
REGION => {NAME => 't1,,1228833363456', STARTKEY => '', ENDKEY => '', ENCODED
=> 2133923429, TABLE => {{NAME => 't1', IS_ROOT => 'false', IS_META => 'false',
FAMILIES => [{NAME => 'f1', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS
=> '3', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}}, SERVER => '10.131.237.52:60020', STARTCODE => 1228833327601
2008-12-09 06:38:38,794 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scan of meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020}
complete
2008-12-09 06:38:38,794 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions
scanned
2008-12-09 06:38:43,930 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 3, Avg Load: 1.0
2008-12-09 06:38:57,168 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020,
location region name t1,,1228833363456
2008-12-09 06:38:58,950 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 3, Avg Load: 1.0
2008-12-09 06:39:13,970 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 3, Avg Load: 1.0
2008-12-09 06:39:27,310 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020,
location region name t1,,1228833363456
2008-12-09 06:39:28,996 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 3, Avg Load: 1.0
2008-12-09 06:39:31,813 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020}
2008-12-09 06:39:31,844 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,
TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES =>
[{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE',
LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'},
{NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION
=> 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}}, SERVER => '10.131.237.53:60020', STARTCODE => 1228833326870
2008-12-09 06:39:31,846 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020}
complete
2008-12-09 06:39:38,797 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020}
2008-12-09 06:39:38,806 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
REGION => {NAME => 't1,,1228833363456', STARTKEY => '', ENDKEY => '', ENCODED
=> 2133923429, TABLE => {{NAME => 't1', IS_ROOT => 'false', IS_META => 'false',
FAMILIES => [{NAME => 'f1', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS
=> '3', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}}, SERVER => '10.131.237.52:60020', STARTCODE => 1228833327601
2008-12-09 06:39:38,808 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scan of meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020}
complete
2008-12-09 06:39:38,808 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions
scanned
2008-12-09 06:39:44,016 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 3, Avg Load: 1.0
2008-12-09 06:39:57,462 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020,
location region name t1,,1228833363456
2008-12-09 06:39:58,440 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020,
location region name t1,,1228833363456
2008-12-09 06:39:59,036 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 3, Avg Load: 1.0
2008-12-09 06:40:14,056 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 3, Avg Load: 1.0
2008-12-09 06:40:28,547 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020,
location region name t1,,1228833363456
2008-12-09 06:40:29,075 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 3, Avg Load: 1.0
2008-12-09 06:40:31,823 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020}
2008-12-09 06:40:31,849 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,
TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES =>
[{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE',
LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'},
{NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION
=> 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}}, SERVER => '10.131.237.53:60020', STARTCODE => 1228833326870
2008-12-09 06:40:31,851 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.131.237.54:60020}
complete
2008-12-09 06:40:38,808 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020}
2008-12-09 06:40:38,812 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
REGION => {NAME => 't1,,1228833363456', STARTKEY => '', ENDKEY => '', ENCODED
=> 2133923429, TABLE => {{NAME => 't1', IS_ROOT => 'false', IS_META => 'false',
FAMILIES => [{NAME => 'f1', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS
=> '3', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}}, SERVER => '10.131.237.52:60020', STARTCODE => 1228833327601
2008-12-09 06:40:38,814 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scan of meta region {regionname: .META.,,1, startKey: <>, server: 10.131.237.53:60020}
complete
2008-12-09 06:40:38,814 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions
scanned
2008-12-09 06:40:44,096 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 3, Avg Load: 1.0
2008-12-09 06:40:58,661 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit in table locations for row <> and tableName t1: location server 10.131.237.51:60020,
location region name t1,,1228833363456
2008-12-09 06:40:59,115 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load: 3,
Num Servers: 3, Avg Load: 1.0



Mime
View raw message