hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stanislaw Kogut <sko...@sistyma.net>
Subject HBase 0.20.5 issues
Date Tue, 29 Jun 2010 13:21:01 GMT
Hi everyone!

Has someone noticed same behaviour of hbase-0.20.5 after upgrade from
0.20.3?

$hadoop jar hbase/hbase-0.20.5-test.jar sequentialWrite 1
10/06/29 16:03:21 INFO zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.2.2-888565, built on 12/08/2009 21:51 GMT
10/06/29 16:03:21 INFO zookeeper.ZooKeeper: Client environment:host.name
=se002.cluster.local
10/06/29 16:03:21 INFO zookeeper.ZooKeeper: Client
environment:java.version=1.6.0_20
10/06/29 16:03:21 INFO zookeeper.ZooKeeper: Client
environment:java.vendor=Sun Microsystems Inc.
10/06/29 16:03:21 INFO zookeeper.ZooKeeper: Client
environment:java.home=/usr/java/jdk1.6.0_20/jre
10/06/29 16:03:21 INFO zookeeper.ZooKeeper: Client
environment:java.class.path=/opt/hadoop/common/bin/../conf:/usr/java/latest/lib/tools.jar:/opt/hadoop/common/bin/..:/opt/hadoop/common/bin/../hadoop-0.20.2-core.jar:/opt/hadoop/common/bin/../lib/commons-cli-1.2.jar:/opt/hadoop/common/bin/../lib/commons-codec-1.3.jar:/opt/hadoop/common/bin/../lib/commons-el-1.0.jar:/opt/hadoop/common/bin/../lib/commons-httpclient-3.0.1.jar:/opt/hadoop/common/bin/../lib/commons-logging-1.0.4.jar:/opt/hadoop/common/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/common/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/common/bin/../lib/core-3.1.1.jar:/opt/hadoop/common/bin/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/common/bin/../lib/jasper-compiler-5.5.12.jar:/opt/hadoop/common/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/common/bin/../lib/jets3t-0.6.1.jar:/opt/hadoop/common/bin/../lib/jetty-6.1.14.jar:/opt/hadoop/common/bin/../lib/jetty-util-6.1.14.jar:/opt/hadoop/common/bin/../lib/junit-3.8.1.jar:/opt/hadoop/common/bin/../lib/kfs-0.2.2.jar:/opt/hadoop/common/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/common/bin/../lib/mockito-all-1.8.0.jar:/opt/hadoop/common/bin/../lib/oro-2.0.8.jar:/opt/hadoop/common/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/hadoop/common/bin/../lib/slf4j-api-1.4.3.jar:/opt/hadoop/common/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/common/bin/../lib/xmlenc-0.52.jar:/opt/hadoop/common/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/common/bin/../lib/jsp-2.1/jsp-api-2.1.jar:/opt/hadoop/hbase/lib/zookeeper-3.2.2.jar:/opt/hadoop/hbase/conf:/opt/hadoop/hbase/hbase-0.20.5.jar
10/06/29 16:03:22 INFO hbase.PerformanceEvaluation: Table {NAME =>
'TestTable', FAMILIES => [{NAME => 'info', COMPRESSION => 'NONE', VERSIONS
=> '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false',
BLOCKCACHE => 'true'}]} created
10/06/29 16:03:22 INFO hbase.PerformanceEvaluation: Start class
org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest at offset
0 for 1048576 rows
10/06/29 16:03:37 INFO hbase.PerformanceEvaluation: 0/104857/1048576
10/06/29 16:03:52 INFO hbase.PerformanceEvaluation: 0/209714/1048576
10/06/29 16:04:09 INFO hbase.PerformanceEvaluation: 0/314571/1048576
10/06/29 16:04:27 INFO hbase.PerformanceEvaluation: 0/419428/1048576
10/06/29 16:06:06 ERROR hbase.PerformanceEvaluation: Failed
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server Some server, retryOnlyOne=true, index=0, islastrow=false,
tries=9, numtries=10, i=0, listsize=9650, region=TestTable,,1277816601856
for region TestTable,,1277816601856, row '0000511450', but failed after 10
attempts.
Exceptions:

    at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1149)
    at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1230)
    at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:666)
    at
org.apache.hadoop.hbase.PerformanceEvaluation$Test.testTakedown(PerformanceEvaluation.java:621)
    at
org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:637)
    at
org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:889)
    at
org.apache.hadoop.hbase.PerformanceEvaluation.runNIsOne(PerformanceEvaluation.java:907)
    at
org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:939)
    at
org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:1036)
    at
org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:1061)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

Looks like it happens on region splits.

Also, some other strange things:
1. After writing something to TestTable, some regionservers log these:
2010-06-29 16:05:06,458 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
.META.,,1

After that, there comes another .META. region on this server in 'status
detailed' output.
Even more, sometimes it opens not only .META., but also other regions from
data tables, such as these TestTable from performanceEvaluation.

2. After disabling and removing table its regions still look like assigned
to regionservers in 'status detailed'.

3. After 3-4 tries to write into TestTable from performanceEvaluation there
another strange thing:
10/06/29 16:01:59 ERROR hbase.PerformanceEvaluation: Failed
org.apache.hadoop.hbase.TableExistsException:
org.apache.hadoop.hbase.TableExistsException: TestTable

but, table not exists. You cannot disable and drop it, and hbase shells not
lists it in 'list' output. But you also cannot create it, because it is
"exists", and it's regions are assigned to regionservers. Note, nobody drops
this table.

I spent some time to find out why this all happens, trying to play around
hadoop cluster versions (first it was cloudera, than "vanilla" 0.20.2), but
still having this issue. So, I will hope, if someone help to find cause for
this.

-- 
Regards,
Stanislaw Kogut

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message