hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ryan rawson (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1560) TIF (and other clients?) cant seem to find one region (getClosestRowBefore issue?)
Date Mon, 22 Jun 2009 03:30:07 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722466#action_12722466
] 

ryan rawson commented on HBASE-1560:
------------------------------------

this looks like a TIF issue:

2009-06-21 20:18:31,590 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Got ZooKeeper event, state: SyncConnected, type: None, path: null
2009-06-21 20:18:31,608 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
/hbase/root-region-server got 10.20.20.165:60020
2009-06-21 20:18:31,646 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Found ROOT at 10.20.20.165:60020
2009-06-21 20:18:31,656 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Creating
scanner over table_name starting at key '^@^R^?c^@^@^A^V�v�@^@B��'
2009-06-21 20:18:31,656 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing
internal scanner to startKey at '^@^R^?c^@^@^A^V�v�@^@B��'
2009-06-21 20:18:31,671 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 1
2009-06-21 20:18:31,679 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
2009-06-21 20:18:31,801 INFO org.apache.hadoop.mapred.MapTask: data buffer = 79691776/99614720
2009-06-21 20:18:31,801 INFO org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680
2009-06-21 20:19:31,874 DEBUG org.apache.hadoop.hbase.mapred.TableInputFormatBase: recovered
from org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScan
nerException: 7976142196877173570
        at org.apache.hadoop.hbase.regionserver.HRegionServer.close(HRegionServer.java:1905)
        at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:643)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:928)
        at org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1809)
        at org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1904)
        at org.apache.hadoop.hbase.mapred.TableInputFormatBase$TableRecordReader.next(TableInputFormatBase.java:219)
        at org.apache.hadoop.hbase.mapred.TableInputFormatBase$TableRecordReader.next(TableInputFormatBase.java:90)
        at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:191)
        at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:175)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:356)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)

2009-06-21 20:19:31,874 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Creating
scanner over table_name starting at key 'null'
2009-06-21 20:19:31,874 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing
internal scanner to startKey at 'null'
2009-06-21 20:22:07,895 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server
null for region , row '', but failed after 10 attempts.
Exceptions:
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
(elided, same as before)

Looks like the problem is in TIF & scan client code, since if you give 'null' as start
key, it will fail in this ugly way.

But there seems to be more problems in TIF, while this one failed after the server threw a
scanner issue, in the 'successful' version the logfile is also problematic:

2009-06-21 20:18:31,637 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Creating
scanner over table_name starting at key ''
2009-06-21 20:18:31,637 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing
internal scanner to startKey at ''
2009-06-21 20:18:31,637 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit for row <> in tableName table_name: location server 10.20.20.155:60020, location
region name table_name,,1245570713261
2009-06-21 20:18:31,647 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 1
2009-06-21 20:18:31,665 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
2009-06-21 20:18:31,786 INFO org.apache.hadoop.mapred.MapTask: data buffer = 79691776/99614720
2009-06-21 20:18:31,786 INFO org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680
2009-06-21 20:19:06,664 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing
forward from region REGION => {NAME => 'table_name,,1245570713261', STARTKEY => '',
ENDKEY => '\x00\x02\x13\x88\x00\x00\x01\x05\xB6\xBFB\xA0\x00\x05B\xE0', ENCODED => 497118635,
TABLE => {{NAME => 'table_name', MEMCACHE_FLUSHSIZE => '67108864', MAX_FILESIZE =>
'536870912', READONLY => 'false', FAMILIES => [{NAME => 'default', COMPRESSION =>
'LZO', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY =>
'false', BLOCKCACHE => 'true'}]}}
2009-06-21 20:19:06,665 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing
internal scanner to startKey at '^@^B^S�^@^@^A^E��B�^@^EB�'
2009-06-21 20:19:06,677 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing
forward from region REGION => {NAME => 'table_name,\x00\x02\x13\x88\x00\x00\x01\x05\xB6\xBFB\xA0\x00\x05B\xE0,1245570713261',
STARTKEY => '\x00\x02\x13\x88\x00\x00\x01\x05\xB6\xBFB\xA0\x00\x05B\xE0', ENDKEY =>
'\x00\x03\xF6\x5C\x00\x00\x01\x05\xD5\x27d\xE8\x00\x09eQ', ENCODED => 1054973557, TABLE
=> {{NAME => 'table_name', MEMCACHE_FLUSHSIZE => '67108864', MAX_FILESIZE => '536870912',
READONLY => 'false', FAMILIES => [{NAME => 'default', COMPRESSION => 'LZO', VERSIONS
=> '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE
=> 'true'}]}}
2009-06-21 20:19:06,677 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing
internal scanner to startKey at '^@^C�\^@^@^A^E�'d�^@      eQ'
2009-06-21 20:19:06,689 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing
forward from region REGION => {NAME => 'table_name,\x00\x03\xF6\x5C\x00\x00\x01\x05\xD5\x27d\xE8\x00\x09eQ,1245570960707',
STARTKEY => '\x00\x03\xF6\x5C\x00\x00\x01\x05\xD5\x27d\xE8\x00\x09eQ', ENDKEY => '\x00\x05\xF0\x9F\x00\x00\x01\x03\x7D\xBB\x858\x00\x04\x7B\xDE',
ENCODED => 171207314, TABLE => {{NAME => 'table_name', MEMCACHE_FLUSHSIZE => '67108864',
MAX_FILESIZE => '536870912', READONLY => 'false', FAMILIES => [{NAME => 'default',
COMPRESSION => 'LZO', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}

Even though the scanner starts at '' it seems to continue to more than 1 region, which is
not supposed to happen since # of mappers = # of regions.

Other mappers indicate the same issue - going past the end of the region they were assigned
to.

Very mysterious problem here!

> TIF (and other clients?) cant seem to find one region (getClosestRowBefore issue?)
> ----------------------------------------------------------------------------------
>
>                 Key: HBASE-1560
>                 URL: https://issues.apache.org/jira/browse/HBASE-1560
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: ryan rawson
>            Priority: Blocker
>             Fix For: 0.20.0
>
>
> running a full TIF-mr on a table, it eventually fails, all on 1 of the splits, and all
with the same exception set, which is:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server
null for region , row '', but failed after 10 attempts.
> Exceptions:
> java.lang.NullPointerException
> java.lang.NullPointerException
> java.lang.NullPointerException
> java.lang.NullPointerException
> java.lang.NullPointerException
> java.lang.NullPointerException
> java.lang.NullPointerException
> java.lang.NullPointerException
> java.lang.NullPointerException
> java.lang.NullPointerException
> 	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:935)
> 	at org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1842)
> 	at org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable.java:1790)
> 	at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:369)
> 	at org.apache.hadoop.hbase.mapred.TableInputFormatBase$TableRecordReader.restart(TableInputFormatBase.java:121)
> 	at org.apache.hadoop.hbase.mapred.TableInputFormatBase$TableRecordReader.next(TableInputFormatBase.java:222)
> 	at org.apache.hadoop.hbase.mapred.TableInputFormatBase$TableRecordReader.next(TableInputFormatBase.java:90)
> 	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:191)
> 	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:175)
> 	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> 	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:356)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Suspicion: We can't locate the 'root' region with key '' or null.  Probably an issue
with getClosestRowBefore.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message