hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1603) MR failed "RetriesExhaustedException: Trying to contact region server Some server for region TestTable..."
Date Mon, 06 Jul 2009 22:31:14 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727817#action_12727817
] 

stack commented on HBASE-1603:
------------------------------

Looks like we are making incorrect splits:

{code}
2009-07-06 21:29:00,591 [regionserver/208.76.44.141:60020.compactor] INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread:
region split, META updated, and report to master all successful. Old region=REGION => {NAME
=> 'TestTable,\x00\x04\x05\x02\x01\x04\x09
\x09\x05\x03,1246915671694', STARTKEY => '\x00\x04\x05\x02\x01\x04\x09\x09\x05\x03', ENDKEY
=> '', ENCODED => 532846028, OFFLINE => true, SPLIT => true, TABLE => {{NAME
=> 'TestTable', FAMILIES => [{NAME => 'info', COMPRESSION => 'NONE', VERSIONS
=> '3', TTL => '21
47483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}},
new regions: TestTable,\x00\x04\x05\x02\x01\x04\x09\x09\x05\x03,1246915729181, TestTable,\x00\x03\x09\x02\x04\x09\x04\x08\x01\x04,1246915729181.
Split took 11sec
{code}

It would seem we're making split with a key that is less than that of the parent region (if
the binary output can be trusted);

TestTable,\x00\x03\x09\x02\x04\x09\x04\x08\x01\x04,1246915729181 is <
TestTable,\x00\x04\x05\x02\x01\x04\x09\x09\x05\x03,1246915671694

@Andrew -- I'm looking for Andrew Lee issue...

> MR failed "RetriesExhaustedException: Trying to contact region server Some server for
region TestTable..."
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1603
>                 URL: https://issues.apache.org/jira/browse/HBASE-1603
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>
> Here is the master.  Region TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246462685358
was split at 16:11:42,865.  My MR job failed at 18:12:26,462 with this:
> {code}
> 2009-07-01 18:12:26,462 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server
Some server for region TestTable,�,1246464670313, row '��		', but failed after 10 attempts.
> Exceptions:
> ...
> {code}
> Why after ten attempts did the client not find the region?
> {code}
> 2009-07-01 16:11:42,865 [IPC Server handler 2 on 60001] INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_SPLIT: TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246462685358:
Daughters; TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313, TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313
from aa0-000-15.u.powerset.com,60020,1246461673026; 1 of 3
> 2009-07-01 16:11:42,866 [IPC Server handler 2 on 60001] INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 to aa0-000-15.u.powerset.com,60020,1246461673026
> 2009-07-01 16:11:42,866 [IPC Server handler 2 on 60001] INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 to aa0-000-15.u.powerset.com,60020,1246461673026
> 2009-07-01 16:11:45,905 [IPC Server handler 8 on 60001] INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_PROCESS_OPEN: TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313
from aa0-000-15.u.powerset.com,60020,1246461673026; 1 of 3
> 2009-07-01 16:11:45,905 [IPC Server handler 8 on 60001] INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_OPEN: TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313
from aa0-000-15.u.powerset.com,60020,1246461673026; 2 of 3
> 2009-07-01 16:11:45,906 [IPC Server handler 8 on 60001] INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_OPEN: TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313
from aa0-000-15.u.powerset.com,60020,1246461673026; 3 of 3
> 2009-07-01 16:11:45,906 [HMaster] INFO org.apache.hadoop.hbase.master.RegionServerOperation:
TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 open on 208.76.44.142:60020
> 2009-07-01 16:11:45,906 [HMaster] INFO org.apache.hadoop.hbase.master.RegionServerOperation:
updating row TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 in region .META.,,1
with startcode 1246461673026 and server 208.76.44.142:60020
> 2009-07-01 16:11:45,908 [HMaster] INFO org.apache.hadoop.hbase.master.RegionServerOperation:
TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 open on 208.76.44.142:60020
> 2009-07-01 16:11:45,908 [HMaster] INFO org.apache.hadoop.hbase.master.RegionServerOperation:
updating row TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 in region .META.,,1
with startcode 1246461673026 and server 208.76.44.142:60020
> 2009-07-01 17:46:42,670 [IPC Server handler 0 on 60001] INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_SPLIT: TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313:
Daughters; TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246470379467, TestTable,\x00\x00\x08\x04\x05\x07\x02\x05\x04\x08,1246470379467
from aa0-000-15.u.powerset.com,60020,1246461673026; 5 of 7
> {code}
> Here is over on the regionserver:
> {code}
> 2009-07-01 16:11:42,865 [IPC Server handler 2 on 60001] INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_SPLIT: TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246462685358:
Daughters; TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313, TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313
from aa0-000-15.u.powerset.com,60020,1246461673026; 1 of 3
> 2009-07-01 16:11:42,866 [IPC Server handler 2 on 60001] INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 to aa0-000-15.u.powerset.com,60020,1246461673026
> 2009-07-01 16:11:42,866 [IPC Server handler 2 on 60001] INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 to aa0-000-15.u.powerset.com,60020,1246461673026
> 2009-07-01 16:11:45,905 [IPC Server handler 8 on 60001] INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_PROCESS_OPEN: TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313
from aa0-000-15.u.powerset.com,60020,1246461673026; 1 of 3
> 2009-07-01 16:11:45,905 [IPC Server handler 8 on 60001] INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_OPEN: TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313
from aa0-000-15.u.powerset.com,60020,1246461673026; 2 of 3
> 2009-07-01 16:11:45,906 [IPC Server handler 8 on 60001] INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_OPEN: TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313
from aa0-000-15.u.powerset.com,60020,1246461673026; 3 of 3
> 2009-07-01 16:11:45,906 [HMaster] INFO org.apache.hadoop.hbase.master.RegionServerOperation:
TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 open on X.X.X.142:60020
> 2009-07-01 16:11:45,906 [HMaster] INFO org.apache.hadoop.hbase.master.RegionServerOperation:
updating row TestTable,\x00\x00\x06\x05\x01\x05\x07\x09\x08\x00,1246464670313 in region .META.,,1
with startcode 1246461673026 and server X.X.X4.142:60020
> 2009-07-01 16:11:45,908 [HMaster] INFO org.apache.hadoop.hbase.master.RegionServerOperation:
TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 open on X.X.X.142:60020
> 2009-07-01 16:11:45,908 [HMaster] INFO org.apache.hadoop.hbase.master.RegionServerOperation:
updating row TestTable,\x00\x01\x01\x04\x04\x07\x02\x08\x08\x03,1246464670313 in region .META.,,1
with startcode 1246461673026 and server X.X.X.142:60020
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message