hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Samir Ahmic (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-15121) ConnectionImplementation#locateRegionInMeta() issue when master is restarted
Date Mon, 18 Jan 2016 16:56:39 GMT

     [ https://issues.apache.org/jira/browse/HBASE-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Samir Ahmic updated HBASE-15121:
--------------------------------
    Description: 
I notice this issue while i was running IntegrationTestMTTR#testRestartMaster() test was failing
on put operation. Here is sequence of events from logs leading to failed put operation:
Master restart
{code}
INFO  [pool-5-thread-1] util.Shell: Executing full command [/usr/bin/ssh  hnode2 "sudo -u
hbase ps aux | grep proc_master | grep -v grep | tr -s ' ' | cut -d ' ' -f2 | xargs kill -s
SIGKILL"]
{code} 
Client trying to locate region for row=70efdf2ec9b086079795c442636b55fb-17  (this is additional
logging inspecting metaKey which is used to search hbase:meta )
{code}
2016-01-15 10:26:05,169 INFO  [HBaseWriterThread_9] client.ConnectionImplementation: metaKey
inspection: table=IntegrationTestMTTRLoadTestTool row= 70efdf2ec9b086079795c442636b55fb-17
metaKey= IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99999999999999
{code}
Client throwing TableNotFoundException (hbase:meta scan returned null)
{code}
2016-01-15 10:32:58,154 INFO  [HBaseWriterThread_5] client.ConnectionImplementation: regionInfo
result is null: HBaseWriterThread_5 throwing TableNotFoundException logging details table=IntegrationTestMTTRLoadTestTool
row=70efdf2ec9b086079795c442636b55fb-17 metaKey=IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99999999999999
2016-01-15 10:32:58,154 ERROR [HBaseWriterThread_5] client.AsyncProcess: Failed to get region
location
org.apache.hadoop.hbase.TableNotFoundException: IntegrationTestMTTRLoadTestTool
        at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890)
        at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781)
        at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396)
        at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344)
        at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239)
        at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191)
        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949)
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569)
        at org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146)
        at org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111)
{code}

And as result we have failed insert  operation:
{code}
2016-01-15 10:32:58,179 ERROR [HBaseWriterThread_5] util.MultiThreadedWriter: Failed to insert:
17 after 60046ms; region information: cached: region=IntegrationTestMTTRLoadTestTool,66666660,1452849956427.05b437185a9437f178726a55a29a79b7.,
hostname=hnode4,16020,1452776418437, seqNum=5; cache is up to date; errors: exception from
null for 70efdf2ec9b086079795c442636b55fb-17
org.apache.hadoop.hbase.TableNotFoundException: IntegrationTestMTTRLoadTestTool
        at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890)
        at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781)
        at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396)
        at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344)
        at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239)
        at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191)
        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949)
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569)
        at org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146)
        at org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111)
{code}
leading to test failing:
{code}
Failed to write key: 17
2016-01-15 10:33:53,984 INFO  [main] mttr.IntegrationTestMTTR: RestartMaster failed after
469878ms.
java.util.concurrent.ExecutionException: java.lang.AssertionError: Load failed expected:<0>
but was:<1>
{code}

Here is snippet from ConnectionImplementation#locateRegionInMeta() throwing exception:
{code}
      try {
        Result regionInfoRow = null;
        ReversedClientScanner rcs = null;
        try {
          rcs = new ClientSmallReversedScanner(conf, s, TableName.META_TABLE_NAME, this,
            rpcCallerFactory, rpcControllerFactory, getMetaLookupPool(), 0);
          regionInfoRow = rcs.next();
        } finally {
          if (rcs != null) {
            rcs.close();
          }
        }

        if (regionInfoRow == null) {
          throw new TableNotFoundException(tableName);
{code}
I was able to avoid this issue by removing thrown  declaration and adding continue allowing
client to retry to locate region. This sounds like simplest solution here. 
Thoughts ?

  was:
I notice this issue while i was running IntegrationTestMTTR#testRestartMaster() test was failing
on put operation. Here is sequence of events from logs leading to failed put operation:
Master restart
{code}
INFO  [pool-5-thread-1] util.Shell: Executing full command [/usr/bin/ssh  hnode2 "sudo -u
hbase ps aux | grep proc_master | grep -v grep | tr -s ' ' | cut -d ' ' -f2 | xargs kill -s
SIGKILL"]
{code} 
Client trying to locate region for row=70efdf2ec9b086079795c442636b55fb-17  (this is additional
logging inspecting metaKey which is used to search hbase:meta )
{code}
2016-01-15 10:26:05,169 INFO  [HBaseWriterThread_9] client.ConnectionImplementation: metaKey
inspection: table=IntegrationTestMTTRLoadTestTool row= 70efdf2ec9b086079795c442636b55fb-17
metaKey= IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99999999999999
{code}
Client throwing TableNotFoundException (hbase:meta scan returned null)
{code}
2016-01-15 10:32:58,154 INFO  [HBaseWriterThread_5] client.ConnectionImplementation: regionInfo
result is null: HBaseWriterThread_5 throwing TableNotFoundException logging details table=IntegrationTestMTTRLoadTestTool
row=70efdf2ec9b086079795c442636b55fb-17 metaKey=IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99999999999999
2016-01-15 10:32:58,154 ERROR [HBaseWriterThread_5] client.AsyncProcess: Failed to get region
location
org.apache.hadoop.hbase.TableNotFoundException: IntegrationTestMTTRLoadTestTool
        at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890)
        at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781)
        at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396)
        at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344)
        at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239)
        at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191)
        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949)
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569)
        at org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146)
        at org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111)
{code}

And as result we have failed insert  operation:
{code}
2016-01-15 10:32:58,179 ERROR [HBaseWriterThread_5] util.MultiThreadedWriter: Failed to insert:
17 after 60046ms; region information: cached: region=IntegrationTestMTTRLoadTestTool,66666660,1452849956427.05b437185a9437f178726a55a29a79b7.,
hostname=hnode4,16020,1452776418437, seqNum=5; cache is up to date; errors: exception from
null for 70efdf2ec9b086079795c442636b55fb-17
org.apache.hadoop.hbase.TableNotFoundException: IntegrationTestMTTRLoadTestTool
        at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890)
        at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781)
        at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396)
        at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344)
        at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239)
        at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191)
        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949)
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569)
        at org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146)
        at org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111)
{code}
leading to test failing:
{code}
Failed to write key: 17
2016-01-15 10:33:53,984 INFO  [main] mttr.IntegrationTestMTTR: RestartMaster failed after
469878ms.
java.util.concurrent.ExecutionException: java.lang.AssertionError: Load failed expected:<0>
but was:<1>
{code}

Here is snippet from ConnectionImplementation#locateRegionInMeta() throwing exception:
{code}
      try {
        Result regionInfoRow = null;
        ReversedClientScanner rcs = null;
        try {
          rcs = new ClientSmallReversedScanner(conf, s, TableName.META_TABLE_NAME, this,
            rpcCallerFactory, rpcControllerFactory, getMetaLookupPool(), 0);
          regionInfoRow = rcs.next();
        } finally {
          if (rcs != null) {
            rcs.close();
          }
        }

        if (regionInfoRow == null) {
          throw new TableNotFoundException(tableName);
{code}
I was able to avoid this issue by removing thrown  declaration with continue allowing client
to retry to locate region. This sounds like simplest solution here. 
Thoughts ?


> ConnectionImplementation#locateRegionInMeta() issue when master is restarted
> ----------------------------------------------------------------------------
>
>                 Key: HBASE-15121
>                 URL: https://issues.apache.org/jira/browse/HBASE-15121
>             Project: HBase
>          Issue Type: Bug
>          Components: Client
>    Affects Versions: 2.0.0
>            Reporter: Samir Ahmic
>            Assignee: Samir Ahmic
>             Fix For: 2.0.0
>
>
> I notice this issue while i was running IntegrationTestMTTR#testRestartMaster() test
was failing on put operation. Here is sequence of events from logs leading to failed put operation:
> Master restart
> {code}
> INFO  [pool-5-thread-1] util.Shell: Executing full command [/usr/bin/ssh  hnode2 "sudo
-u hbase ps aux | grep proc_master | grep -v grep | tr -s ' ' | cut -d ' ' -f2 | xargs kill
-s SIGKILL"]
> {code} 
> Client trying to locate region for row=70efdf2ec9b086079795c442636b55fb-17  (this is
additional logging inspecting metaKey which is used to search hbase:meta )
> {code}
> 2016-01-15 10:26:05,169 INFO  [HBaseWriterThread_9] client.ConnectionImplementation:
metaKey inspection: table=IntegrationTestMTTRLoadTestTool row= 70efdf2ec9b086079795c442636b55fb-17
metaKey= IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99999999999999
> {code}
> Client throwing TableNotFoundException (hbase:meta scan returned null)
> {code}
> 2016-01-15 10:32:58,154 INFO  [HBaseWriterThread_5] client.ConnectionImplementation:
regionInfo result is null: HBaseWriterThread_5 throwing TableNotFoundException logging details
table=IntegrationTestMTTRLoadTestTool row=70efdf2ec9b086079795c442636b55fb-17 metaKey=IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99999999999999
> 2016-01-15 10:32:58,154 ERROR [HBaseWriterThread_5] client.AsyncProcess: Failed to get
region location
> org.apache.hadoop.hbase.TableNotFoundException: IntegrationTestMTTRLoadTestTool
>         at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890)
>         at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781)
>         at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396)
>         at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344)
>         at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239)
>         at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191)
>         at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949)
>         at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569)
>         at org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146)
>         at org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111)
> {code}
> And as result we have failed insert  operation:
> {code}
> 2016-01-15 10:32:58,179 ERROR [HBaseWriterThread_5] util.MultiThreadedWriter: Failed
to insert: 17 after 60046ms; region information: cached: region=IntegrationTestMTTRLoadTestTool,66666660,1452849956427.05b437185a9437f178726a55a29a79b7.,
hostname=hnode4,16020,1452776418437, seqNum=5; cache is up to date; errors: exception from
null for 70efdf2ec9b086079795c442636b55fb-17
> org.apache.hadoop.hbase.TableNotFoundException: IntegrationTestMTTRLoadTestTool
>         at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890)
>         at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781)
>         at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396)
>         at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344)
>         at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239)
>         at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191)
>         at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949)
>         at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569)
>         at org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146)
>         at org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111)
> {code}
> leading to test failing:
> {code}
> Failed to write key: 17
> 2016-01-15 10:33:53,984 INFO  [main] mttr.IntegrationTestMTTR: RestartMaster failed after
469878ms.
> java.util.concurrent.ExecutionException: java.lang.AssertionError: Load failed expected:<0>
but was:<1>
> {code}
> Here is snippet from ConnectionImplementation#locateRegionInMeta() throwing exception:
> {code}
>       try {
>         Result regionInfoRow = null;
>         ReversedClientScanner rcs = null;
>         try {
>           rcs = new ClientSmallReversedScanner(conf, s, TableName.META_TABLE_NAME, this,
>             rpcCallerFactory, rpcControllerFactory, getMetaLookupPool(), 0);
>           regionInfoRow = rcs.next();
>         } finally {
>           if (rcs != null) {
>             rcs.close();
>           }
>         }
>         if (regionInfoRow == null) {
>           throw new TableNotFoundException(tableName);
> {code}
> I was able to avoid this issue by removing thrown  declaration and adding continue allowing
client to retry to locate region. This sounds like simplest solution here. 
> Thoughts ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message