phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-4094) ParallelWriterIndexCommitter incorrectly applys local updates to index tables for 4.x-HBase-0.98
Date Thu, 17 Aug 2017 16:46:00 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-4094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130798#comment-16130798
] 

Hadoop QA commented on PHOENIX-4094:
------------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12882385/PHOENIX-4094_4.x-HBase-0.98_v1.patch
  against 4.x-HBase-0.98 branch at commit b13413614fef3cdb87233fd1543081e7198d685f.
  ATTACHMENT ID: 12882385

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:red}-1 tests included{color}.  The patch doesn't appear to include any new or modified
tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    {color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-PHOENIX-Build/1271//console

This message is automatically generated.

> ParallelWriterIndexCommitter incorrectly applys local updates to index tables for 4.x-HBase-0.98
> ------------------------------------------------------------------------------------------------
>
>                 Key: PHOENIX-4094
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-4094
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.11.0
>            Reporter: chenglei
>            Assignee: chenglei
>             Fix For: 4.12.0
>
>         Attachments: PHOENIX-4094_4.x-HBase-0.98_v1.patch
>
>
> I used phoenix-4.x-HBase-0.98 in my hbase cluster.When I restarted my hbase cluster a
certain time, I noticed some  RegionServers have plenty of  {{WrongRegionException}} as following:
> {code:java}
> 2017-08-01 11:53:10,669 WARN  [rsync.slave005.bizhbasetest.sjs.ted,60020,1501511894174-index-writer--pool2-t786]
regionserver.HRegion: Failed getting lock in batch put, row=\x10\x00\x00\x00913f0eed-6710-4de9-8bac-077a106bb9ae_0
> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out of range
for row lock on HRegion BIZARCH_NS_PRODUCT.BIZTRACER_SPAN,90ffd783-b0a3-4f8a-81ef-0a7535fea197_0,1490066612493.463220cd8fad7254481595911e62d74d.,
startKey='90ffd783-b0a3-4f8a-81ef-0a7535fea197_0', getEndKey()='917fc343-3331-47fa-907c-df83a6f302f7_0',
row='\x10\x00\x00\x00913f0eed-6710-4de9-8bac-077a106bb9ae_0'
>         at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:3539)
>         at org.apache.hadoop.hbase.regionserver.HRegion.getRowLock(HRegion.java:3557)
>         at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2394)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2261)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2213)
>         at org.apache.phoenix.util.IndexUtil.writeLocalUpdates(IndexUtil.java:671)
>         at org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:157)
>         at org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:134)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> {code}
> The problem is caused by the ParallelWriterIndexCommitter.write method, in following
line 151, if {{allowLocalUpdates}}  is true, it would wiite index mutations to current data
table region unconditionlly,which is obviously inappropriate: 
> {code:java}
>  150            try {
>  151                      if (allowLocalUpdates && env != null) {
>  152                           try {
>  153                               throwFailureIfDone();
>  154                               IndexUtil.writeLocalUpdates(env.getRegion(), mutations,
true);
>  155                               return null;
>  156                           } catch (IOException ignord) {
>  157                               // when it's failed we fall back to the standard &
slow way
>  158                               if (LOG.isDebugEnabled()) {
>  159                                   LOG.debug("indexRegion.batchMutate failed and
fall back to HTable.batch(). Got error="
>  160                                           + ignord);
>  161                               }
>  162                           }
>  163                       }
> {code}
> If a data table has a global index table , and when we replay the WALs to index table
in Indexer.postOpen method in following 
> line 691, which the {{allowLocalUpdates}} parameter is true, the  {{updates}} parameter
for the global index table would  incorrectly be written to the current data table region:
> {code:java}
> 688        // do the usual writer stuff, killing the server again, if we can't manage
to make the index
> 689        // writes succeed again
> 690        try {
> 691            writer.writeAndKillYourselfOnFailure(updates, true);
> 692        } catch (IOException e) {
> 693                LOG.error("During WAL replay of outstanding index updates, "
> 694                        + "Exception is thrown instead of killing server during index
writing", e);
> 695        }
> 696    } finally {
> {code}
> However, ParallelWriterIndexCommitter.write method in the master and other 4.x branches
is correct, just as following line 150 and line 151 :
> {code:java}
> 147                   try {
> 148                        if (allowLocalUpdates
> 149                                && env != null
> 150                                && tableReference.getTableName().equals(
> 151                                    env.getRegion().getTableDesc().getNameAsString()))
{
> 152                            try {
> 153                                throwFailureIfDone();
> 154                                IndexUtil.writeLocalUpdates(env.getRegion(), mutations,
true);
> 155                                return null;
> 156                            } catch (IOException ignord) {
> 157                                // when it's failed we fall back to the standard &
slow way
> 158                                if (LOG.isDebugEnabled()) {
> 159                                    LOG.debug("indexRegion.batchMutate failed and
fall back to HTable.batch(). Got error="
> 160                                            + ignord);
> 161                                }
> 162                            }
> 163                        }
> {code}
> This inconsistency of branches is introduced by PHOENIX-1734 and PHOENIX-3018, because
lack of unit tests or IT tests for Indexer.preWALRestore /postOpen,  the  inconsistency is
not detected.
> BTW,the TrackingParallelWriterIndexCommitter  is right for master and all the 4.x branches.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message