Mailing-List: contact issues-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Tue, 6 Dec 2016 18:20:58 +0000 (UTC)
From: "Vihang Karajgaonkar (JIRA)" <jira@apache.org>
To: issues@hive.apache.org
Message-ID: <JIRA.13025446.1480888074000.453008.1481048458499@Atlassian.JIRA>
In-Reply-To: <JIRA.13025446.1480888074000@Atlassian.JIRA>
References: <JIRA.13025446.1480888074000@Atlassian.JIRA> <JIRA.13025446.1480888074352@arcas>
Subject: [jira] [Commented] (HIVE-15355) Concurrency issues during parallel
 moveFile due to HDFSUtils.setFullFileStatus
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 06 Dec 2016 18:21:00 -0000


    [ https://issues.apache.org/jira/browse/HIVE-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726275#comment-15726275 ] 

Vihang Karajgaonkar commented on HIVE-15355:
--------------------------------------------

Thanks for the review [~spena]

> Concurrency issues during parallel moveFile due to HDFSUtils.setFullFileStatus
> ------------------------------------------------------------------------------
>
>                 Key: HIVE-15355
>                 URL: https://issues.apache.org/jira/browse/HIVE-15355
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 2.1.0, 2.2.0
>            Reporter: Vihang Karajgaonkar
>            Assignee: Vihang Karajgaonkar
>         Attachments: HIVE-15355.01.patch, HIVE-15355.02.patch
>
>
> It is possible to run into concurrency issues during multi-threaded moveFile issued when processing queries like {{INSERT OVERWRITE TABLE ... SELECT ..}} when there are multiple files in the staging directory which is a subdirectory of the target directory. The issue is hard to reproduce but following stacktrace is one such example:
> {noformat}
> INFO  : Loading data to table functional_text_gzip.alltypesaggmultifilesnopart from hdfs://localhost:20500/test-warehouse/alltypesaggmultifilesnopart_text_gzip/.hive-staging_hive_2016-12-01_19-58-21_712_8968735301422943318-1/-ext-10000
> ERROR : Failed with exception java.lang.ArrayIndexOutOfBoundsException
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException
>         at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2858)
>         at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:3124)
>         at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1701)
>         at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:313)
>         at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
>         at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
>         at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1976)
>         at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1689)
>         at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1421)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1205)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1200)
>         at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
>         at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
>         at org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
> Getting log thread is interrupted, since query is done!
>         at org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ArrayIndexOutOfBoundsException
>         at java.lang.System.arraycopy(Native Method)
>         at java.util.ArrayList.removeRange(ArrayList.java:616)
>         at java.util.ArrayList$SubList.removeRange(ArrayList.java:1021)
>         at java.util.AbstractList.clear(AbstractList.java:234)
>         at com.google.common.collect.Iterables.removeIfFromRandomAccessList(Iterables.java:213)
>         at com.google.common.collect.Iterables.removeIf(Iterables.java:184)
>         at org.apache.hadoop.hive.shims.Hadoop23Shims.removeBaseAclEntries(Hadoop23Shims.java:865)
>         at org.apache.hadoop.hive.shims.Hadoop23Shims.setFullFileStatus(Hadoop23Shims.java:757)
>         at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:2835)
>         at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:2828)
>         ... 4 more
> ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
> {noformat}
> Quick online search also shows some other instances like the one mentioned in http://stackoverflow.com/questions/38900333/get-concurrentmodificationexception-in-step-2-create-intermediate-flat-hive-tab
> The issue seems to be coming from the below code :
> {code}
> if (aclEnabled) {
>       aclStatus =  sourceStatus.getAclStatus();
>       if (aclStatus != null) {
>         LOG.trace(aclStatus.toString());
>         aclEntries = aclStatus.getEntries();
>         removeBaseAclEntries(aclEntries);
>         //the ACL api's also expect the tradition user/group/other permission in the form of ACL
>         aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.USER, sourcePerm.getUserAction()));
>         aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.GROUP, sourcePerm.getGroupAction()));
>         aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.OTHER, sourcePerm.getOtherAction()));
>       }
>     }
> {code}
> removeBaseAclEntries removes objects from {{List<AclEntry> aclEntries}} When HDFSUtils.setFullFileStatus() method is called from multiple threads like from https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2835 it is possible that multiple threads try to modify the {{List<AclEntry> aclEntries}} leading to concurrency issues. 
> We should either move that block into a thread-safe region or call setFullFileStatus when all the threads converge.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)