hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vihang Karajgaonkar (JIRA)" <>
Subject [jira] [Created] (HIVE-15355) Concurrency issues during parallel moveFile due to HDFSUtils.setFullFileStatus
Date Sun, 04 Dec 2016 21:47:58 GMT
Vihang Karajgaonkar created HIVE-15355:

             Summary: Concurrency issues during parallel moveFile due to HDFSUtils.setFullFileStatus
                 Key: HIVE-15355
             Project: Hive
          Issue Type: Bug
    Affects Versions: 2.1.0, 2.2.0
            Reporter: Vihang Karajgaonkar
            Assignee: Vihang Karajgaonkar

It is possible to run into concurrency issues during multi-threaded moveFile issued when processing
queries like {{INSERT OVERWRITE TABLE ... SELECT ..}} when there are multiple files in the
staging directory which is a subdirectory of the target directory. The issue is hard to reproduce
but following stacktrace is one such example:

INFO  : Loading data to table functional_text_gzip.alltypesaggmultifilesnopart from hdfs://localhost:20500/test-warehouse/alltypesaggmultifilesnopart_text_gzip/.hive-staging_hive_2016-12-01_19-58-21_712_8968735301422943318-1/-ext-10000
ERROR : Failed with exception java.lang.ArrayIndexOutOfBoundsException
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException
        at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(
        at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(
        at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(
        at org.apache.hadoop.hive.ql.exec.MoveTask.execute(
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(
        at org.apache.hadoop.hive.ql.Driver.launchTask(
        at org.apache.hadoop.hive.ql.Driver.execute(
        at org.apache.hadoop.hive.ql.Driver.runInternal(
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(
        at org.apache.hive.service.cli.operation.SQLOperation.access$300(
        at org.apache.hive.service.cli.operation.SQLOperation$3$
        at Method)
Getting log thread is interrupted, since query is done!
        at org.apache.hive.service.cli.operation.SQLOperation$
        at java.util.concurrent.Executors$
        at java.util.concurrent.ThreadPoolExecutor.runWorker(
        at java.util.concurrent.ThreadPoolExecutor$
Caused by: java.lang.ArrayIndexOutOfBoundsException
        at java.lang.System.arraycopy(Native Method)
        at java.util.ArrayList.removeRange(
        at java.util.ArrayList$SubList.removeRange(
        at java.util.AbstractList.clear(
        at org.apache.hadoop.hive.shims.Hadoop23Shims.removeBaseAclEntries(
        at org.apache.hadoop.hive.shims.Hadoop23Shims.setFullFileStatus(
        at org.apache.hadoop.hive.ql.metadata.Hive$
        at org.apache.hadoop.hive.ql.metadata.Hive$
        ... 4 more

ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask

Quick online search also shows some other instances like the one mentioned in

The issue seems to be coming from the below code :

if (aclEnabled) {
      aclStatus =  sourceStatus.getAclStatus();
      if (aclStatus != null) {
        aclEntries = aclStatus.getEntries();

        //the ACL api's also expect the tradition user/group/other permission in the form
of ACL
        aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.USER, sourcePerm.getUserAction()));
        aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.GROUP, sourcePerm.getGroupAction()));
        aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.OTHER, sourcePerm.getOtherAction()));

removeBaseAclEntries removes objects from {{List<AclEntry> aclEntries}} When HDFSUtils.setFullFileStatus()
method is called from multiple threads like from
it is possible that multiple threads try to modify the {{List<AclEntry> aclEntries}}
leading to concurrency issues. 

We should either move that block into a thread-safe region or call setFullFileStatus when
all the threads converge.

This message was sent by Atlassian JIRA

View raw message