Mailing-List: contact issues-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Wed, 23 Nov 2016 21:58:58 +0000 (UTC)
From: "Hive QA (JIRA)" <jira@apache.org>
To: issues@hive.apache.org
Message-ID: <JIRA.13017474.1478137776000.355352.1479938338396@Atlassian.JIRA>
In-Reply-To: <JIRA.13017474.1478137776000@Atlassian.JIRA>
References: <JIRA.13017474.1478137776000@Atlassian.JIRA> <JIRA.13017474.1478137776368@arcas>
Subject: [jira] [Commented] (HIVE-15114) Remove extra MoveTask operators
 from the ConditionalTask
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Wed, 23 Nov 2016 21:59:00 -0000


    [ https://issues.apache.org/jira/browse/HIVE-15114?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1569=
1466#comment-15691466 ]=20

Hive QA commented on HIVE-15114:
--------------------------------


Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12840309/HIVE-15114.4.patc=
h

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10721 tests ex=
ecuted
*Failed tests:*
{noformat}
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (=
batchId=3D118)
=09[stats12.q,groupby4.q,union_top_level.q,groupby10.q,subquery_in.q,mapjoi=
n_filter_on_outerjoin.q,auto_sortmerge_join_4.q,limit_partition_metadataonl=
y.q,load_dyn_part4.q,union3.q,groupby_multi_single_reducer.q,smb_mapjoin_14=
.q,groupby3_noskew_multi_distinct.q,stats18.q,union_remove_21.q]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_sche=
ma_evol_3a] (batchId=3D133)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_pp=
r2] (batchId=3D133)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_ac=
id_non_acid] (batchId=3D150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_f=
ast_stats] (batchId=3D145)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyz=
e_5] (batchId=3D90)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2267/testR=
eport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2267/con=
sole
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2267/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12840309 - PreCommit-HIVE-Build

> Remove extra MoveTask operators from the ConditionalTask
> --------------------------------------------------------
>
>                 Key: HIVE-15114
>                 URL: https://issues.apache.org/jira/browse/HIVE-15114
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Hive
>    Affects Versions: 2.1.0
>            Reporter: Sahil Takiar
>            Assignee: Sergio Pe=C3=B1a
>         Attachments: HIVE-15114.3.patch, HIVE-15114.4.patch, HIVE-15114.W=
IP.1.patch, HIVE-15114.WIP.2.patch
>
>
> When running simple insert queries (e.g. {{INSERT INTO TABLE ... VALUES .=
..}}) there an extraneous {{MoveTask}s is created.
> This is problematic when the scratch directory is on S3 since renames req=
uire copying the entire dataset.
> For simple queries (like the one above), there are two MoveTasks. The fir=
st one moves the output data from one file in the scratch directory to anot=
her file in the scratch directory. The second MoveTask moves the data from =
the scratch directory to its final table location.
> The first MoveTask should not be necessary. The goal of this JIRA it to r=
emove it. This should help improve performance when running on S3.
> It seems that the first Move might be caused by a dependency resolution p=
roblem in the optimizer, where a dependent task doesn't get properly remove=
d when the task it depends on is filtered by a condition resolver.
> A dummy {{MoveTask}} is added in the {{GenMapRedUtils.createMRWorkForMerg=
ingFiles}} method. This method creates a conditional task which launches a =
job to merge tasks at the end of the file. At the end of the conditional jo=
b there is a MoveTask.
> Even though Hive decides that the conditional merge job is no needed, it =
seems the MoveTask is still added to the plan.
> Seems this extra {{MoveTask}} may have been added intentionally. Not sure=
 why yet. The {{ConditionalResolverMergeFiles}} says that one of three task=
s will be returned: move task only, merge task only, merge task followed by=
 a move task.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)