Mailing-List: contact issues-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Wed, 16 Nov 2016 02:01:05 +0000 (UTC)
From: "Sahil Takiar (JIRA)" <jira@apache.org>
To: issues@hive.apache.org
Message-ID: <JIRA.13017474.1478137776000.288107.1479261665467@Atlassian.JIRA>
In-Reply-To: <JIRA.13017474.1478137776000@Atlassian.JIRA>
References: <JIRA.13017474.1478137776000@Atlassian.JIRA> <JIRA.13017474.1478137776368@arcas>
Subject: [jira] [Commented] (HIVE-15114) Remove extra MoveTask operators
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Wed, 16 Nov 2016 02:01:07 -0000


    [ https://issues.apache.org/jira/browse/HIVE-15114?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1566=
9061#comment-15669061 ]=20

Sahil Takiar commented on HIVE-15114:
-------------------------------------

[~spena] can we add some sets for other execution engines as well? I haven'=
t test the 2nd patch, but the first patch doesn't seem to take affect for H=
oS.

> Remove extra MoveTask operators
> -------------------------------
>
>                 Key: HIVE-15114
>                 URL: https://issues.apache.org/jira/browse/HIVE-15114
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Hive
>    Affects Versions: 2.1.0
>            Reporter: Sahil Takiar
>            Assignee: Sergio Pe=C3=B1a
>         Attachments: HIVE-15114.WIP.1.patch, HIVE-15114.WIP.2.patch
>
>
> When running simple insert queries (e.g. {{INSERT INTO TABLE ... VALUES .=
..}}) there an extraneous {{MoveTask}s is created.
> This is problematic when the scratch directory is on S3 since renames req=
uire copying the entire dataset.
> For simple queries (like the one above), there are two MoveTasks. The fir=
st one moves the output data from one file in the scratch directory to anot=
her file in the scratch directory. The second MoveTask moves the data from =
the scratch directory to its final table location.
> The first MoveTask should not be necessary. The goal of this JIRA it to r=
emove it. This should help improve performance when running on S3.
> It seems that the first Move might be caused by a dependency resolution p=
roblem in the optimizer, where a dependent task doesn't get properly remove=
d when the task it depends on is filtered by a condition resolver.
> A dummy {{MoveTask}} is added in the {{GenMapRedUtils.createMRWorkForMerg=
ingFiles}} method. This method creates a conditional task which launches a =
job to merge tasks at the end of the file. At the end of the conditional jo=
b there is a MoveTask.
> Even though Hive decides that the conditional merge job is no needed, it =
seems the MoveTask is still added to the plan.
> Seems this extra {{MoveTask}} may have been added intentionally. Not sure=
 why yet. The {{ConditionalResolverMergeFiles}} says that one of three task=
s will be returned: move task only, merge task only, merge task followed by=
 a move task.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)