Mailing-List: contact issues-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Mon, 14 Nov 2016 21:14:58 +0000 (UTC)
From: =?utf-8?Q?Sergio_Pe=C3=B1a_=28JIRA=29?= <jira@apache.org>
To: issues@hive.apache.org
Message-ID: <JIRA.13017474.1478137776000.273342.1479158098800@Atlassian.JIRA>
In-Reply-To: <JIRA.13017474.1478137776000@Atlassian.JIRA>
References: <JIRA.13017474.1478137776000@Atlassian.JIRA> <JIRA.13017474.1478137776368@arcas>
Subject: [jira] [Updated] (HIVE-15114) Remove extra MoveTask operators
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Mon, 14 Nov 2016 21:15:00 -0000


     [ https://issues.apache.org/jira/browse/HIVE-15114?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sergio Pe=C3=B1a updated HIVE-15114:
-------------------------------
    Attachment: HIVE-15114.WIP.2.patch

Attaching another file to run a full set of tests. A couple of things are n=
eeded before committing this:

- HIVE-15199 must be fixed
- S3 optimizations must be disabled by default (I enabled it for running th=
e tests)

> Remove extra MoveTask operators
> -------------------------------
>
>                 Key: HIVE-15114
>                 URL: https://issues.apache.org/jira/browse/HIVE-15114
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Hive
>    Affects Versions: 2.1.0
>            Reporter: Sahil Takiar
>            Assignee: Sergio Pe=C3=B1a
>         Attachments: HIVE-15114.WIP.1.patch, HIVE-15114.WIP.2.patch
>
>
> When running simple insert queries (e.g. {{INSERT INTO TABLE ... VALUES .=
..}}) there an extraneous {{MoveTask}s is created.
> This is problematic when the scratch directory is on S3 since renames req=
uire copying the entire dataset.
> For simple queries (like the one above), there are two MoveTasks. The fir=
st one moves the output data from one file in the scratch directory to anot=
her file in the scratch directory. The second MoveTask moves the data from =
the scratch directory to its final table location.
> The first MoveTask should not be necessary. The goal of this JIRA it to r=
emove it. This should help improve performance when running on S3.
> It seems that the first Move might be caused by a dependency resolution p=
roblem in the optimizer, where a dependent task doesn't get properly remove=
d when the task it depends on is filtered by a condition resolver.
> A dummy {{MoveTask}} is added in the {{GenMapRedUtils.createMRWorkForMerg=
ingFiles}} method. This method creates a conditional task which launches a =
job to merge tasks at the end of the file. At the end of the conditional jo=
b there is a MoveTask.
> Even though Hive decides that the conditional merge job is no needed, it =
seems the MoveTask is still added to the plan.
> Seems this extra {{MoveTask}} may have been added intentionally. Not sure=
 why yet. The {{ConditionalResolverMergeFiles}} says that one of three task=
s will be returned: move task only, merge task only, merge task followed by=
 a move task.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)