Mailing-List: contact issues-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Tue, 6 Dec 2016 21:35:59 +0000 (UTC)
From: =?utf-8?Q?Sergio_Pe=C3=B1a_=28JIRA=29?= <jira@apache.org>
To: issues@hive.apache.org
Message-ID: <JIRA.13025822.1480989090000.454806.1481060159441@Atlassian.JIRA>
In-Reply-To: <JIRA.13025822.1480989090000@Atlassian.JIRA>
References: <JIRA.13025822.1480989090000@Atlassian.JIRA> <JIRA.13025822.1480989090452@arcas>
Subject: [jira] [Commented] (HIVE-15367) CTAS with LOCATION should write
 temp data under location directory rather than database location
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Tue, 06 Dec 2016 21:36:01 -0000


    [ https://issues.apache.org/jira/browse/HIVE-15367?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1572=
6782#comment-15726782 ]=20

Sergio Pe=C3=B1a commented on HIVE-15367:
------------------------------------

[~stakiar] The patch looks good. Could you add some tests to validate the L=
OCATION scenarios? Or are there tests that already do that?
Can you add hive-blobstore tests as well?

> CTAS with LOCATION should write temp data under location directory rather=
 than database location
> -------------------------------------------------------------------------=
-----------------------
>
>                 Key: HIVE-15367
>                 URL: https://issues.apache.org/jira/browse/HIVE-15367
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
>         Attachments: HIVE-15367.1.patch
>
>
> For regular CTAS queries, temp data from a SELECT query will be written t=
o to a staging directory under the database location. The code to control t=
his is in {{SemanticAnalyzer.java}}
> {code}
>              // allocate a temporary output dir on the location of the ta=
ble
>               String tableName =3D getUnescapedName((ASTNode) ast.getChil=
d(0));
>               String[] names =3D Utilities.getDbTableName(tableName);
>               Path location;
>               try {
>                 Warehouse wh =3D new Warehouse(conf);
>                 //Use destination table's db location.
>                 String destTableDb =3D qb.getTableDesc() !=3D null? qb.ge=
tTableDesc().getDatabaseName(): null;
>                 if (destTableDb =3D=3D null) {
>                   destTableDb =3D names[0];
>                 }
>                 location =3D wh.getDatabasePath(db.getDatabase(destTableD=
b));
>               } catch (MetaException e) {
>                 throw new SemanticException(e);
>               }
> {code}
> However, CTAS queries allow specifying a {{LOCATION}} for the new table. =
Its possible for this location to be on a different filesystem than the dat=
abase location. If this happens temp data will be written to the database f=
ilesystem and will be copied to the table filesystem in {{MoveTask}}.
> This extra copying of data can drastically affect performance. Rather tha=
n always use the database location as the staging dir for CTAS queries, Hiv=
e should first check if there is an explicit {{LOCATION}} specified in the =
CTAS query. If there is, staging data should be stored under the {{LOCATION=
}} directory.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)