hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Poepping (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14270) Write temporary data to HDFS when doing inserts on tables located on S3
Date Fri, 22 Jul 2016 16:52:20 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389817#comment-15389817
] 

Thomas Poepping commented on HIVE-14270:
----------------------------------------

The unit tests you want would verify that the scheme of the temporary directories is hdfs
if the scheme of the original directory "isS3".

I suggest you write a kind of helper method (getTempDir, maybe) that returns a proper temporary
directory based on the scheme we have. That will be more easily unit testable, and will prevent
future developers adding new features that need temp directories to remember to check "isS3"
before they do so.

> Write temporary data to HDFS when doing inserts on tables located on S3
> -----------------------------------------------------------------------
>
>                 Key: HIVE-14270
>                 URL: https://issues.apache.org/jira/browse/HIVE-14270
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Sergio Peña
>            Assignee: Sergio Peña
>         Attachments: HIVE-14270.1.patch
>
>
> Currently, when doing INSERT statements on tables located at S3, Hive writes and reads
temporary (or intermediate) files to S3 as well. 
> If HDFS is still the default filesystem on Hive, then we can keep such temporary files
on HDFS to keep things run faster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message