hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markovitz, Dudu" <>
Subject RE: Is it possible to use LOAD DATA INPATH with a PARTITIONED, STORED AS PARQUET table?
Date Tue, 04 Apr 2017 15:26:18 GMT
Since LOAD DATA INPATH  only moves files the answer is very simple.
If you’re files are already in a format that matches the destination table (storage type,
number and types of columns etc.) then – yes and if not, then – no.

But –
You don’t need to load the files into intermediary table.
You should only create an external table which is an interface to read the files and use it
in an INSERT operation.


From: Dmitry Goldenberg []
Sent: Tuesday, April 04, 2017 4:52 PM
Subject: Is it possible to use LOAD DATA INPATH with a PARTITIONED, STORED AS PARQUET table?

We have a table such as the following defined:

  `item_id` string,
  `timestamp` string,
  `item_comments` string)
PARTITIONED BY (`date`, `content_type`)

Currently we insert data into this PARQUET, PARTITIONED table as follows, using an intermediary

INSERT INTO TABLE db.mytable PARTITION(date, content_type)
SELECT itemid as item_id, itemts as timestamp, date, content_type
FROM db.origtable
GROUP BY item_id, date, content_type;
Our question is, would it be possible to use the LOAD DATA INPATH.. INTO TABLE syntax to load
the data from delimited data files into 'mytable' rather than populating mytable from the
intermediary table?

I see in the Hive documentation that:
* Load operations are currently pure copy/move operations that move datafiles into locations
corresponding to Hive tables.
* If the table is partitioned, then one must specify a specific partition of the table by
specifying values for all of the partitioning columns.

This seems to indicate that using LOAD is possible; however looking at this discussion:,
perhaps not?

We'd like to understand if using LOAD in the case of PARQUET, PARTITIONED tables is possible
and if so, then how does one go about using LOAD in that case?

- Dmitry

View raw message