hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mahender Sarangam <mahender.bigd...@outlook.com>
Subject Re: Hive External Table with Zero Bytes files
Date Sat, 28 Apr 2018 09:05:14 GMT
Gentle Ping. Please help me on below issue. Has any one faced same issue

On 4/27/2018 1:28 AM, Mahender Sarangam wrote:

Hi,

Can any one faced issue while fetching data from external table. We are copying data from
upstream system into our storage S3. As part of copy, directories along with Zero bytes files
are been copied. Source File Format is in JSON format.  Below is Folder Hierarchy Structure


 DATE  -->  <Folder>

       <DAY=201803250> ---> Folder

                     1.json.gz  --> File

                      2.json.gz

        <day=201803250> ---> Empty Zero Bytes Files.

Please find below screenshot

[cid:part1.4FF9D08A.0847F3AF@outlook.com]

We are trying to create external table with JSON Serde.

ADD JAR wasb://jsonserde@XYZ.blob.core.windows.net/json/json-serde-1.3.9.jar<mailto:wasb://jsonserde@XYZ.blob.core.windows.net/json/json-serde-1.3.9.jar>;
 SET hive.mapred.supports.subdirectories=TRUE;
 SET mapred.input.dir.recursive=TRUE;
SET hive.merge.mapfiles = true;
SET hive.merge.mapredfiles = true;
SET hive.merge.tezfiles = true;


 DROP TABLE IF EXISTS Ext_STG1;
 CREATE EXTERNAL TABLE Ext_STG1(Col1 String, Col2 String, Col3 String) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES ("case.insensitive" = "true", "ignore.malformed.json" = "true")
STORED AS TEXTFILE LOCATION 'wasb://container1@xyz.blob.core.windows.net/date/day=201803250/<mailto:wasb://container1@xyz.blob.core.windows.net/date/day=201803250/>'
TBLPROPERTIES ('serialization.null.format' = '');

select * from Ext_STG1 limit 100;


Above Query shows Empty Results.


When I delete Zero bytes files, then i could see data from select external table. Is this
expected behaviour. Is there any setting for ignoring Zero bytes files in hive external table


-Mahens

Mime
View raw message