pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-4512) No performance improvement using OrcStorage
Date Mon, 27 Apr 2015 17:03:38 GMT

    [ https://issues.apache.org/jira/browse/PIG-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14514446#comment-14514446
] 

Daniel Dai commented on PIG-4512:
---------------------------------

This is not a target use case for predicate pushdown. Predicate pushdown improves performance
when Pig is able to prune the entire Orc file/stripes/column groups using the stats inside
the file/stripes/column groups. However in this case, uri is universally distributed and Pig
is not able to prune any column file/stripes/column groups, thus there is no performance improvement.
If you want to see the benefit of predicate pushdown, you shall sort the data on uri column,
so only a certain file/stripes/column contains the value you are looking for.

> No performance improvement using OrcStorage
> -------------------------------------------
>
>                 Key: PIG-4512
>                 URL: https://issues.apache.org/jira/browse/PIG-4512
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.14.0
>         Environment: Hortonworks 2.2, Pig 14.0, Hive 0.14.0, Tez
>            Reporter: Ángel Álvarez
>            Priority: Minor
>
> I've been doing some tests with Pig & Hive, trying to gain some performance using
the OrcStorage class and his "Predicate Push Down" loader. I've followed the next steps:
> 1, Download a dataset
> ftp://ita.ee.lbl.gov/traces/NASA_access_log_Aug95.gz
> 2. Create a new larger file by copying the same original file multiple times.
> cat NASA_access_log_Aug95 NASA_access_log_Aug95 ... > NASA
> 3. Add a new line in the data file.
> echo 'slppp6.intermind.net - - [01/Aug/1995:00:00:11 -0400] "GET test HTTP/1.0" 200 9202'
>> NASA
> 4. Create the ORC table in Hive
> DROP TABLE nasadata_txt;
> DROP TABLE nasadata_orc;
> CREATE TABLE nasadata_txt(ip VARCHAR(50), user_identifier VARCHAR(50), user_id VARCHAR(50),date_time
VARCHAR(50),zone VARCHAR(10),method VARCHAR(5),uri VARCHAR(200),version VARCHAR(10),status
DECIMAL(3,0),size DECIMAL(10,0)) ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ' STORED AS TEXTFILE;
> CREATE TABLE nasadata_orc(ip VARCHAR(50), user_identifier VARCHAR(50), user_id VARCHAR(50),date_time
VARCHAR(50),zone VARCHAR(10),method VARCHAR(5),uri VARCHAR(200),version VARCHAR(10),status
DECIMAL(3,0),size DECIMAL(10,0)) STORED AS ORC;
> -- Load into Text table
> LOAD DATA LOCAL INPATH 'NASA_access_log_Aug95' INTO TABLE nasadata_txt;
> LOAD DATA LOCAL INPATH 'NASA.*' INTO TABLE nasadata_txt;
> -- Copy to ORC table
> INSERT OVERWRITE TABLE nasadata_orc SELECT * FROM nasadata_txt;
> 5.  Execute this pig script
> rmf /tmp/pruebaPPD;
> A = LOAD '/apps/hive/warehouse/nasadata_orc' using OrcStorage() as (ip,user_identifier,user_id,date_time,zone,method,uri,version,status,size);
> A = foreach A generate ip,uri,status;
> A = filter A by uri == 'test';
> A = group A by uri;
> A = foreach A generate group,COUNT(*);
> store A into '/tmp/pruebaPPD' using PigStorage(';');
> 6. Execute the previous script replacing OrcStorage by org.apache.hive.hcatalog.pig.HCatLoader.
> I can't see any difference in performance between using OrcStorage and HCatLoader. Is
there anything wrong in what I'm doing? Do I have to set any property?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message