tajo-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TAJO-2030) Use list S3 files using AmazonS3Client instead of using S3A
Date Mon, 03 Apr 2017 22:05:41 GMT

    [ https://issues.apache.org/jira/browse/TAJO-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15954223#comment-15954223
] 

Steve Loughran commented on TAJO-2030:
--------------------------------------

probably a bit late for you now, but HADOOP-13208 does the speedup you want in S3a. You know,
if you do think there is some opportunity for speedup in that hadoop s3a code, happy to take
it on provided it comes with decent set of integration tests.

> Use list S3 files using AmazonS3Client instead of using S3A
> -----------------------------------------------------------
>
>                 Key: TAJO-2030
>                 URL: https://issues.apache.org/jira/browse/TAJO-2030
>             Project: Tajo
>          Issue Type: Sub-task
>          Components: S3
>            Reporter: Jaehwa Jung
>            Assignee: Jaehwa Jung
>             Fix For: 0.12.0
>
>
> AWS S3 provides bulk listing API. It takes the common prefix of all input paths as a
parameter and returns all the objects whose prefixes start with the common prefix in blocks
of 1000.
> If we will use AmazonS3Client for listing S3 files instead of using S3A, this will improve
performance. To prove this idea, I adopted PrestoFileSystem instead of S3AFileSystem. When
pruning partition filters, PrestoFileSystem was faster much more than S3AFileSystem.
> Here is my benchmark results for the following queries:
> {code}
> 1 partition : select count(*) from lineitem where l_shipdate = '1992-01-02';
> 30 partitions: select count(*) from lineitem  where l_shipdate > '1992-01-01' and
l_shipdate < '1992-02-01';
> 90 partitions: select count(*) from lineitem  where l_shipdate >= '1992-01-01' and
l_shipdate < '1992-04-01';
> 151 partitions: select count(*) from lineitem where l_shipdate >= '1992-01-01' and
l_shipdate < '1992-06-01';
> {code}
> || (#) of partitions||PrestoFileSystem(ms)||S3AFileSystem(ms)||
> |1|677|800|
> |30|2753|6977|
> |90|6825|13772|
> |151|13834|25701|
> For the reference, I used tpc-h 1g dataset and set {{l_shipdate}} column of {{lineitem}}
table to partition column.
> I think there are ways to resolve this as following:
> - Borrow PrestoFileSystem and related codes from Presto
> - Implement necessary codes to S3TableSpace by referencing Presto



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message