hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Balaji Varadarajan (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (HUDI-637) Investigate slower hudi queries in S3 vs HDFS
Date Thu, 27 Feb 2020 03:49:00 GMT

    [ https://issues.apache.org/jira/browse/HUDI-637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17046119#comment-17046119
] 

Balaji Varadarajan edited comment on HUDI-637 at 2/27/20 3:48 AM:
------------------------------------------------------------------

[~vinoth]: This is from [https://github.com/apache/incubator-hudi/issues/588#issuecomment-591430119] 

I thought this was the background issue you were looking into. 


was (Author: vbalaji):
[~vinoth]: This is from [https://github.com/apache/incubator-hudi/issues/588#issuecomment-591430119] 

> Investigate slower hudi queries in S3 vs HDFS
> ---------------------------------------------
>
>                 Key: HUDI-637
>                 URL: https://issues.apache.org/jira/browse/HUDI-637
>             Project: Apache Hudi (incubating)
>          Issue Type: Task
>          Components: Performance
>            Reporter: Balaji Varadarajan
>            Priority: Blocker
>             Fix For: 0.5.2
>
>
> Hudi queries in S3 takes abnormally longer time compared to hdfs. 
> S3 listing itself is not taking that long of time. 
> PERFORMANCE BUG:
> the metadata list performance is likely causing performance issues with hudi.
>  
> {{scala> stopwatch(\{ sql("SELECT * FROM ap_invoices_all_compacted_s3").count})}}
> {{Elapsed time: 1m 55.078473113s 
>  res2: Long = xxxxxxxxxxxx}}
> {{}}
> {{scala> stopwatch(\{ sql("SELECT * FROM ap_invoices_all_compacted").count}) – this
is the exact same table in hdfs}}
> {{Elapsed time: 6.581217052s 
>  res3: Long = xxxxxxxxxxx}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message