hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HUDI-1425) Performance loss with the additional hoodieRecords.isEmpty() in HoodieSparkSqlWriter#write
Date Tue, 06 Jul 2021 11:42:01 GMT

    [ https://issues.apache.org/jira/browse/HUDI-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17375637#comment-17375637
] 

ASF GitHub Bot commented on HUDI-1425:
--------------------------------------

vinothchandar commented on pull request #2296:
URL: https://github.com/apache/hudi/pull/2296#issuecomment-874367066


   @nsivabalan mind taking a pass at this again?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Performance loss with the additional hoodieRecords.isEmpty() in HoodieSparkSqlWriter#write
> ------------------------------------------------------------------------------------------
>
>                 Key: HUDI-1425
>                 URL: https://issues.apache.org/jira/browse/HUDI-1425
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: Spark Integration
>    Affects Versions: 0.9.0
>            Reporter: pengzhiwei
>            Assignee: pengzhiwei
>            Priority: Blocker
>              Labels: pull-request-available
>             Fix For: 0.9.0
>
>         Attachments: 截屏2020-11-30 下午9.47.55.png
>
>
> Currently in HoodieSparkSqlWriter#write, there is a _isEmpty()_ test for _hoodieRecords._
This may be a heavy operator in the case when the _hoodieRecords_ contains complex RDD operate.
> !截屏2020-11-30 下午9.47.55.png|width=1255,height=161!
> IMO this test does nothing to do with the performance improve,but rather affects performance.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message