hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinoth Chandar (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HUDI-84) Benchmark write/read paths on Hudi vs non-Hudi datasets
Date Tue, 21 Jan 2020 03:15:00 GMT

    [ https://issues.apache.org/jira/browse/HUDI-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17019830#comment-17019830
] 

Vinoth Chandar commented on HUDI-84:
------------------------------------

Simple command to reproduce the spark df/rdd conversion issue 

 
{code:java}
val df = spark.read.parquet("file:///tmp/hudi-benchmark/input/*/*.parquet")// some input data
df.write.format("parquet").mode("overwrite").save("file:///tmp/parquet-write")

val schema = df.schema
val encoder = org.apache.spark.sql.catalyst.encoders.RowEncoder.apply(schema).resolveAndBind()
val df2 = spark.createDataFrame(df.queryExecution.toRdd.map(encoder.fromRow), schema)
df2.write.format("parquet").mode("overwrite").save("file:///tmp/parquet-write")
 {code}

> Benchmark write/read paths on Hudi vs non-Hudi datasets
> -------------------------------------------------------
>
>                 Key: HUDI-84
>                 URL: https://issues.apache.org/jira/browse/HUDI-84
>             Project: Apache Hudi (incubating)
>          Issue Type: Test
>          Components: Performance
>            Reporter: Vinoth Chandar
>            Assignee: Vinoth Chandar
>            Priority: Major
>              Labels: realtime-data-lakes
>         Attachments: df-toRdd-write.pdf, df-write-stage.pdf
>
>
> * Index performance
>  * SparkSQL (https://github.com/apache/incubator-hudi/issues/588#issuecomment-468055059)
>  * Query planning Planning 
>  * Bulk_insert, log ingest
>  * upsert, database change log. 
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message