spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liang-Chi Hsieh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-9347) spark load of existing parquet files extremely slow if large number of files
Date Thu, 30 Jul 2015 13:46:04 GMT

    [ https://issues.apache.org/jira/browse/SPARK-9347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14647630#comment-14647630
] 

Liang-Chi Hsieh commented on SPARK-9347:
----------------------------------------

It will merge different schema if the parquet schema merging configuration is enabled.

> spark load of existing parquet files extremely slow if large number of files
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-9347
>                 URL: https://issues.apache.org/jira/browse/SPARK-9347
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 1.3.1
>            Reporter: Samphel Norden
>
> When spark sql shell is launched and we point it to a folder containing a large number
of parquet files, the sqlContext.parquetFile() command takes a very long time to load the
tables. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message