spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Armbrust (JIRA)" <>
Subject [jira] [Created] (SPARK-4413) Parquet support through datasource API
Date Fri, 14 Nov 2014 20:44:35 GMT
Michael Armbrust created SPARK-4413:

             Summary: Parquet support through datasource API
                 Key: SPARK-4413
             Project: Spark
          Issue Type: Improvement
          Components: SQL
            Reporter: Michael Armbrust
            Assignee: Michael Armbrust
            Priority: Critical

Right now there are several issues with out parquet support.  Specifically, the only way to
access parquet files though pure SQL is by including Hive, which has the following issues
 - fairly verbose syntax
 - requires you to explicitly add partitions
 - does not support decimal types.
 - querying tables with many partitions results in metadata operations dominating the query
time (even worse when reading from S3).

It would be great to have better native support here though the new datasources API.  Ideally
once that is in place we can deprecate the existing ParquetRelation.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message