spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongjoon Hyun (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-20682) Support a new faster ORC data source based on Apache ORC
Date Tue, 09 May 2017 19:11:04 GMT

     [ https://issues.apache.org/jira/browse/SPARK-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dongjoon Hyun updated SPARK-20682:
----------------------------------
    Description: 
Since SPARK-2883, Apache Spark supports Apache ORC inside `sql/hive` module with Hive dependency.
This issue aims to add a new and faster ORC data source inside `sql/core` and to replace the
old ORC data source eventually. In this issue, the latest Apache ORC 1.4.0 (released yesterday)
is used.

There are four key benefits.

- Speed: Use both Spark `ColumnarBatch` and ORC `RowBatch` together. This is faster than the
current implementation in Spark.
- Stability: Apache ORC 1.4.0 has many fixes and we can depend on ORC community more.
- Usability: User can use `ORC` data sources without hive module, i.e, `-Phive`.
- Maintainability: Reduce the Hive dependency and can remove old legacy code later.

  was:
Since SPARK-2883, Apache Spark supports Apache ORC inside `sql/hive` module with Hive dependency.
This issue aims to add a new and faster ORC data source inside `sql/core` and to replace the
old ORC data source eventually. In this issue, the latest Apache ORC 1.4.0 (released yesterday)
is used.

There are four key benefits.

- Speed: Use both Spark `ColumnarBatch` and ORC `RowBatch` together. This is faster than the
current implementation in Spark.
- Stability: Apache ORC 1.4.0 has many fixes and we can depend on ORC community more.
- Usability: User can use `ORC` data sources with hive module, i.e, `-Phive`.
- Maintainability: Reduce the Hive dependency and can remove old legacy code later.


> Support a new faster ORC data source based on Apache ORC
> --------------------------------------------------------
>
>                 Key: SPARK-20682
>                 URL: https://issues.apache.org/jira/browse/SPARK-20682
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.4.1, 1.5.2, 1.6.3, 2.1.1
>            Reporter: Dongjoon Hyun
>
> Since SPARK-2883, Apache Spark supports Apache ORC inside `sql/hive` module with Hive
dependency. This issue aims to add a new and faster ORC data source inside `sql/core` and
to replace the old ORC data source eventually. In this issue, the latest Apache ORC 1.4.0
(released yesterday) is used.
> There are four key benefits.
> - Speed: Use both Spark `ColumnarBatch` and ORC `RowBatch` together. This is faster than
the current implementation in Spark.
> - Stability: Apache ORC 1.4.0 has many fixes and we can depend on ORC community more.
> - Usability: User can use `ORC` data sources without hive module, i.e, `-Phive`.
> - Maintainability: Reduce the Hive dependency and can remove old legacy code later.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message