spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-18457) ORC and other columnar formats using HiveShim read all columns when doing a simple count
Date Wed, 16 Nov 2016 02:39:58 GMT

    [ https://issues.apache.org/jira/browse/SPARK-18457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669150#comment-15669150
] 

Hyukjin Kwon edited comment on SPARK-18457 at 11/16/16 2:39 AM:
----------------------------------------------------------------

Hm, don't we push down no column when performing

{code}
count(*)
{code} 

?

{code}
val data = (0 to 255).zip(0 to 255).toDF("a", "b")
val path = "/tmp/aa"
data.write.orc(path)
spark.read.orc(path).createOrReplaceTempView("a_orc_table")
spark.sql("select count(*) from a_orc_table").explain(true)
{code}

{code}
== Parsed Logical Plan ==
'Project [unresolvedalias('count(1), None)]
+- 'UnresolvedRelation `a_orc_table`

== Analyzed Logical Plan ==
count(1): bigint
Aggregate [count(1) AS count(1)#28L]
+- SubqueryAlias a_orc_table
   +- Relation[a#16,b#17] orc

== Optimized Logical Plan ==
Aggregate [count(1) AS count(1)#28L]
+- Project
   +- Relation[a#16,b#17] orc

== Physical Plan ==
*HashAggregate(keys=[], functions=[count(1)], output=[count(1)#28L])
+- Exchange SinglePartition
   +- *HashAggregate(keys=[], functions=[partial_count(1)], output=[count#30L])
      +- *FileScan orc [] Batched: false, Format: ORC, Location: InMemoryFileIndex[file:/tmp/aa],
PartitionFilters: [], PushedFilters: [], ReadSchema: struct<>
{code}

It seems ORC datasource does not try to read all columns and I just verified this by debugging
with IDE.


was (Author: hyukjin.kwon):
Hm, don't we push down no column when performing {{count(*)}}? 

{code}
val data = (0 to 255).zip(0 to 255).toDF("a", "b")
val path = "/tmp/aa"
data.write.orc(path)
spark.read.orc(path).createOrReplaceTempView("a_orc_table")
spark.sql("select count(*) from a_orc_table").explain(true)
{code}

{code}
== Parsed Logical Plan ==
'Project [unresolvedalias('count(1), None)]
+- 'UnresolvedRelation `a_orc_table`

== Analyzed Logical Plan ==
count(1): bigint
Aggregate [count(1) AS count(1)#28L]
+- SubqueryAlias a_orc_table
   +- Relation[a#16,b#17] orc

== Optimized Logical Plan ==
Aggregate [count(1) AS count(1)#28L]
+- Project
   +- Relation[a#16,b#17] orc

== Physical Plan ==
*HashAggregate(keys=[], functions=[count(1)], output=[count(1)#28L])
+- Exchange SinglePartition
   +- *HashAggregate(keys=[], functions=[partial_count(1)], output=[count#30L])
      +- *FileScan orc [] Batched: false, Format: ORC, Location: InMemoryFileIndex[file:/tmp/aa],
PartitionFilters: [], PushedFilters: [], ReadSchema: struct<>
{code}

It seems ORC datasource does not try to read all columns and I just verified this by debugging
with IDE.

> ORC and other columnar formats using HiveShim read all columns when doing a simple count
> ----------------------------------------------------------------------------------------
>
>                 Key: SPARK-18457
>                 URL: https://issues.apache.org/jira/browse/SPARK-18457
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.6.3, 2.0.2
>         Environment: Hadoop 2.7.0
>            Reporter: Andrew Ray
>            Priority: Minor
>
> Doing a `select count(*) from a_orc_table` reads all columns and thus is slower than
a query selecting one like `select count(a_column) from a_orc_table`. Data read can be seen
in the UI (appears to only be accurate for Hadoop 2.5+ based on comment in FileScanRDD.scala
line 80).
> I will create a PR shortly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message