spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reynold Xin (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-9699) Test whether we should turn compression on or off by default for external sorting
Date Fri, 11 Sep 2015 07:33:45 GMT

     [ https://issues.apache.org/jira/browse/SPARK-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Reynold Xin resolved SPARK-9699.
--------------------------------
    Resolution: Fixed
      Assignee: Reynold Xin

I tested this locally and it is unclear when it should be turned on or off, because in some
cases it is still faster with it on. Safer to just keep it on for now.


> Test whether we should turn compression on or off by default for external sorting
> ---------------------------------------------------------------------------------
>
>                 Key: SPARK-9699
>                 URL: https://issues.apache.org/jira/browse/SPARK-9699
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>            Reporter: Reynold Xin
>            Assignee: Reynold Xin
>
> As of Spark 1.4, compression is on by default for spilling, trading off CPU cycles for
less I/O throughput usage. It is unclear whether it is better to have compression on or off
when we spill data. We should test that and adjust the default setting accordingly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message