spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-22540) HighlyCompressedMapStatus's avgSize is incorrect
Date Fri, 17 Nov 2017 13:55:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-22540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Owen resolved SPARK-22540.
-------------------------------
       Resolution: Fixed
    Fix Version/s: 2.2.2
                   2.3.0

> HighlyCompressedMapStatus's avgSize is incorrect
> ------------------------------------------------
>
>                 Key: SPARK-22540
>                 URL: https://issues.apache.org/jira/browse/SPARK-22540
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.3.0
>            Reporter: yucai
>            Assignee: yucai
>             Fix For: 2.3.0, 2.2.2
>
>
> The calculation of HighlyCompressedMapStatus's avgSize is incorrect. 
> Currently, it looks like "sum of small blocks / count of all non empty blocks", the count
of all non empty blocks not only contains small blocks, which contains huge blocks number
also, but we need the count of small blocks only.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message