spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-6738) EstimateSize is difference with spill file size
Date Tue, 07 Apr 2015 10:44:16 GMT

    [ https://issues.apache.org/jira/browse/SPARK-6738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14482987#comment-14482987
] 

Sean Owen commented on SPARK-6738:
----------------------------------

Is that the only file spilled though? I'm not an expert but it looks like lots of files are
spilled to here.

> EstimateSize  is difference with spill file size
> ------------------------------------------------
>
>                 Key: SPARK-6738
>                 URL: https://issues.apache.org/jira/browse/SPARK-6738
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.2.0
>            Reporter: Hong Shen
>
> ExternalAppendOnlyMap spill 1100M data to disk:
> 15/04/07 16:39:48 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling in-memory
map of 1106.5 MB to disk (12 times so far)
> /data11/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-994b/30/temp_local_e4347165-6263-4678-9f1d-67ad4bcd8fb5
> 15/04/07 16:39:49 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling in-memory
map of 1106.3 MB to disk (13 times so far)
> /data6/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-1e29/26/temp_local_76f9900b-1b3d-4cef-b3a2-6afcde14bbd9
> 15/04/07 16:39:49 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling in-memory
map of 1105.8 MB to disk (14 times so far)
> /data7/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-f883/26/temp_local_3ade0aec-ac1d-469d-bc99-b6fa87cb649b
> 15/04/07 16:39:50 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling in-memory
map of 1106.8 MB to disk (15 times so far)
> But the file size is only 1.1M.
> [tdwadmin@tdw-10-215-149-231 ~/tdwenv/tdwgaia/logs/container-logs/application_1423737010718_40308573/container_1423737010718_40308573_01_000008]$
ll -h /data7/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-f883/26/temp_local_3ade0aec-ac1d-469d-bc99-b6fa87cb649b
> -rw-r----- 1 spark users 1.1M Apr  7 16:39 /data7/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-f883/26/temp_local_3ade0aec-ac1d-469d-bc99-b6fa87cb649b
> The estimateSize  is hugh difference with spill file size



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message