spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ilya Ganelin (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-4927) Spark does not clean up properly during long jobs.
Date Wed, 31 Dec 2014 00:34:14 GMT

    [ https://issues.apache.org/jira/browse/SPARK-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14261724#comment-14261724
] 

Ilya Ganelin edited comment on SPARK-4927 at 12/31/14 12:33 AM:
----------------------------------------------------------------

The below code can produce this issue. I've also included some log output.
{code: scala}
def showMemoryUsage(sc : SparkContext) = {
  
  val usersPerStep = 25000
  val count = 1000000
  val numSteps = count/usersPerStep
  
  val users = sc.parallelize(1 to count)
  val zippedUsers = users.zipWithIndex().cache()
  val userFeatures : RDD[(Int, Int)] = sc.parallelize(1 to count).map(s=>(s,2)).cache()
  val productFeatures : RDD[(Int, Int)] = sc.parallelize(1 to 50000000)
    .map(s => (s, 4)).cache()
  
  for (i <- 1 to numSteps) {
    val usersFiltered = zippedUsers.filter(s => {
      ((i - 1) * usersPerStep <= s._2) && (s._2 < i * usersPerStep)
    }).map(_._1).collect()
    
    usersFiltered.foreach(user => {
      val mult = productFeatures.map(s => s._2 * userFeatures.lookup(user).head)
      mult.takeOrdered(20)
      
      // Normally this would then be written to disk
      // For the sake of the example this is all we're doing
    })
  }
}

{code}
Example broadcast variable added:
14/12/30 19:25:19 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on CLIENT_NODE
(size: 794.0 B, free: 441.9 MB)

And then if I parse the entire log looking for “free : XXX.X MB” I see the available memory
slowly ticking away:
Free 441.8 MB
Free 441.8 MB
Free 441.8 MB
Free 441.8 MB
Free 441.8 MB
Free 441.8 MB
Free 441.8 MB
…
Free 441.7 MB
Free 441.7 MB
Free 441.7 MB
Free 441.7 MB
And so on.

Clearly the above code is not persisting the intermediate RDD (mult), yet memory is never
being properly freed up.



was (Author: ilganeli):
The below code can produce this issue. I've also included some log output.
{code: java}
def showMemoryUsage(sc : SparkContext) = {
  
  val usersPerStep = 25000
  val count = 1000000
  val numSteps = count/usersPerStep
  
  val users = sc.parallelize(1 to count)
  val zippedUsers = users.zipWithIndex().cache()
  val userFeatures : RDD[(Int, Int)] = sc.parallelize(1 to count).map(s=>(s,2)).cache()
  val productFeatures : RDD[(Int, Int)] = sc.parallelize(1 to 50000000)
    .map(s => (s, 4)).cache()
  
  for (i <- 1 to numSteps) {
    val usersFiltered = zippedUsers.filter(s => {
      ((i - 1) * usersPerStep <= s._2) && (s._2 < i * usersPerStep)
    }).map(_._1).collect()
    
    usersFiltered.foreach(user => {
      val mult = productFeatures.map(s => s._2 * userFeatures.lookup(user).head)
      mult.takeOrdered(20)
      
      // Normally this would then be written to disk
      // For the sake of the example this is all we're doing
    })
  }
}

{code}
Example broadcast variable added:
14/12/30 19:25:19 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on CLIENT_NODE
(size: 794.0 B, free: 441.9 MB)

And then if I parse the entire log looking for “free : XXX.X MB” I see the available memory
slowly ticking away:
Free 441.8 MB
Free 441.8 MB
Free 441.8 MB
Free 441.8 MB
Free 441.8 MB
Free 441.8 MB
Free 441.8 MB
…
Free 441.7 MB
Free 441.7 MB
Free 441.7 MB
Free 441.7 MB
And so on.

Clearly the above code is not persisting the intermediate RDD (mult), yet memory is never
being properly freed up.


> Spark does not clean up properly during long jobs. 
> ---------------------------------------------------
>
>                 Key: SPARK-4927
>                 URL: https://issues.apache.org/jira/browse/SPARK-4927
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.1.0
>            Reporter: Ilya Ganelin
>
> On a long running Spark job, Spark will eventually run out of memory on the driver node
due to metadata overhead from the shuffle operation. Spark will continue to operate, however
with drastically decreased performance (since swapping now occurs with every operation).
> The spark.cleanup.tll parameter allows a user to configure when cleanup happens but the
issue with doing this is that it isn’t done safely, e.g. If this clears a cached RDD or
active task in the middle of processing a stage, this ultimately causes a KeyNotFoundException
when the next stage attempts to reference the cleared RDD or task.
> There should be a sustainable mechanism for cleaning up stale metadata that allows the
program to continue running. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message