spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bxshi <...@git.apache.org>
Subject [GitHub] spark pull request: [GraphX][SPARK-2245] add a materialize method ...
Date Wed, 25 Feb 2015 23:04:39 GMT
Github user bxshi commented on the pull request:

    https://github.com/apache/spark/pull/1177#issuecomment-76080970
  
    It's not as simple as I thought...
    
    Here is my reply on JIRA about this PR
    
    I edited my original comment to add the updates, but I do not know if you can get them
via email. So I resubmit it again. Hope that won't bother you. Ankur Dave
    Hi Ankur Dave, I changed my pull request. But there is another exception, ShippableVertexPartition
is not serializable. So I serialized it, but there is another exception org.apache.spark.graphx.impl.RoutingTablePartition
is not serializable. Then I serialized it again, but on iteration 2 there will be an exception:
org.apache.spark.graphx.impl.ShippableVertexPartition cannot be cast to scala.Tuple2
    The code I'm using are:
    ```
    val conf = new SparkConf().setAppName("HDTM")
    .setMaster("local[4]")
    val sc = new SparkContext(conf)
    sc.setCheckpointDir("./checkpoint")
    val v = sc.parallelize(Seq[(VertexId, Long)]((0L, 0L), (1L, 1L), (2L, 2L)))
    val e = sc.parallelize(Seq[Edge[Long]](Edge(0L, 1L, 0L), Edge(1L, 2L, 1L), Edge(2L, 0L,
2L)))
    var g = Graph(v, e)
    val vertexIds = Seq(0L, 1L, 2L)
    var prevG: Graph[VertexId, Long] = null
    for (i <- 1 to 2000) {
    vertexIds.toStream.foreach(id =>
    { prevG = g g = Graph(g.vertices, g.edges) g.vertices.cache() g.edges.cache() prevG.unpersistVertices(blocking
= false) prevG.edges.unpersist(blocking = false) }
    )
    g.vertices.checkpoint()
    g.edges.checkpoint()
    g.edges.count()
    g.vertices.count()
    println(s"$
    {g.vertices.isCheckpointed}
    $
    {g.edges.isCheckpointed}
    ")
    println(" iter " + i + " finished")
    }
    println(g.vertices.collect().mkString(" "))
    println(g.edges.collect().mkString(" "))
    ```
    Am I on the right track? Or Should there be another way to change it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message