Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 4 Oct 2016 07:03:20 +0000 (UTC)
From: "Joseph K. Bradley (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.13005336.1474001349000.728521.1475564600538@Atlassian.JIRA>
In-Reply-To: <JIRA.13005336.1474001349000@Atlassian.JIRA>
References: <JIRA.13005336.1474001349000@Atlassian.JIRA> <JIRA.13005336.1474001349384@arcas>
Subject: [jira] [Commented] (SPARK-17559) PeriodicGraphCheckpointer did not
 persist edges as expected in some cases
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 04 Oct 2016 07:03:22 -0000


    [ https://issues.apache.org/jira/browse/SPARK-17559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15544567#comment-15544567 ] 

Joseph K. Bradley commented on SPARK-17559:
-------------------------------------------

Did I use the correct JIRA username for the "Assignee?"  I tried "ding" but it wasn't working with JIRA admin, but "dingding" worked and has the same email.

> PeriodicGraphCheckpointer did not persist edges as expected in some cases
> -------------------------------------------------------------------------
>
>                 Key: SPARK-17559
>                 URL: https://issues.apache.org/jira/browse/SPARK-17559
>             Project: Spark
>          Issue Type: Bug
>          Components: MLlib
>            Reporter: ding
>            Assignee: dingding
>            Priority: Minor
>             Fix For: 2.0.2, 2.1.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> When use PeriodicGraphCheckpointer to persist graph, sometimes the edge isn't persisted. As currently only when vertices's storage level is none, graph is persisted. However there is a chance vertices's storage level is not none while edges's is none. Eg. graph created by a outerJoinVertices operation, vertices is automatically cached while edges is not. In this way, edges will not be persisted if we use PeriodicGraphCheckpointer do persist.
> See below minimum example:
>    val graphCheckpointer = new PeriodicGraphCheckpointer[Array[String], Int](2, sc)
>     val users = sc.textFile("data/graphx/users.txt")
>       .map(line => line.split(",")).map(parts => (parts.head.toLong, parts.tail))
>     val followerGraph = GraphLoader.edgeListFile(sc, "data/graphx/followers.txt")
>     val graph = followerGraph.outerJoinVertices(users) {
>       case (uid, deg, Some(attrList)) => attrList
>       case (uid, deg, None) => Array.empty[String]
>     }
>     graphCheckpointer.update(graph)    


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org