Return-Path: X-Original-To: apmail-spark-dev-archive@minotaur.apache.org Delivered-To: apmail-spark-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 75FC310C1E for ; Fri, 25 Jul 2014 20:03:55 +0000 (UTC) Received: (qmail 74207 invoked by uid 500); 25 Jul 2014 20:03:55 -0000 Delivered-To: apmail-spark-dev-archive@spark.apache.org Received: (qmail 74140 invoked by uid 500); 25 Jul 2014 20:03:54 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@spark.apache.org Delivered-To: mailing list dev@spark.apache.org Received: (qmail 74126 invoked by uid 99); 25 Jul 2014 20:03:54 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jul 2014 20:03:54 +0000 X-ASF-Spam-Status: No, hits=1.5 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ankurdave@gmail.com designates 209.85.192.46 as permitted sender) Received: from [209.85.192.46] (HELO mail-qg0-f46.google.com) (209.85.192.46) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jul 2014 20:03:50 +0000 Received: by mail-qg0-f46.google.com with SMTP id z60so5670302qgd.19 for ; Fri, 25 Jul 2014 13:03:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=opYkTEBa/oIEkfwmjPd4ZunxLl/CKBYipC+8KVFOE/Q=; b=krl5jepXtatzheTwqeX/hSD43zMJKYXceD6B6leDtzYPax3b8x/dd5eqwPIleeQblD GArdL8eIftKoadpa9/3om6qkkGT+yb1Lv0BrA0M6sb4fD/whOhj2ZixF0nYTmT5zmeXg Fw1sQYpIda7vgl/SmPbBYVMTxvKOJbIX8euAmqN7HLQx7wWngBpXg9vKHmVY6BAga/Z3 moA4L6jABKD4JMU4RPq5C2EBvYOvb9r1OvaqaFmeyKe+czlyFvu563QdUHwAjRtWpn7z O//DjVwYfmQKUBp8uQkYivNDzQWmbsU+RGbbZoDkS9lj9HkygqEUszKRzc+YZf5TGJEq 8xHA== X-Received: by 10.224.55.202 with SMTP id v10mr31749866qag.10.1406318609466; Fri, 25 Jul 2014 13:03:29 -0700 (PDT) MIME-Version: 1.0 Received: by 10.140.37.132 with HTTP; Fri, 25 Jul 2014 13:03:09 -0700 (PDT) In-Reply-To: References: <53D0AECE.2070608@sjtu.edu.cn> From: Ankur Dave Date: Fri, 25 Jul 2014 13:03:09 -0700 Message-ID: Subject: Re: GraphX graph partitioning strategy To: dev@spark.apache.org Content-Type: multipart/alternative; boundary=001a11c3029a288dde04ff0a1187 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c3029a288dde04ff0a1187 Content-Type: text/plain; charset=UTF-8 Oops, the code should be: val unpartitionedGraph: Graph[Int, Int] = ...val numPartitions: Int = 128 def getTripletPartition(e: EdgeTriplet[Int, Int]): PartitionID = ... // Get the triplets using GraphX, then use Spark to repartition themval partitionedEdges = unpartitionedGraph.triplets .map(e => (getTripletPartition(e), e)) .partitionBy(new HashPartitioner(numPartitions)) *.map(pair => Edge(pair._2.srcId, pair._2.dstId, pair._2.attr))* val partitionedGraph = Graph(unpartitionedGraph.vertices, partitionedEdges) Ankur --001a11c3029a288dde04ff0a1187--