Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CA27C176CF for ; Wed, 8 Oct 2014 19:03:07 +0000 (UTC) Received: (qmail 3005 invoked by uid 500); 8 Oct 2014 19:03:05 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 2936 invoked by uid 500); 8 Oct 2014 19:03:05 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 2910 invoked by uid 99); 8 Oct 2014 19:03:05 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Oct 2014 19:03:05 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of codedeft@gmail.com designates 209.85.218.42 as permitted sender) Received: from [209.85.218.42] (HELO mail-oi0-f42.google.com) (209.85.218.42) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Oct 2014 19:02:39 +0000 Received: by mail-oi0-f42.google.com with SMTP id a141so8697558oig.29 for ; Wed, 08 Oct 2014 12:02:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:from:date:message-id:subject:to:content-type; bh=Rv0SgEI0dMTdprzFUC2U35w8u7ltv5gXjgoNBhzGB+E=; b=DQa4G25YTGZLehTC0K8SX6j+lNM2Anmwfh02vnEIDBUt7VaqQn8xd3ITbIFRM6eydK kJbeYWzBXWuLYSGMMQ5Mx3CNkWACW0VxMQRWUPF5jmRYUHATXdax05r4Jpo2ldpkvpiX LtzTPybYQSW4INbIQeGrfykPUt/bSnlhTt+Y1hl46CopuevZA/+Fr2BwqbQrQrskLGkU wicnFyFoUr44KpkNc7gPqjXiRe/L9dypyX4nQ8rT7DDK2vesqRv5bmnlYjRXUrajN4V4 U8BrZkhn3m6XR1DjpDz6hNbCZBg1ye3J9LhCa0RA8didBJJNMG/YNoU+J9u7T67wxJ3N dRCA== X-Received: by 10.182.163.114 with SMTP id yh18mr15134426obb.0.1412794957551; Wed, 08 Oct 2014 12:02:37 -0700 (PDT) MIME-Version: 1.0 Sender: codedeft@gmail.com Received: by 10.76.189.43 with HTTP; Wed, 8 Oct 2014 12:01:57 -0700 (PDT) From: Sung Hwan Chung Date: Wed, 8 Oct 2014 12:01:57 -0700 X-Google-Sender-Auth: 2sNDZGdC8T5KIcr1lgR9dtVDPv0 Message-ID: Subject: Is there a way to look at RDD's lineage? Or debug a fault-tolerance error? To: user@spark.apache.org Content-Type: multipart/alternative; boundary=e89a8f83aa2595dbc70504edf566 X-Virus-Checked: Checked by ClamAV on apache.org --e89a8f83aa2595dbc70504edf566 Content-Type: text/plain; charset=UTF-8 My job is not being fault-tolerant (e.g., when there's a fetch failure or something). The lineage of RDDs are constantly updated every iteration. However, I think that when there's a failure, the lineage information is not being correctly reapplied. It goes something like this: val rawRDD = read(...) val repartRDD = rawRDD.repartition(X) val tx1 = repartRDD.map(...) var tx2 = tx1.map(...) while (...) { tx2 = tx1.zip(tx2).map(...) } Is there any way to monitor RDD's lineage, maybe even including? I want to make sure that there's no unexpected things happening. --e89a8f83aa2595dbc70504edf566 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
My job is not being fault-tolerant (e.g., when there's= a fetch failure or something).

The lineage of RDDs are = constantly updated every iteration. However, I think that when there's = a failure, the lineage information is not being correctly reapplied.
<= div>
It goes something like this:

va= l rawRDD =3D read(...)
val repartRDD =3D rawRDD.repartition(X)

val tx1 =3D repartRDD.map(...)
var tx2 =3D= tx1.map(...)

while (...) {
=C2=A0 tx2 = =3D tx1.zip(tx2).map(...)
}


Is there any way to monitor RDD's lineage, maybe even including? I w= ant to make sure that there's no unexpected things happening.
--e89a8f83aa2595dbc70504edf566--