Mailing-List: contact dev-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
Date: Wed, 15 Jul 2015 12:31:28 -0500
Message-ID: 
 <CADtDQQK099=czsQMUi-YVwL=awC7Jihh0MoTKhB0rBBc-8MKFg@mail.gmail.com>
Subject: Record metadata with RDDs and DataFrames
From: RJ Nowling <rnowling@gmail.com>
To: "dev@spark.apache.org" <dev@spark.apache.org>
Content-Type: multipart/alternative; boundary=001a113dfc3a2c94f4051aed53f6

--001a113dfc3a2c94f4051aed53f6
Content-Type: text/plain; charset=UTF-8

Hi all,

I'm working on an ETL task with Spark.  As part of this work, I'd like to
mark records with some info such as:

1. Whether the record is good or bad (e.g, Either)
2. Originating file and lines

Part of my motivation is to prevent errors with individual records from
stopping the entire pipeline.  I'd also like to filter out and log bad
records at various stages.

I could use RDD[Either[T]] for everything but that won't work for
DataFrames.  I was wondering if anyone has had a similar situation and if
they found elegant ways to handle this?

Thanks,
RJ

--001a113dfc3a2c94f4051aed53f6
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi all,<div><br></div><div>I&#39;m working on an ETL task =
with Spark.=C2=A0 As part of this work, I&#39;d like to mark records with s=
ome info such as:</div><div><br></div><div>1. Whether the record is good or=
 bad (e.g, Either)</div><div>2. Originating file and lines</div><div><br></=
div><div>Part of my motivation is to prevent errors with individual records=
 from stopping the entire pipeline.=C2=A0 I&#39;d also like to filter out a=
nd log bad records at various stages.</div><div><br></div><div>I could use =
RDD[Either[T]] for everything but that won&#39;t work for DataFrames.=C2=A0=
 I was wondering if anyone has had a similar situation and if they found el=
egant ways to handle this? =C2=A0</div><div><br></div><div>Thanks,</div><di=
v>RJ</div></div>

--001a113dfc3a2c94f4051aed53f6--