Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@spark.apache.org
Received-SPF: pass (nike.apache.org: domain of sowen@cloudera.com designates
 209.85.220.175 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <1393938372226-2285.post@n3.nabble.com>
References: <1393938372226-2285.post@n3.nabble.com>
Date: Tue, 4 Mar 2014 13:10:38 +0000
Message-ID: 
 <CAMAsSdLFzoB4RD60gRpA=0yjo5dY7qHDA44+__YMskNiq3ZABg@mail.gmail.com>
Subject: Re: RDD Manipulation in Scala.
From: Sean Owen <sowen@cloudera.com>
To: user@spark.apache.org
Content-Type: text/plain; charset=UTF-8

data.filter(_.split("\t")(1) == "A2")

?
--
Sean Owen | Director, Data Science | London


On Tue, Mar 4, 2014 at 1:06 PM, trottdw <trottdw@gmail.com> wrote:
> Hello, I am using Spark with Scala and I am attempting to understand the
> different filtering and mapping capabilities available.  I haven't found an
> example of the specific task I would like to do.
>
> I am trying to read in a tab spaced text file and filter specific entries.
> I would like this filter to be applied to different "columns" and not lines.
> I was using the following to split the data but attempts to filter by
> "column" afterwards are not working.
> -----------------------------
>    val data = sc.textFile("test_data.txt")
>    var parsedData = data.map( _.split("\t").map(_.toString))
> ------------------------------
>
> To try to give a more concrete example of my goal,
> Suppose the data file is:
> A1    A2     A3     A4
> B1    B2     A3     A4
> C1    A2     C2     C3
>
>
> How would I filter the data based on the second column to only return those
> entries which have A2 in column two?  So, that the resulting RDD would just
> be:
>
> A1    A2     A3     A4
> C1    A2     C2     C3
>
> Is there a convenient way to do this?  Any suggestions or assistance would
> be appreciated.
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/RDD-Manipulation-in-Scala-tp2285.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.