apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Devendra Tagare <devend...@datatorrent.com>
Subject Re: Apex DataFrame
Date Wed, 23 Mar 2016 22:12:49 GMT
Hi,

You can create a case class.Then map the incoming RDD's to the case class &
convert the map to a data frame.By doing this you would have a Dataframe
with the respective fields and associated datatypes set as per the ETL
rules defined before setting the members of the case class.

Sample below (from the zepplin tutorial)

import org.apache.commons.io.IOUtils
import java.net.URL
import java.nio.charset.Charset

// load bank data
val bankText = sc.parallelize(
    IOUtils.toString(
        new URL("
https://s3.amazonaws.com/apache-zeppelin/tutorial/bank/bank.csv"),
        Charset.forName("utf8")).split("\n"))

case class Bank(age: Integer, job: String, marital: String, education:
String, balance: Integer)

val bank = bankText.map(s => s.split(";")).filter(s => s(0) !=
"\"age\"").map(
    s => Bank(s(0).toInt,
            s(1).replaceAll("\"", ""),
            s(2).replaceAll("\"", ""),
            s(3).replaceAll("\"", ""),
            s(5).replaceAll("\"", "").toInt
        )
).toDF()
bank.registerTempTable("bank")

The objects the above code creates are,

bankText: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[32] at
parallelize at <console>:65 defined class Bank bank:
org.apache.spark.sql.DataFrame = [age: int, job: string, marital: string,
education: string, balance: int]

Thanks,
Dev

On Wed, Mar 23, 2016 at 2:36 PM, Ashwin Chandra Putta <
ashwinchandrap@gmail.com> wrote:

> Can you also provide types for fields in the code example? How should a
> fieldname associate itself with a type?
>
> Regards,
> Ashwin.
>
> On Wed, Mar 23, 2016 at 9:32 AM, Amol Kekre <amol@datatorrent.com> wrote:
>
> > Ilya, Brendan,
> > Thanks for driving this feature set. We are looking into it. We should
> have
> > feedback soon.
> >
> > Thks,
> > Amol
> >
> >
> > On Tue, Mar 22, 2016 at 6:27 PM, Ganelin, Ilya <
> > Ilya.Ganelin@capitalone.com>
> > wrote:
> >
> > > Would love the community's input on this idea and thoughts on technical
> > > challenges we may run into when implementing this within Apex.
> > >
> > >
> > >
> > > Sent with Good (www.good.com)
> > > ________________________________
> > > From: Herger, Brendan <Brendan.Herger@capitalone.com>
> > > Sent: Tuesday, March 22, 2016 7:49:51 PM
> > > To: dev@apex.incubator.apache.org
> > > Subject: Apex DataFrame
> > >
> > > Hey all,
> > >
> > > I and the Data Innovation Lab team at Capital One (including Ilya
> > Ganelin,
> > > Brennon York, Dongming Liang and Rikin Shah)  would like to propose the
> > > DataFrame for Apache Apex.
> > >
> > > We feel that a DataFrame object would solve Apex’s need for a
> > standardized
> > > internal data format (schema aware object containing data), and a high
> > > level API for common data operations (integrated with the data store).
> > >
> > > This standardized data storage and data operations allow for rapid
> > > development and iteration on data pipelines, by allowing users to
> > leverage
> > > common code components. It also allows for more standardized workflows,
> > > built on common components with regularized syntax.
> > >
> > > For further details, please see the design document located at:
> > >
> > >
> >
> https://docs.google.com/document/d/13Eidx-Xg5DrLtbVX-UXhpkjt5iwxoZqTUMGhzh48GPw/edit?usp=sharing
> > >
> > > Thanks,
> > > Brendan Herger
> > > Data Scientist, Member of Technical Staff
> > > Data Innovation Lab @ Capital One
> > >
> > > ________________________________________________________
> > >
> > > The information contained in this e-mail is confidential and/or
> > > proprietary to Capital One and/or its affiliates and may only be used
> > > solely in performance of work or services for Capital One. The
> > information
> > > transmitted herewith is intended only for use by the individual or
> entity
> > > to which it is addressed. If the reader of this message is not the
> > intended
> > > recipient, you are hereby notified that any review, retransmission,
> > > dissemination, distribution, copying or other use of, or taking of any
> > > action in reliance upon this information is strictly prohibited. If you
> > > have received this communication in error, please contact the sender
> and
> > > delete the material from your computer.
> > > ________________________________________________________
> > >
> > > The information contained in this e-mail is confidential and/or
> > > proprietary to Capital One and/or its affiliates and may only be used
> > > solely in performance of work or services for Capital One. The
> > information
> > > transmitted herewith is intended only for use by the individual or
> entity
> > > to which it is addressed. If the reader of this message is not the
> > intended
> > > recipient, you are hereby notified that any review, retransmission,
> > > dissemination, distribution, copying or other use of, or taking of any
> > > action in reliance upon this information is strictly prohibited. If you
> > > have received this communication in error, please contact the sender
> and
> > > delete the material from your computer.
> > >
> >
>
>
>
> --
>
> Regards,
> Ashwin.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message