crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: Spark Examples
Date Mon, 12 Jan 2015 22:57:40 GMT
Hey Danushka,

The spark branch on my crunch-demo app has a Spark example as well:

https://github.com/jwills/crunch-demo/tree/spark

I/you would need to update the POM to work against version 0.11.0-hadoop2
and the latest Spark release, either Spark 1.1 or 1.2 should work fine.

J

On Mon, Jan 12, 2015 at 2:27 PM, Danushka Menikkumbura <
danushka.menikkumbura@gmail.com> wrote:

> Thanks Jinal!
>
> On Mon, Jan 12, 2015 at 4:43 PM, Jinal Shah <jinalshah2007@gmail.com>
> wrote:
>
> > Hi Danushka,
> >
> > The user guide might be a good resource to start with
> > https://crunch.apache.org/user-guide.html#sparkpipeline
> >
> > You can use the Crunch WordCount example:
> >
> >
> https://github.com/apache/crunch/blob/master/crunch-archetype/src/main/resources/archetype-resources/src/main/java/WordCount.java
> >
> > These are some blogs you can use to learn
> > http://crunch-tutor.blogspot.com/2014/12/crunch-dofn-example.html
> > http://engineering.cerner.com/blog/scaling-people-with-apache-crunch/
> >
> >
> > Thanks
> > Jinal
> >
> > On Mon, Jan 12, 2015 at 1:19 PM, Danushka Menikkumbura <
> > danushka.menikkumbura@gmail.com> wrote:
> >
> > > Hi devs,
> > >
> > > I am trying to understand the behavior of Crunch with Spark. In order
> to
> > do
> > > that I need to run an example so that I can step through the code.
> Could
> > > someone point me to any resources that you think would be helpful
> > please?.
> > >
> > > Thanks,
> > > Danushka
> > >
> >
>



-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message