incubator-crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthias Friedrich <m...@mafr.de>
Subject Re: clojure + crunch = crackle
Date Wed, 28 Nov 2012 17:07:32 GMT
Hi Victor,

just for the record, I'm also very interested in this. It's just that
in the time before christmas, things are really busy at work. But
I'll definitely play around with crackle.

Thanks,
  Matthias

On Tuesday, 2012-11-27, Victor Iacoban wrote:
> Hey Josh,
> 
> Nice to see some interest, I just pushed from my local repo with several
> bigger changes. I've separated crackle into 3 parts core, hbase and example
> on my todo list:
> - jar file assembly, currently I'm using jar command from shell to create
> the job jar, this obviously needs to be rewritten in order to make crackle
> portable
> - I need to add support for all sources and targets you have in crunch
> - need to integrate crunch hbase: sources, targets and types
> 
> after these are done, some nice to do tasks:
> - cannot define mr pipelines from clojure REPL, although crackle compiles
> pipeline classes on the fly it still needs the code to be written to a
> local file, so it's not as nice as it should be
> - DSL sucks:
>  * in current shape you don't have access to PObjects from intermediate
> steps
>  * users have to know crunch api very well otherwise they will get
> confused: what type goes where and why they have to use this particular
> function type
> 
> Regards
> 
> PS I'm also a clojure noob, I did learn common lisp several years ago but
> playing with clojure only for several months
> 
> 
> On Mon, Nov 26, 2012 at 11:48 PM, Josh Wills <jwills@cloudera.com> wrote:
> 
> > Victor,
> >
> > Just got my own personal fork-- congrats on getting the MR pipeline impl
> > working. What needs doing? Keep in mind that I'm a total clojure n00b,
> > despite repeated encouragement from lots of developers I respect and
> > admire.
> >
> > Josh
> >
> >
> > On Tue, Nov 20, 2012 at 2:33 PM, Victor Iacoban <victor.iacoban@gmail.com
> > >wrote:
> >
> > > Hi,
> > >
> > > I have the basics done here:
> > > https://github.com/viacoban/crackle
> > >
> > > It's only MemPipeline for now, still have to build the jar in background
> > > for MRPipeline, but before going there I have a small issue to solve.
> > >
> > > So if anyone has written several clojure macroses or know somebody who
> > did
> > > please write to me directly and we will take it from there
> > >
> > > Any comments or input is welcome
> > >
> > > Victor
> > >
> >
> >
> >
> > --
> > Director of Data Science
> > Cloudera <http://www.cloudera.com>
> > Twitter: @josh_wills <http://twitter.com/josh_wills>
> >

Mime
View raw message