incubator-crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joseph Adler <joseph.ad...@gmail.com>
Subject Re: clojure + crunch = crackle
Date Wed, 28 Nov 2012 19:16:29 GMT
Also interested in this project, but low on time for a few weeks.

One quick bit of feedback: I strongly suspect that there is a way to
eliminate all the type related code from Clojure, probably by using Macros

-- Joe


On Wed, Nov 28, 2012 at 9:07 AM, Matthias Friedrich <matt@mafr.de> wrote:

> Hi Victor,
>
> just for the record, I'm also very interested in this. It's just that
> in the time before christmas, things are really busy at work. But
> I'll definitely play around with crackle.
>
> Thanks,
>   Matthias
>
> On Tuesday, 2012-11-27, Victor Iacoban wrote:
> > Hey Josh,
> >
> > Nice to see some interest, I just pushed from my local repo with several
> > bigger changes. I've separated crackle into 3 parts core, hbase and
> example
> > on my todo list:
> > - jar file assembly, currently I'm using jar command from shell to create
> > the job jar, this obviously needs to be rewritten in order to make
> crackle
> > portable
> > - I need to add support for all sources and targets you have in crunch
> > - need to integrate crunch hbase: sources, targets and types
> >
> > after these are done, some nice to do tasks:
> > - cannot define mr pipelines from clojure REPL, although crackle compiles
> > pipeline classes on the fly it still needs the code to be written to a
> > local file, so it's not as nice as it should be
> > - DSL sucks:
> >  * in current shape you don't have access to PObjects from intermediate
> > steps
> >  * users have to know crunch api very well otherwise they will get
> > confused: what type goes where and why they have to use this particular
> > function type
> >
> > Regards
> >
> > PS I'm also a clojure noob, I did learn common lisp several years ago but
> > playing with clojure only for several months
> >
> >
> > On Mon, Nov 26, 2012 at 11:48 PM, Josh Wills <jwills@cloudera.com>
> wrote:
> >
> > > Victor,
> > >
> > > Just got my own personal fork-- congrats on getting the MR pipeline
> impl
> > > working. What needs doing? Keep in mind that I'm a total clojure n00b,
> > > despite repeated encouragement from lots of developers I respect and
> > > admire.
> > >
> > > Josh
> > >
> > >
> > > On Tue, Nov 20, 2012 at 2:33 PM, Victor Iacoban <
> victor.iacoban@gmail.com
> > > >wrote:
> > >
> > > > Hi,
> > > >
> > > > I have the basics done here:
> > > > https://github.com/viacoban/crackle
> > > >
> > > > It's only MemPipeline for now, still have to build the jar in
> background
> > > > for MRPipeline, but before going there I have a small issue to solve.
> > > >
> > > > So if anyone has written several clojure macroses or know somebody
> who
> > > did
> > > > please write to me directly and we will take it from there
> > > >
> > > > Any comments or input is welcome
> > > >
> > > > Victor
> > > >
> > >
> > >
> > >
> > > --
> > > Director of Data Science
> > > Cloudera <http://www.cloudera.com>
> > > Twitter: @josh_wills <http://twitter.com/josh_wills>
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message