incubator-crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Victor Iacoban <victor.iaco...@gmail.com>
Subject Re: clojure + crunch = crackle
Date Tue, 27 Nov 2012 11:56:31 GMT
Hey Josh,

Nice to see some interest, I just pushed from my local repo with several
bigger changes. I've separated crackle into 3 parts core, hbase and example
on my todo list:
- jar file assembly, currently I'm using jar command from shell to create
the job jar, this obviously needs to be rewritten in order to make crackle
portable
- I need to add support for all sources and targets you have in crunch
- need to integrate crunch hbase: sources, targets and types

after these are done, some nice to do tasks:
- cannot define mr pipelines from clojure REPL, although crackle compiles
pipeline classes on the fly it still needs the code to be written to a
local file, so it's not as nice as it should be
- DSL sucks:
 * in current shape you don't have access to PObjects from intermediate
steps
 * users have to know crunch api very well otherwise they will get
confused: what type goes where and why they have to use this particular
function type

Regards

PS I'm also a clojure noob, I did learn common lisp several years ago but
playing with clojure only for several months


On Mon, Nov 26, 2012 at 11:48 PM, Josh Wills <jwills@cloudera.com> wrote:

> Victor,
>
> Just got my own personal fork-- congrats on getting the MR pipeline impl
> working. What needs doing? Keep in mind that I'm a total clojure n00b,
> despite repeated encouragement from lots of developers I respect and
> admire.
>
> Josh
>
>
> On Tue, Nov 20, 2012 at 2:33 PM, Victor Iacoban <victor.iacoban@gmail.com
> >wrote:
>
> > Hi,
> >
> > I have the basics done here:
> > https://github.com/viacoban/crackle
> >
> > It's only MemPipeline for now, still have to build the jar in background
> > for MRPipeline, but before going there I have a small issue to solve.
> >
> > So if anyone has written several clojure macroses or know somebody who
> did
> > please write to me directly and we will take it from there
> >
> > Any comments or input is welcome
> >
> > Victor
> >
>
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message