hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@veoh.com>
Subject Re: Groovy Scripting for Hadoop
Date Tue, 06 May 2008 00:26:10 GMT

Chris,

I have been meaning to write to you.

Have you seen my grool system which allows simple MR programs to be written
simply?

I have been thinking for some time that it would make a good match with
Cascades.

In addition, I have been working on a layer over Zookeeper to handle
collection of data feed oriented information about availability of files
containing data.  This is similar in some sense to Amazon's simple queue
service except that it describes content as files, rather than opaque blobs
passed through queues.  This allows simpler retrospective processing of
data.  It would make a very good substrate for something like Cascades since
it would allow clean coordination semantics between multiple workers on
independent machines as well as provide notification of new (if desired)
without polling.  That would allow much lower latency systems to be built.




On 5/5/08 5:09 PM, "Chris K Wensel" <chris@wensel.net> wrote:

> Hey all
> 
> Just wanted to let interested parties know we just released 0.1.0 of
> our Groovy 'builder' extension.
> 
> We think this will be a great tool for those groups that need to
> expose Hadoop to the 'casual' user who needs to get and manipulate
> valuable data on a Hadoop cluster, but doesn't have the time to learn
> Java, the Hadoop API, or to think in MapReduce to solve problems that
> are a notch or more above trivial.
> 
> It is worthy of mentioning here that no Groovy code is run in the
> cluster (on the slave nodes). Groovy is only being used as a
> configuration language to allow for the assembly of complex workflows
> to be run on Hadoop.
> 
> An introduction:
> http://www.cascading.org/documentation/groovy.html
> 
> Links to samples included in the distro:
> 
> The canonical word count example (.groovy)
> http://tinyurl.com/6gp8xp
> 
> Or a "wide finder" example:
> http://tinyurl.com/5korhj
> 
> These examples don't show it, but splits and joins are fully supported
> (as they are in Cascading). Further, local libraries can be used, but
> there is still work to do to make this transparent.
> 
> Please feel free to join our mail-list and post feedback.
> http://groups.google.com/group/cascading-user
> 
> cheers,
> ckw
> 
> Chris K Wensel
> chris@wensel.net
> http://chris.wensel.net/
> http://www.cascading.org/
> 
> 
> 
> 


Mime
View raw message