incubator-crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Victor Iacoban <>
Subject extending crunch
Date Thu, 15 Nov 2012 02:18:33 GMT

I'm very interested in writing a wrapper library around Apache Crunch for
Clojure, something similar to existing Scrunch.
How do you recommend to start?

I was looking through Crunch code and it looks like I can pretty easily
integrate it in clojure by adding some custom WritableType type.
Something like WritableType<Object, ByteWritable> with a custom converter
or inputFn/outputFn functions.

Regretfully there are several issues with this approach and instead I'd
have to duplicate all those type classes for a new type set
* WritableType has a package visible constructor so I cannot extend it and
cannot instantiate it
* Converter is instantiated inside WritableType constructor so in case I
need a different converter I'm stuck
* Writables has a factory method for WritableType but it's private
* it looks like there is an attempt to support additional WritableTypes
through EXTENSIONS in Writables but it would only work for cases where in
WritableType<T, W> both T and W are hadoop writables

So what do you think is a best solution, is it possible to open up the api
to support custom WritableTypes or the only option for me is to implement a
new ClojurePType and all related classes?

Hope I'm not too detailed, but at this stage you all are probably very
familiar with the code


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message