spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <>
Subject Re: spork?
Date Wed, 22 Jan 2014 19:43:28 GMT
Yeah, this is true for new jobs, but some organizations have large libraries of Pig scripts
and UDFs. I think what would be ideal is a Pig-to-Scala or Pig-to-Python binding layer similar
to Shark’s sql2rdd. That way you could run your existing Pig scripts, but also call them
from Scala or Python to transition over to these programming languages.


On Jan 22, 2014, at 11:12 AM, Paul Brown <> wrote:

> Just an IMHO.  We've moved a number of Pig jobs to Spark jobs, and the
> benefits of a genuine conversion (type safety, terseness, debugging, lack
> of gratuitous shells and shims around UDFs, ...) are sufficient that a
> rewrite is probably preferable from an adapter unless you have an enormous
> Pig investment that you want to continue investing in.
> -- Paul
> —
> | Multifarious, Inc. |
> On Wed, Jan 22, 2014 at 9:07 AM, Tom Graves <> wrote:
>> Hey everyone,
>> At one point I heard of pig on Spark (Spork).  The only thing I could find
>> on it was and it doesn't look like there
>> has been any work on it in quite a while.
>> Does anyone know the current status of that or has tried it?
>> Thanks,
>> Tom

View raw message