pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Coveney <jcove...@gmail.com>
Subject Re: Fixing a broken dependency // can we include a patched piece of JRuby source code in Pig?
Date Fri, 23 Mar 2012 21:50:11 GMT

My idea wasn't to have a patched version included with Pig, just one
specific file that needed to be patched, that way it would be earlier in
the classpath than the version in JRuby. If you think it is cleaner to just
have a patched version of JRuby that we depend on, it would accomplish the
same thing. But I definitely agree that we shouldn't include JRuby itself
with Pig. But right now I have a file at src/org/jruby/path/to/file.java,
and since there will be this lone JRuby file, in the case that it gets
called, we are good


The issue only cropped up because, in a sense, another bug was hiding it. I
thought 1.9 was the default version of Ruby that JRuby intepreted, but I
was wrong (1.8 was the default). I fixed this, and then the issue cropped

For people who aren't rubyists and don't know much about Ruby versions:
1.8.7 is the equivalent of Python 2.7, and 1.9.2 is the equivalent of
Python 3.3, the difference being that most everyone with new projects uses
1.9.2 because it is better in many ways, and didn't break the language like
Python did (this is where the analogy breaks down). Another way to think of
it might be that we want to use Pig 0.9 (1.9) instead of Pig 0.8 (1.8),
especially because 1.9 is getting all of the active development, so in the
future most bug fixes are going to be for 1.9 etc. That said, the 1.9 piece
has this one bug (much as people using Pig 0.9 instead of Pig 0.8 ran into
some weird bugs b/c of the parser changes).

The bug is as follows:
If you have a file on your classpath, and then "require" a resource with
that exact same name, you get a String index error. This crops up when,
internally to Pig, it loads the pigudf.rb file into the ruby runtime on the
same JVM. pig.jar is on the classpath, but the script has to 'require
pig.jar' because it needs to know about things like DataBags and whatnot.
This causes an error because of a subtle bug that rarely comes up in pure
JRuby projects, since pure JRuby projects usually don't set the classpath
on the jvm level. This is a pretty difficult bug to workaround: pig.jar has
to be on the classpath, and requiring it in ruby is also necessary from
within pig.

Either way, the fix is trivial, but 1.7.0 won't be out for a month or two,
and in the meantime, it's pretty bad. I can try to come up with some weird
hack workaround, but depending on a fork of JRuby seems like it would be a
lot more reasonable. Having the one .java file that needs to be fixed is
also easy, but might be ugly. I personally would really prefer to just go
with the fixed version, because the workaround is going to be pretty
difficult (since the error is baked deep in there) and ugly. If we have a
forked dependency, when 1.7.0 comes out we can just depend on that and bam.

Sorry if that's a bit long, I just wanted to thoroughly document the issue.

2012/3/23 Alan Gates <gates@hortonworks.com>

> Won't a lot of people already have their version of JRuby and not want a
> special one?  I'm fine with having a patched version on github and
> referring it in our release notes.  I'm not wild about including a version
> of JRuby with Pig, for both licensing reasons and because our tar file is
> bloated enough as it is.
> Alan.
> On Mar 23, 2012, at 11:38 AM, Daniel Dai wrote:
> > Hi, Jonathan,
> > What bug is it? Last time when I try, it seems work well for me. We
> > can leave a small hole and describe the limitation clearly in release
> > notes/code comments/javadocs, we can also provide a link to the ticket
> > tracking the issue. I remember we did something similar for javacc
> > before. However, I don't think we shall include a JRuby patch in Pig.
> >
> > Daniel
> >
> > On Fri, Mar 23, 2012 at 10:01 AM, Jonathan Coveney <jcoveney@gmail.com>
> wrote:
> >> First off: JRuby patch is almost done. It's passing tests, I have some
> more
> >> to add, but I think the definitive version to work off will be out today
> >> (assuming we can reconcile what follows :)
> >>
> >> I hit a bug in JRuby that is pretty impossible to avoid (it's a bug in
> the
> >> way files were found on the classpath). I figured out the bug and let
> the
> >> JRuby devs know and they patched master, but that means that our
> version is
> >> still buggy. I put a patched version of the file in the Pig project
> pending
> >> a new JRuby release, and this works, but there are two issues:
> >> 1) Is this how we want this to be structued? It's weird to have this
> random
> >> file in there, but on the other hand, it's a clean and clear fix.
> >> 2) Is this legal? JRuby has a kind of odd triple license and I think you
> >> can choose 1 for pieces that aren't explicitly GPL (of which there are
> very
> >> few). One of those licenses is the CPL, which Apache says is kosher as
> long
> >> as you're explicit, but I don't know. Is this fine? Should I talk to
> JRuby
> >> or Apache legal?
> >>
> >> I suppose the alternative would be to publish a patched version of JRuby
> >> (we could fork it on Github) and depend on that.
> >>
> >> I appreciate your comments
> >> Jon

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message