pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitriy V. Ryaboy (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-1511) Pig removes packages from its own jar when building the JAR to ship to Hadoop
Date Sun, 02 Sep 2012 23:23:07 GMT

     [ https://issues.apache.org/jira/browse/PIG-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Dmitriy V. Ryaboy updated PIG-1511:

    Resolution: Won't Fix
        Status: Resolved  (was: Patch Available)

No movement on this ticket in 2 years, I think it's safe to close this.

(In addition, I can vouch for the fact that making the job jar large can have significant
negative performance consequences. Would recommend you simply register your own job jar via
Pig's "register" command if you need it -- this will save your TTs and, depending on the hadoop
version, your JT, a lot of grief).
> Pig removes packages from its own jar when building the JAR to ship to Hadoop
> -----------------------------------------------------------------------------
>                 Key: PIG-1511
>                 URL: https://issues.apache.org/jira/browse/PIG-1511
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.7.0
>            Reporter: Eric Tschetter
>         Attachments: pig-1511.diff
> Pig generates a new jar file to ship over to Hadoop.  Pig has a couple of packages whitelisted
that it includes from its own jar.  Pig throws away everything else.
> I package all of my dependencies into a single jar file.  Pig is included in this jar
file.  I do it this way because my code needs to run reliably and reproducibly in production.
 Pig throws away all of my dependencies.
> I don't know what the performance gain is of shaving ~5MB off of a jar that is pushed
to a job tracker once and then used to run over 100s of GB of data.  The overhead is minimal
on my cluster.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message