pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-4845) Parallel instantiation of classes in Tez cause tasks to fail
Date Tue, 22 Mar 2016 10:46:25 GMT

     [ https://issues.apache.org/jira/browse/PIG-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Rohini Palaniswamy updated PIG-4845:
      Resolution: Fixed
    Hadoop Flags: Reviewed
          Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks for the review Daniel.

> Parallel instantiation of classes in Tez cause tasks to fail
> ------------------------------------------------------------
>                 Key: PIG-4845
>                 URL: https://issues.apache.org/jira/browse/PIG-4845
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Rohini Palaniswamy
>            Assignee: Rohini Palaniswamy
>             Fix For: 0.16.0
>         Attachments: PIG-4845-1.patch
> OptionBuilder is not thread-safe
> https://commons.apache.org/proper/commons-cli/javadocs/api-release/org/apache/commons/cli/OptionBuilder.html
> {code}
> This class is NOT thread safe. See CLI-209
> {code}
> HBaseStorage, elephantbird SequenceFileConfig, etc use it in their constructor. This
leads to NoSuchMethodException, UnrecognizedOptionException etc when processor, inputs and
outputs are initialized in parallel in Tez making the task fail. Retry attempts mostly go
through and job might succeed. Need to synchronize the initialization. synchronize would make
it similar to mapreduce though at a slight performance cost. But that should be lot better
than cost of relaunching containers after hitting failures.
> Following two kinds of exception where seen when HBaseStorage and SequenceFileStorage
of elephantbird were used together.
> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: -loadKey
> {code}
> Caused by: java.lang.RuntimeException: Failed to create WritableConverter instance
>         at com.twitter.elephantbird.pig.util.SequenceFileConfig.getWritableConverter(SequenceFileConfig.java:225)
>         at com.twitter.elephantbird.pig.util.SequenceFileConfig.<init>(SequenceFileConfig.java:101)
>         at com.twitter.elephantbird.pig.util.SequenceFileConfig.<init>(SequenceFileConfig.java:115)
>         ... 11 more
> Caused by: java.lang.NoSuchMethodException: com.twitter.elephantbird.pig.util.TextConverter.<init>(java.lang.String)
>         at java.lang.Class.getConstructor0(Class.java:3082)
>         at java.lang.Class.getConstructor(Class.java:1825)
>         at com.twitter.elephantbird.pig.util.SequenceFileConfig.getWritableConverter(SequenceFileConfig.java:213)
>         ... 13 more
> {code}

This message was sent by Atlassian JIRA

View raw message