hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Richard Ding (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1651) PIG class loading mishandled
Date Tue, 28 Sep 2010 22:35:33 GMT

    [ https://issues.apache.org/jira/browse/PIG-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915945#action_12915945
] 

Richard Ding commented on PIG-1651:
-----------------------------------

The problem here is that PigContext uses LogicalPlanBuilder.classloader to instantiate the
LoadFuncs, but the context ClassLoader for the Thread uses a different class loader, and hence
the static variable set for the class loaded by one loader is not visible by the class loaded
by the other loader. The solution is to use the same LogicalPlanBuilder.classloader as the
context ClassLoader for the Thread.

> PIG class loading mishandled
> ----------------------------
>
>                 Key: PIG-1651
>                 URL: https://issues.apache.org/jira/browse/PIG-1651
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.8.0
>            Reporter: Yan Zhou
>            Assignee: Richard Ding
>             Fix For: 0.8.0
>
>
> If just having zebra.jar as being registered in a PIG script but not in the CLASSPATH,
the query using zebra fails since there appear to be multiple classes loaded into JVM, causing
static variable set previously not seen after one instance of the class is created through
reflection. (After the zebra.jar is specified in CLASSPATH, it works fine.) The exception
stack is as follows:
> ackend error message during job submission
> -------------------------------------------
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Unable to create input
splits for: hdfs://hostname/pathto/zebra_dir :: null
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:284)
>         at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:907)
>         at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:801)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:752)
>         at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>         at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
>         at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
>         at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.NullPointerException
>         at org.apache.hadoop.zebra.io.ColumnGroup.getNonDataFilePrefix(ColumnGroup.java:123)
>         at org.apache.hadoop.zebra.io.ColumnGroup$CGPathFilter.accept(ColumnGroup.java:2413)
>         at org.apache.hadoop.zebra.mapreduce.TableInputFormat$DummyFileInputFormat$MultiPathFilter.accept(TableInputFormat.java:718)
>         at org.apache.hadoop.fs.FileSystem$GlobFilter.accept(FileSystem.java:1084)
>         at org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:919)
>         at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:866)
>         at org.apache.hadoop.zebra.mapreduce.TableInputFormat$DummyFileInputFormat.listStatus(TableInputFormat.java:780)
>         at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:246)
>         at org.apache.hadoop.zebra.mapreduce.TableInputFormat.getRowSplits(TableInputFormat.java:863)
>         at org.apache.hadoop.zebra.mapreduce.TableInputFormat.getSplits(TableInputFormat.java:1017)
>         at org.apache.hadoop.zebra.mapreduce.TableInputFormat.getSplits(TableInputFormat.java:961)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:269)
>         ... 7 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message