pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Feng Peng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-3222) New UDFContextSignature assignments in Pig 0.11 breaks HCatalog.HCatStorer
Date Tue, 02 Apr 2013 20:11:15 GMT

    [ https://issues.apache.org/jira/browse/PIG-3222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620219#comment-13620219
] 

Feng Peng commented on PIG-3222:
--------------------------------

[~daijy], in your script, can you add one more line after the data load:

{noformat}
data = LOAD 'testdata';
data_limited = limit data 10;
samples = foreach data_limited generate 1 as number;
{noformat}

I got the error before the second job is being submitted:

{noformat}
2013-04-02 20:02:30,715 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2013-04-02 20:02:31,216 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2013-04-02 20:02:34,696 [JobControl] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat
- Total input paths to process : 1
2013-04-02 20:02:34,697 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil
- Total input paths to process : 1
2013-04-02 20:02:34,707 [JobControl] INFO  com.hadoop.compression.lzo.GPLNativeCodeLoader
- Loaded native gpl library
2013-04-02 20:02:34,709 [JobControl] INFO  com.hadoop.compression.lzo.LzoCodec - Successfully
loaded & initialized native-lzo library [hadoop-lzo rev 9ab0565b74e4ac11172c29acf1b398f7aacfb767]
2013-04-02 20:02:34,712 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil
- Total input paths (combined) to process : 1
2013-04-02 20:02:43,467 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201301170859_1557597
2013-04-02 20:02:43,467 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Processing aliases data,data_limited
2013-04-02 20:02:43,468 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- detailed locations: M: data[6,7],data_limited[7,15] C:  R: 
2013-04-02 20:02:43,468 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at: http://hadoop-...
2013-04-02 20:03:12,922 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 25% complete
2013-04-02 20:03:24,603 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 25% complete
2013-04-02 20:03:29,049 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 50% complete
2013-04-02 20:03:39,383 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script
settings are added to the job
2013-04-02 20:03:39,384 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-04-02 20:03:39,386 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting Parallelism to 1
2013-04-02 20:03:41,439 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- creating jar file Job1765053817644653272.jar
2013-04-02 20:03:45,563 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- jar file Job1765053817644653272.jar created
2013-04-02 20:03:45,577 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1115: Schema
for data cannot be determined.
Details at logfile: /home/feng/work/log/pig_1364932938335.log
2013-04-02 20:03:45,642 [Thread-5] INFO  org.apache.hcatalog.common.HiveClientCache - Cleaning
up hive client cache in ShutDown hook
{noformat}


                
> New UDFContextSignature assignments in Pig 0.11 breaks HCatalog.HCatStorer 
> ---------------------------------------------------------------------------
>
>                 Key: PIG-3222
>                 URL: https://issues.apache.org/jira/browse/PIG-3222
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: 0.11
>            Reporter: Feng Peng
>              Labels: hcatalog
>         Attachments: hcatstorer.trace.txt, hcat.trace, PigStorerDemo.java
>
>
> Pig 0.11 assigns different UDFContextSignature for different invocations of the same
load/store statement. This change breaks the HCatStorer which assumes all front-end and back-end
invocations of the same store statement has the same UDFContextSignature so that it can read
the previously stored information correctly.
> The related HCatalog code is in https://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.5/hcatalog-pig-adapter/src/main/java/org/apache/hcatalog/pig/HCatStorer.java
(the setStoreLocation() function).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message