hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pradeep Kamath (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-443) Illustrate for the Types branch
Date Tue, 23 Sep 2008 23:01:44 GMT

    [ https://issues.apache.org/jira/browse/PIG-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12633958#action_12633958
] 

Pradeep Kamath commented on PIG-443:
------------------------------------

- new patch is broken - does not contain the files under src/org/apache/pig/backend/local

- The patch gives a Null POinter Exception when used with a hadoop cluster. I wonder if this
is breaking backward compatibility with illustrate in 1.4 - is it worthwhile supporting reading
data from dfs and using for illustrate. At the least, we need to give an error message if
this is not supported.
{noformat}
java -cp pig.jar:<location of hadoopsite.xml> org.apache.pig.Main
grunt> a = load 'st10k' as (name, age, gpa);
grunt> illustrate a;
2008-09-23 15:01:21,948 [main] ERROR org.apache.pig.pen.ExampleGenerator - Error reading data
2008-09-23 15:01:21,949 [main] ERROR org.apache.pig.tools.grunt.GruntParser - java.lang.NullPointerException
        at org.apache.pig.pen.EquivalenceClasses.GetEquivalenceClasses(EquivalenceClasses.java:73)
        at org.apache.pig.pen.DerivedDataVisitor.visit(DerivedDataVisitor.java:242)
        at org.apache.pig.impl.logicalLayer.LOLoad.visit(LOLoad.java:145)
        at org.apache.pig.impl.logicalLayer.LOLoad.visit(LOLoad.java:34)
        at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
        at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
        at org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:98)
        at org.apache.pig.pen.LineageTrimmingVisitor.<init>(LineageTrimmingVisitor.java:90)
        at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:100)
        at org.apache.pig.PigServer.getExamples(PigServer.java:627)
        at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:278)
        at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:183)
        at org.apache.pig.tools.grunt.GruntParser.parseContOnError(GruntParser.java:93)
        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:58)
        at org.apache.pig.Main.main(Main.java:282)

2008-09-23 15:01:21,949 [main] ERROR org.apache.pig.tools.grunt.GruntParser -
2008-09-23 15:01:21,949 [main] ERROR org.apache.pig.tools.grunt.GruntParser - java.lang.NullPointerException
{noformat}

- If schema is not supplied like you said, it gives an exception asking for a schema. Isn't
this too restrictive? Is it possible to sample and generate data based on real user input
without considering the schema?

- In local mode, cogroup seems like cause an issue:
{noformat}
[/tmp/types]java -Xmx1024m -cp pig.jar org.apache.pig.Main
2008-09-23 15:29:38,477 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine
- Connecting to hadoop file system at: file:///
2008-09-23 15:29:38,689 [main] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing
JVM Metrics with processName=JobTracker, sessionId=
grunt> a = load '/home/pradeepk/work/pig/simpletest';
grunt> b = load '/home/pradeepk/work/pig/simpletest';
grunt> c = group a by $0, b by $0;                   
grunt> illustrate c;                                 
2008-09-23 15:29:52,829 [main] ERROR org.apache.pig.pen.AugmentBaseDataVisitor - Error visiting
Cogroup during Augmentation phase of Example Generator! null
2008-09-23 15:29:52,833 [main] ERROR org.apache.pig.pen.ExampleGenerator - Visitor exception
while creating example data Error visiting Cogroup during Augmentation phase of Example Generator!
null
java.lang.NullPointerException
        at org.apache.pig.pen.util.DisplayExamples.MakeArray(DisplayExamples.java:184)
        at org.apache.pig.pen.util.DisplayExamples.PrintTabular(DisplayExamples.java:86)
        at org.apache.pig.pen.util.DisplayExamples.PrintTabular(DisplayExamples.java:80)
        at org.apache.pig.pen.util.DisplayExamples.PrintTabular(DisplayExamples.java:69)
        at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:138)
        at org.apache.pig.PigServer.getExamples(PigServer.java:627)
        at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:278)
        at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:183)
        at org.apache.pig.tools.grunt.GruntParser.parseContOnError(GruntParser.java:93)
        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:58)
        at org.apache.pig.Main.main(Main.java:282)
java.lang.NullPointerException
        at org.apache.pig.pen.util.DisplayExamples.MakeArray(DisplayExamples.java:184)
        at org.apache.pig.pen.util.DisplayExamples.PrintTabular(DisplayExamples.java:86)
        at org.apache.pig.pen.util.DisplayExamples.PrintTabular(DisplayExamples.java:80)
        at org.apache.pig.pen.util.DisplayExamples.PrintTabular(DisplayExamples.java:69)
        at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:138)
        at org.apache.pig.PigServer.getExamples(PigServer.java:627)
        at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:278)
        at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:183)
        at org.apache.pig.tools.grunt.GruntParser.parseContOnError(GruntParser.java:93)
        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:58)
        at org.apache.pig.Main.main(Main.java:282)
----------------------------------------------------
| c     | group: bytearray | a: bag    | b: bag    | 
----------------------------------------------------
|       | a                | {(a, 12)} | {(a, 12)} | 
----------------------------------------------------

grunt> 
{noformat}

> Illustrate for the Types branch
> -------------------------------
>
>                 Key: PIG-443
>                 URL: https://issues.apache.org/jira/browse/PIG-443
>             Project: Pig
>          Issue Type: New Feature
>    Affects Versions: types_branch
>            Reporter: Shubham Chopra
>            Assignee: Shubham Chopra
>             Fix For: types_branch
>
>         Attachments: pig_types_22_09.patch, pig_types_23_09.patch
>
>
> This implementation of Illustrate uses visitors to traverse the plans.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message