hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pradeep Kamath (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-517) Custom Loader Function which takes in a constructor argument fails during typecast
Date Tue, 04 Nov 2008 23:54:44 GMT

     [ https://issues.apache.org/jira/browse/PIG-517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Pradeep Kamath updated PIG-517:
-------------------------------

    Assignee: Pradeep Kamath
      Status: Patch Available  (was: Open)

Attached patch - details on the issue:
In case of a "cast" from bytearray to any other type, TypeCheckingVisitor tries to match the
cast to the corresponding load function for the source of the data so that the load function
can do the cast correctly ( since there is no generic way to convert from bytearray to other
types - only the load function which is the source of the bytearray knows how to interpret
the bytes present in the bytearray). The TypeCheckingVisitor in the current code is setting
a "LoadFunc" reference in LOCast for this purpose. The LoadFunc reference does not have information
of the arguments needed to create the Loader object at runtime. This manifests as a problem
in the LogToPhyTranslator when it tries to create a POCast from the LOCast. 

The patch now uses FuncSpec to represent the loader function rather than a LoadFunc reference.
The FuncSpec has both the classname and the arguments to the constructor of the loader.

> Custom Loader Function which takes in a constructor argument fails during typecast
> ----------------------------------------------------------------------------------
>
>                 Key: PIG-517
>                 URL: https://issues.apache.org/jira/browse/PIG-517
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: types_branch
>            Reporter: Viraj Bhat
>            Assignee: Pradeep Kamath
>             Fix For: types_branch
>
>         Attachments: phonenumber.txt, RegexLoader.java
>
>
> I have a custom loader function,  known as RegexLoader that parses a line of input into
fields using regex and then sets the fields. This RegexLoader extends Utf8StorageConverter
and implements the LoadFunc. It takes in a constructor argument a regex string supplied by
the user.
> The following piece of code, works when the loaded fields are not typecasted.
> {code}
> REGISTER pigudf2.0/java/build/loader.jar
> fullfile = load 'phonenumber.txt'
>                using loader.RegexLoader('4*8')
>                as   (a,z,n) ;
> -- project required fields
> phonerecords = foreach fullfile {
>           generate
>            a                as area,
>            z               as zone,
>            n               as number;
>         }
> dump phonerecords;
> {code}
> But when the alias a is cast to int, the piece of script fails with the error java.io.IOException:
Unable to open iterator for alias: phonerecords [Unable to store for alias: phonerecords [could
not instantiate 'loader.RegexLoader' with arguments 'null']]
> {code}
> REGISTER pigudf2.0/java/build/loader.jar
> fullfile = load 'phonenumber.txt'
>              using loader.RegexLoader('4*8')
>         as   (a,z,n) ;
> -- project required fields
> phonerecords = foreach fullfile {
>           generate
>            (int)a          as area,
>            z               as zone,
>            n               as number;
>         }
> dump phonerecords;
> {code}
> Full stack trace of the error:
> ==================================================================================================================
> java.io.IOException: Unable to open iterator for alias: phonerecords [Unable to store
for alias: phonerecords [could not instantiate 'loader.RegexLoader' with arguments 'null']]
>      at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:448)
>      at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:454)
>      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.instantiateFunc(POCast.java:67)
>      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.setLoadFSpec(POCast.java:73)
>      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.LogToPhyTranslationVisitor.visit(LogToPhyTranslationVisitor.java:1157)
>      at org.apache.pig.impl.logicalLayer.LOCast.visit(LOCast.java:60)
>      at org.apache.pig.impl.logicalLayer.LOCast.visit(LOCast.java:28)
>      at org.apache.pig.impl.plan.DependencyOrderWalkerWOSeenChk.walk(DependencyOrderWalkerWOSeenChk.java:68)
>      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.LogToPhyTranslationVisitor.visit(LogToPhyTranslationVisitor.java:805)
>      at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:121)
>      at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:40)
>      at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
>      at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
>      at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:232)
>      at org.apache.pig.PigServer.compilePp(PigServer.java:731)
>      at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:644)
>      at org.apache.pig.PigServer.store(PigServer.java:452)
>      at org.apache.pig.PigServer.store(PigServer.java:421)
>      at org.apache.pig.PigServer.openIterator(PigServer.java:384)
>      at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269)
>      at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
>      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
>      at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
>      at org.apache.pig.Main.main(Main.java:306)
> Caused by: java.io.IOException: Unable to store for alias: phonerecords [could not instantiate
'loader.RegexLoader' with arguments 'null']
>      ... 24 more
> Caused by: java.lang.RuntimeException: could not instantiate 'loader.RegexLoader' with
arguments 'null'
>      ... 24 more
> Caused by: java.lang.InstantiationException: loader.RegexLoader
>      at java.lang.Class.newInstance0(Class.java:340)
>      at java.lang.Class.newInstance(Class.java:308)
>      at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:418)
>      ... 23 more
> ==================================================================================================================
> Attaching the custom RegexLoader with this Jira

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message