pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Felix Gao (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1748) Add load/store function AvroStorage for avro data
Date Tue, 25 Jan 2011 17:56:45 GMT

    [ https://issues.apache.org/jira/browse/PIG-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986551#action_12986551
] 

Felix Gao commented on PIG-1748:
--------------------------------

I noticed the avro loader does not support file globbing. 

log_load = LOAD '/user/felix/avro/access_log.test.avro' USING org.apache.pig.piggybank.storage.avro.AvroStorage();
  <--- works fine
but
log_load = LOAD '/user/felix/avro/*.avro' USING org.apache.pig.piggybank.storage.avro.AvroStorage();

ERROR 1018: Problem determining schema during load

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Problem
determining schema during load
    at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1342)
    at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1286)
    at org.apache.pig.PigServer.registerQuery(PigServer.java:460)
    at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:738)

    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:324)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:163)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:139)
    at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89)
    at org.apache.pig.Main.main(Main.java:414)
Caused by: org.apache.pig.impl.logicalLayer.parser.ParseException: Problem determining schema
during load
    at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:752)

    at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
    at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1336)
    ... 8 more
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1018: Problem determining
schema during load
    at org.apache.pig.impl.logicalLayer.LOLoad.getSchema(LOLoad.java:156)
    at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:750)
    ... 10 more
Caused by: java.io.FileNotFoundException: File does not exist: /user/felix/avro/*.avro
    at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1586)
    at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1577)
    at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:428)
    at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:185)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:431)
    at org.apache.pig.piggybank.storage.avro.AvroStorage.getSchema(AvroStorage.java:181)

    at org.apache.pig.piggybank.storage.avro.AvroStorage.getAvroSchema(AvroStorage.java:133)
    at org.apache.pig.piggybank.storage.avro.AvroStorage.getAvroSchema(AvroStorage.java:108)
    at org.apache.pig.piggybank.storage.avro.AvroStorage.getSchema(AvroStorage.java:233)
    at org.apache.pig.impl.logicalLayer.LOLoad.determineSchema(LOLoad.java:169)
    at org.apache.pig.impl.logicalLayer.LOLoad.getSchema(LOLoad.java:150)
    ... 11 more

> Add load/store function AvroStorage for avro data
> -------------------------------------------------
>
>                 Key: PIG-1748
>                 URL: https://issues.apache.org/jira/browse/PIG-1748
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: lin guo
>            Assignee: Jakob Homan
>         Attachments: avro_storage.patch, avro_test_files.tar.gz, PIG-1748-2.patch
>
>
> We want to use Pig to process arbitrary Avro data and store results as Avro files. AvroStorage()
extends two PigFuncs: LoadFunc and StoreFunc. 
> Due to discrepancies of Avro and Pig data models, AvroStorage has:
> 1. Limited support for "record": we do not support recursively defined record because
the number of fields in such records is data dependent.
> 2. Limited support for "union": we only accept nullable union like ["null", "some-type"].
> For simplicity, we also make the following assumptions:
> If the input directory is a leaf directory, then we assume Avro data files in it have
the same schema;
> If the input directory contains sub-directories, then we assume Avro data files in all
sub-directories have the same schema.
> AvroStorage takes no input parameters when used as a LoadFunc (except for "debug [debug-level]").

> Users can provide parameters to AvroStorage when used as a StoreFunc. If they don't,
Avro schema of output data is derived from its 
> Pig schema.
> Detailed documentation can be found in http://snaprojects.jira.com/wiki/display/HTOOLS/AvroStorage+-+Pig+support+for+Avro+data

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message