pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prashant Kommireddi (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-784) PigStorage() - need ability to turn off "Attempt to access field" warnings
Date Thu, 13 Oct 2011 19:37:12 GMT

    [ https://issues.apache.org/jira/browse/PIG-784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126849#comment-13126849

Prashant Kommireddi commented on PIG-784:

This is my understanding too, but I notice with 0.9.1 I am seeing warning are not aggregated
in MR mode

> PigStorage() - need ability to turn off "Attempt to access field"  warnings
> ---------------------------------------------------------------------------
>                 Key: PIG-784
>                 URL: https://issues.apache.org/jira/browse/PIG-784
>             Project: Pig
>          Issue Type: Improvement
>    Affects Versions: 0.2.0
>            Reporter: David Ciemiewicz
> I want an option to PigStorage() for LOAD which will allow me to turn off the "Attempt
to access field" warnings.
> Something like:
> {code}
> define PigStorage PigStorage("warn_load_nonexistent_field=off");
> A = load 'mydata.txt' using PigStorage()
>         as (col1: chararray, col2_optional: int, col3_optional: float);
> {code}
> or
> {code}
> A = load 'mydata.txt' using PigStorage("warn_load_nonexistent_field=0")
>         as (col1: chararray, col2_optional: int, col3_optional: float);
> {code}
> If I have a very large data set with optional columns that are not populated (and have
no tab separator), I'd like to just read the file as is and not generate the warnings.
> The warnings are problematic because the fill up the logging output and every System.out.println
will generate slow down the overall processing.  Especially if the data file being processed
is missing one or more columns on every single row.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message