pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitriy V. Ryaboy (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (PIG-2614) AvroStorage crashes on LOADING a single bad error
Date Sun, 29 Apr 2012 21:25:48 GMT

     [ https://issues.apache.org/jira/browse/PIG-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dmitriy V. Ryaboy reassigned PIG-2614:
--------------------------------------

    Assignee: Jonathan Coveney

Looks like Jon's all over this.. assigning to him.

Would be nice to make this a global setting (and punt per-loader settings to ONERROR implementation)

But that's outside the scope of this ticket, let's at least confirm this works.

Russell, if you can post input that knocks you to the wrong offset, that could help with reproducing
the error you are getting with the current patch.
                
> AvroStorage crashes on LOADING a single bad error
> -------------------------------------------------
>
>                 Key: PIG-2614
>                 URL: https://issues.apache.org/jira/browse/PIG-2614
>             Project: Pig
>          Issue Type: Bug
>          Components: piggybank
>    Affects Versions: 0.10.0, 0.11
>            Reporter: Russell Jurney
>            Assignee: Jonathan Coveney
>              Labels: avro, avrostorage, bad, book, cutting, doug, for, my, pig, sadism
>             Fix For: 0.11, 0.10.1
>
>         Attachments: PIG-2614_0.patch, PIG-2614_1.patch
>
>
> AvroStorage dies when a single bad record exists, such as one with missing fields.  This
is very bad on 'big data,' where bad records are inevitable.  See discussion at http://www.quora.com/Big-Data/In-Big-Data-ETL-how-many-records-are-an-acceptable-loss
for more theory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message