avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-1419) java.io.IOException: Invalid sync! throw after random number of sync() calls.
Date Tue, 07 Jan 2014 18:44:04 GMT

    [ https://issues.apache.org/jira/browse/AVRO-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13864521#comment-13864521

Doug Cutting commented on AVRO-1419:

Ideally a standalone JUnit test that reproduces the problem.  Failing that, a data file and
a test program that fails when reading it might permit someone to debug this.  Thanks!

> java.io.IOException: Invalid sync! throw after random number of sync() calls.
> -----------------------------------------------------------------------------
>                 Key: AVRO-1419
>                 URL: https://issues.apache.org/jira/browse/AVRO-1419
>             Project: Avro
>          Issue Type: Bug
>          Components: java
>    Affects Versions: 1.7.5
>            Reporter: Deepak Kumar V
> I have a 340 MB avro data file that contains records sorted and identified by unique
id (duplicate records exists). At the beginning of every unique record a synchronization point
is created with DataFileWriter.sync(). (I cannot or do not want to save the sync points and
i do not want to use SortedKeyValueFile as output format for M/R job)  
> There are at-least 25k synchronization points in a 340 MB file.
> Ex:
> Marker1_RecordA1_RecordA2_RecordA3_Marker2_RecordB1_RecordB2
> As records are sorted and marked, for efficient retrieval, binary search is performed.
Most of the times the search is successful, at times the code throws the following exception
> ------
> org.apache.avro.AvroRuntimeException: java.io.IOException: Invalid sync! at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:210

> ------
> I note down the position that was used to invoke fileReader.sync(mid); and catch AvroRuntimeException,
close and open the file and sync(mid) i do not see exception. 
> Why should Avro throw exception before and not later ?
> 1.7.5v of library is throwing this error. Raising a major defect, adjust the priority
at your convenience. 

This message was sent by Atlassian JIRA

View raw message