avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christopher Ng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-1405) Avro-c may not handle eof correctly if avro data file contains multiple sync markers
Date Thu, 21 Aug 2014 15:13:11 GMT

    [ https://issues.apache.org/jira/browse/AVRO-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105449#comment-14105449
] 

Christopher Ng commented on AVRO-1405:
--------------------------------------

I'm getting a different kind of error with this now, it doesn't detect end of file correctly.
 In the files I'm having a problem with, feof is not set even though file->cur == file->end.
 I'm not really a C coder, but casual googling indicates that feof() returning false doesn't
necessarily mean that the file has data left, ie you can be at the end of the file and feof()
will return 0.

http://stackoverflow.com/questions/5431941/while-feof-file-is-always-wrong

This post says that feof will return 0 until something has already tried to read past the
end of the file.

This manifests itself as "Error reading: Cannot read file block count: Cannot read 1 bytes
from file"

It might be difficult for me to provide a test case for this though, as we're running a local
fork where we have added LZ4 compression to Avro, but only for Java, Python and C (the implementations
that we use).  I still think that the eof check is wrong, but it may be that the behaviour
only manifests itself because of some interaction with the LZ4 codec.

> Avro-c may not handle eof correctly if avro data file contains multiple sync markers
> ------------------------------------------------------------------------------------
>
>                 Key: AVRO-1405
>                 URL: https://issues.apache.org/jira/browse/AVRO-1405
>             Project: Avro
>          Issue Type: Bug
>          Components: c
>    Affects Versions: 1.7.5
>            Reporter: Mika Ristimaki
>            Assignee: Mika Ristimaki
>            Priority: Minor
>             Fix For: 1.7.6
>
>         Attachments: AVRO-1405.patch, AVRO-1405.patch
>
>
> I encountered a bug in the Avro C API. If the following is done, it seems that the Avro
data file reader can not read the file correctly
> {code}
> while (has values to write) {
> 	Open file for writing
> 	Write a value to the file
> 	Close the writer
> }
> {code}
> Reading this file with Avro data file reader fails with EOF after only the first item
has been read from the file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message