avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Russel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-1182) DataFileReader missing seek, sync methods
Date Tue, 18 Jun 2013 21:32:22 GMT

    [ https://issues.apache.org/jira/browse/AVRO-1182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687223#comment-13687223
] 

Daniel Russel commented on AVRO-1182:
-------------------------------------

> Is it possible to address that problem? I understand it is not straightforward because
of no-copy input streams.
The code does (attempt to) address this problem.

Testing it systematically is a little trickier I think as I would have to force the read block
size to match the sync interval. So I would need (header_size + sync_interval) to be equal
to the input buffer size. I presume, for a given record, the header size can be assumed to
be constant over time. And the sync interval I can specify. The input buffer size isn't exposed
though, so the test is a bit brittle to that default value changing.

I could do a more stochastic test where I make the sync size really small and then read small
records from the file. One of the syncs is likely to hit an input buffer boundary and I can
check that none are missed. Do you think that is enough?
                
> DataFileReader missing seek, sync methods
> -----------------------------------------
>
>                 Key: AVRO-1182
>                 URL: https://issues.apache.org/jira/browse/AVRO-1182
>             Project: Avro
>          Issue Type: Improvement
>          Components: c++
>    Affects Versions: 1.7.3
>            Reporter: Daniel Russel
>         Attachments: add_seek
>
>
> The DataFileReader is missing the seek and sync methods that are found in the java version
making it hard to navigate a file except in a linear fashion.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message