avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Kleppmann <mkleppm...@linkedin.com>
Subject Re: Reading large AVRO files
Date Fri, 23 May 2014 08:14:00 GMT
Which language are you using? Afaik, most language implementations of Avro only have an interface
for reading one record at a time, but they do buffer the input file internally, so there shouldn't
be a performance disadvantage to reading one record at a time.

If you have an example that is particularly slow, you could be a great help to the Avro community
by getting out a profiler and finding the bottleneck :)

Thanks,
Martin

On 14 May 2014, at 20:13, yael aharon <yael.aharon.m@gmail.com> wrote:
> I am building a java utility that reads large AVRO files and does some processing. These
files have millions of records in them and it can take minutes to read them using DataFileReader.next().
> Is there a way to read more than one record at a time?
> thanks, Yael


Mime
View raw message