avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Douglas Creager <doug...@creagertino.net>
Subject Re: Using AVRO C with a large schema
Date Sun, 18 Aug 2013 15:30:44 GMT
>> 3. AVRO C cannot write values that are bigger than the file writer's
>> specified block_size.  I don't think there's enough memory to hold both the
>> original structures and a gigantic block_size.
> I don't know enough about the C implementation to verify this one and
> will leave it to others.

#3 is also true.  AVRO-724 [1] is the relevant issue.  Matt Massie's
comment on that issue discusses a couple of proposed solutions.

If your data structures are large arrays, then one option is to have a
separate file for each array, and have the array elements correspond to
records in the file.  Then it's only each individual array element that
needs to fit in the file's block_size.

View raw message