avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Carey <scottca...@apache.org>
Subject Re: Is it possible to append to an already existing avro file
Date Tue, 21 Feb 2012 17:02:55 GMT

On 2/21/12 7:29 AM, "Vyacheslav Zholudev" <vyacheslav.zholudev@gmail.com>

>Yep, I saw that method as well as the stackoverflow post. However, I'm
>interested how to append to a file on the arbitrary file system, not only
>on the local one. 
>I want to get an OutputStream based on the Path and the FileSystem
>implementation and then pass it for appending to avro methods.
>Is that possible?

It is not possible without modifying DataFileWriter. Please open a JIRA

It could not simply append to an OutputStream, since it must either:
* Seek to the start to validate the schemas match and find the sync
marker, or
* Trust that the schemas match and find the sync marker from the last block

DataFileWriter cannot refer to Hadoop classes such as FileSystem, but we
could add something to the mapred module that takes a Path and FileSystem
and returns
something that implemements an interface that DataFileWriter can append
to.  This would be something that is both a
and an OutputStream, or has both an InputStream from the start of the
existing file and an OutputStream at the end.

>On Feb 21, 2012, at 5:29 AM, Harsh J wrote:
>> Hi,
>> Use the appendTo feature of the DataFileWriter. See
>> For a quick setup example, read also:
>> On Tue, Feb 21, 2012 at 3:15 AM, Vyacheslav Zholudev
>> <vyacheslav.zholudev@gmail.com> wrote:
>>> Hi,
>>> is it possible to append to an already existing avro file when it was
>>> written and closed before?
>>> If I use
>>> outputStream = fs.append(avroFilePath);
>>> then later on I get: java.io.IOException: Invalid sync!
>>> Probably because the schema is written twice and some other issues.
>>> If I use outputStream = fs.create(avroFilePath); then the avro file
>>> overwritten.
>>> Thanks,
>>> Vyacheslav
>> -- 
>> Harsh J
>> Customer Ops. Engineer
>> Cloudera | http://tiny.cloudera.com/about

View raw message