Mailing-List: contact user-help@avro.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@avro.apache.org
Received-SPF: pass (nike.apache.org: domain of
 prvs=162cff910=mkleppmann@linkedin.com designates 69.28.149.80 as permitted
 sender)
From: Martin Kleppmann <mkleppmann@linkedin.com>
To: "<user@avro.apache.org>" <user@avro.apache.org>
Subject: Re: Dynamic Schema
Thread-Topic: Dynamic Schema
Thread-Index: AQHPTezJoeSh+fIOr0+9Oh8m+JKAJpr/RYwA
Date: Wed, 2 Apr 2014 21:01:08 +0000
Message-ID: <350707B7-D311-4E46-A13B-4A104A729728@linkedin.com>
References: 
 <CANGA=ecKnLZigZ7uux6x3q7dnNmJVbUW0-1Os1_WO_hMy2pbJQ@mail.gmail.com>
In-Reply-To: 
 <CANGA=ecKnLZigZ7uux6x3q7dnNmJVbUW0-1Os1_WO_hMy2pbJQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-ID: <69084E2DA44E5F478A2BA7EF2B7FE3CC@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi Amit,

The Avro data file format requires the writer to know the schema from the s=
tart, because all records in the file are then written with the same schema=
. So there probably isn't an alternative to what you're doing -- to buffer =
as much as you can in memory, write it out to file when the memory buffer i=
s full, and then start a new file.

You can't change the schema of a data file once it has been written, but yo=
u can run a background process which merges several data files together, an=
d writes the result to a new file. You can make the merged file's schema th=
e union of all the input file schemas, or you can write some application-sp=
ecific code which combines the schemas into one, and evolve all the records=
 into that merged schema. This can be done by streaming through the files -=
- you don't need to keep all the data in memory.

Martin


On 1 Apr 2014, at 21:55, amit nanda <amitwip@gmail.com> wrote:
> I have very dynamic data that i want to write to an avro file. The soluti=
on i have is to store all that data in the memory and then calculate the sc=
hema, and then start the writing.=20
>=20
> This causes the files to be smaller in size, because of the memory limita=
tions.
>=20
> What i am looking for is that i will start data as and when it is collect=
ed, but how should i compute the schema in this case? Can i change the sche=
ma for an avro file?
>=20
> Thanks
> Amit