Mailing-List: contact avro-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: avro-dev@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=TQXkXlXRECZmpQWxwB9gNfClVUQHslDFnlPCFuWNkkdhgFMhWGo31sOBI7iK7gAq
User-Agent: Microsoft-Entourage/12.23.0.091001
Date: Wed, 31 Mar 2010 17:23:29 -0700
Subject: clarifications on file format
From: Scott Banachowski <sbanacho@yahoo-inc.com>
To: avro-dev <avro-dev@hadoop.apache.org>
Message-ID: <C7D93591.3F789%sbanacho@yahoo-inc.com>
Thread-Topic: clarifications on file format
Thread-Index: AcrRMY1vGyWBavFlvUeaMFaV72USyw==
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit

Hi, 

I'm looking at the spec for the container file, and have 2 questions:

The map of metadata key/value pairs begins with a long, then a number of
string-key/bytes-value pairs.  To be consistent with avro maps, should this
be followed by a long of 0?  The spec doesn't say explicitly, but if the
header is described by an avro schema I would suspect yes.

Are the longs that describe the file block varint longs?  Or 64-bit longs?
I assume avro varints.  But if so, if you ever wanted to expand the size of
block by writing more objects to it, you'd be in trouble because you'd
potentially be unable to fit the new size in the varint's location.

Also, I looked around the repo for some example container files, but didn't
see any.  Are there any examples checked in that we can use to examine their
layout and test our readers?

thanks,
Scott