Return-Path: Delivered-To: apmail-hadoop-avro-user-archive@minotaur.apache.org Received: (qmail 33120 invoked from network); 17 Mar 2010 22:19:49 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 17 Mar 2010 22:19:49 -0000 Received: (qmail 88938 invoked by uid 500); 17 Mar 2010 22:19:49 -0000 Delivered-To: apmail-hadoop-avro-user-archive@hadoop.apache.org Received: (qmail 88891 invoked by uid 500); 17 Mar 2010 22:19:49 -0000 Mailing-List: contact avro-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: avro-user@hadoop.apache.org Delivered-To: mailing list avro-user@hadoop.apache.org Received: (qmail 88883 invoked by uid 99); 17 Mar 2010 22:19:49 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Mar 2010 22:19:49 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of bruce.mitchener@gmail.com designates 209.85.219.227 as permitted sender) Received: from [209.85.219.227] (HELO mail-ew0-f227.google.com) (209.85.219.227) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Mar 2010 22:19:42 +0000 Received: by ewy27 with SMTP id 27so1498716ewy.34 for ; Wed, 17 Mar 2010 15:19:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=+WJE4oNztUee1z50RP7cf3O+alq04uEcuuz0RRQZ4io=; b=XXje6THyfAbxCDBws66+JeaQoli+Eik7Zh4mjrdOA7ZAGCJ1mEXheRqnFfmuYdlCH4 bXk2NIBy0mwp9SXK2eg8rKv3ovwftYMVFUSXpoUguCtHSao7zeFEiJz3EU2Fc36vHsz6 CeOr/7xEtvr7zOaVipy6Bbnq8WGpo9iLxB0nQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=Hb+OW1JXHYF4cX0AIUac1tVSuYLZv6ccRKcWkD/hjIRovCX0jqKAYYZ3Geu34MH/ha UMYm1hD07HjdLDdUQmV4mbXd9aciPlzOAh2OMIWPIJ3KeaQoazoCGHZC8eDbY8QfzU4U GlDtt+js3OT0YvO1yDQ2MhX4UPnxZEf1f2jzk= MIME-Version: 1.0 Received: by 10.216.86.206 with SMTP id w56mr191343wee.1.1268864361692; Wed, 17 Mar 2010 15:19:21 -0700 (PDT) In-Reply-To: <7cd67c5d1003171459rf7e7cf4y157298a2a3a927a@mail.gmail.com> References: <7e45e2ac1003171221i476beeaerf45b24a66b7b1559@mail.gmail.com> <7cd67c5d1003171347y1f9d59e7p8d5166d3a58b6bbf@mail.gmail.com> <7cd67c5d1003171459rf7e7cf4y157298a2a3a927a@mail.gmail.com> Date: Wed, 17 Mar 2010 16:19:21 -0600 Message-ID: <7cd67c5d1003171519h293faf34wf21bd905e307d269@mail.gmail.com> Subject: Re: C Implementation, missing avro_flush() From: Bruce Mitchener To: avro-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=0016e6d783fc8bff1b04820681c7 X-Virus-Checked: Checked by ClamAV on apache.org --0016e6d783fc8bff1b04820681c7 Content-Type: text/plain; charset=ISO-8859-1 This is AVRO-480 in JIRA. Patch is attached for dealing with flush. - Bruce On Wed, Mar 17, 2010 at 3:59 PM, Bruce Mitchener wrote: > Niraj, > > So, I've been looking at this. > > I have a patch ready to roll for the first thing (dealing with > avro_flush()). I'll go ahead and open a bug about that and attach the patch > for massie to commit. > > However, for fsync(), I think we'd like to wait until 1.4 to address this > as it will lead a confusing API at the moment that we'll just have to remove > in short order. > > Right now, the container files are implemented with buffered I/O, so you'd > have to call 3 functions to ensure that it hit disk. I think we can get > away without buffered I/O and that would make it a more reasonable 2 > functions. > > Do you have an overwhelming need for fsync functionality on container files > (datafile.c stuff) in the next month or two? > > - Bruce > > > On Wed, Mar 17, 2010 at 2:47 PM, Bruce Mitchener < > bruce.mitchener@gmail.com> wrote: > >> Niraj, >> >> The header says avro_flush(), but the implementation says >> avro_writer_flush(). We'll get this addressed shortly. >> >> There isn't currently a way to call fsync() directly ... but since you >> pass the FILE* to the file writer, you could call fsync(fileno(FILE*)) on >> your own, unless you're using the container file. >> >> If you want to open a bug on each of these, I'll work up the patches and >> work with massie to get them into SVN. >> >> Cheers, >> >> - Bruce >> >> >> On Wed, Mar 17, 2010 at 1:21 PM, Niraj Tolia wrote: >> >>> I was going through the avro.h header file (from the 1.3.0 release) >>> and noticed that the avro_flush() call is defined but has no >>> implementation. If someone does try to use it, compilation will fail >>> with an undefined reference error. I am not sure if this was an >>> accidental oversight but figured I should let someone know. >>> >>> Also, would I be correct in assuming that the C API doesn't allow me >>> to actually call fsync() on a file writer? Digging through the code >>> didn't turn up anything obvious. >>> >>> Cheers, >>> Niraj >>> >>> -- >>> http://www.tolia.org/ >>> >> >> > --0016e6d783fc8bff1b04820681c7 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable This is AVRO-480 in JIRA. Patch is attached for dealing with flush.

=A0- Bruce

On Wed, Mar 1= 7, 2010 at 3:59 PM, Bruce Mitchener <bruce.mitchener@gmail.com> wrote:=
Niraj,

So, I've been= looking at this.

I have a patch ready to roll for= the first thing (dealing with avro_flush()). =A0I'll go ahead and open= a bug about that and attach the patch for massie to commit.

However, for fsync(), I think we'd like to wait unt= il 1.4 to address this as it will lead a confusing API at the moment that w= e'll just have to remove in short order.

Right now, the container files are implemented with buffered I/O, so you= 9;d have to call 3 functions to ensure that it hit disk. =A0I think we can = get away without buffered I/O and that would make it a more reasonable 2 fu= nctions.

Do you have an overwhelming need for fsync functionalit= y on container files (datafile.c stuff) in the next month or two?

=A0- Bruce


On Wed, Mar 17, 2010 at 2:47 PM, Bruce M= itchener <bruce.mitchener@gmail.com> wrote:
Niraj,

The header says av= ro_flush(), but the implementation says avro_writer_flush(). =A0We'll g= et this addressed shortly.

There isn't currently a way to call fsync() directl= y ... but since you pass the FILE* to the file writer, you could call fsync= (fileno(FILE*)) on your own, unless you're using the container file.

If you want to open a bug on each of these, I'll wo= rk up the patches and work with massie to get them into SVN.

=
Cheers,

=A0- Br= uce


On Wed, Mar 17, 2010 at 1:21 PM, Niraj Tolia <ntolia@gmail.com> wrote:
I was going through the avro.h header file (from the 1.3.0 release)
and noticed that the avro_flush() call is defined but has no
implementation. If someone does try to use it, compilation will fail
with an undefined reference error. I am not sure if this was an
accidental oversight but figured I should let someone know.

Also, would I be correct in assuming that the C API doesn't allow me to actually call fsync() on a file writer? Digging through the code
didn't turn up anything obvious.

Cheers,
Niraj

--
http://www.tolia.org/



--0016e6d783fc8bff1b04820681c7--