avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Douglas Creager <doug...@creagertino.net>
Subject Re: Avro-c with libhdfs
Date Mon, 21 Oct 2013 12:57:58 GMT
> I would like to write Avro files to HDFS using C as the programming
> language. I have a simple queston: Can I use the avro-c API with
> libhdfs? I tried the hdfs-nfs-proxy, but it doesn't support the append
> operation.

I don't have any experience with libhdfs, so I'm not sure.  The Avro C
API lets you write Avro data files to a file on the local filesystem,
identified by a `const char *` filename, or a `FILE *` instance that
you've already opened, so my guess is that, out of the box, the only way
it would work would be to create the file locally and then transfer the
existing local file to HDFS.

To get it to work directly with libhdfs, we'd probably need to add a new
API for providing a callback function, which the Avro bindings would
call whenever data needs to be written.  (This is already what happens
under the covers, so it would really just be exposing parts of the
existing functionality in the public API.)


View raw message