hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Slava Rodionov <bep...@gmail.com>
Subject Is there any way to run C++ code with native avro libraries on Hadoop / YARN?
Date Mon, 13 Jan 2014 11:55:10 GMT
Hi everybody,

in our project we have a lot of C++ code which we'd like to run on Hadoop.
Because of the complexity of input and output structures for this code I'd
like to use AVRO as serialization/deserialization format. I figured out a
way to do that with custom Java input and output format classes, but this
way I deal with a lot of JSON on C++ side which leads to code complexity
and not optimal performance.

What I'd like to do is write a wrapper code which uses Pipes and native C++
AVRO libraries to read avro files. I understand how to write/read avro FILE
in C++, but in Pipes we don't have file we have a stream.

Is there any possible way to do aht I want or is it completely impossible
at the moment.

Here is a link with the same question on stack overflow I asked aerlier:

Thanks in advance,

View raw message