Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
Received-SPF: pass (nike.apache.org: domain of sebastiano.dipaola@gmail.com
 designates 209.85.192.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <53FBA9ED.6010507@cloudera.com>
References: 
 <CACLTFx6EOmvN6Rt5uH5FVieydF85FRM5_GyL_Fo1XR8S=X2-tg@mail.gmail.com>
	<53FBA9ED.6010507@cloudera.com>
Date: Tue, 26 Aug 2014 09:19:45 +0200
Message-ID: 
 <CACLTFx7i8-+KdVUgktDtYBahB98Yemkdmkf2WL3TMY7Ld50WpA@mail.gmail.com>
Subject: Re: Flume Avro Source and Avro C api
From: Sebastiano Di Paola <sebastiano.dipaola@gmail.com>
To: user@flume.apache.org
Content-Type: multipart/alternative; boundary=047d7bacb2aabe84920501832078

--047d7bacb2aabe84920501832078
Content-Type: text/plain; charset=UTF-8

Hi,
As I'm still planning how to do to transport my data, I can use Thrift as
well, but how does Thrift and Avro integrates?
Avro seems very interesting to serialize data and also pretty
straightforward (define a schema -> transform data -> read data again using
the same schema).
It's not clear to me the role of Thrift (ok it will generate code to make
the RPC call) and the integration with Avro.
Can you please add more details?
Regards.
Seba


On Mon, Aug 25, 2014 at 11:26 PM, Hari Shreedharan <
hshreedharan@cloudera.com> wrote:

> Can you use thrift? That is the recommended mechanism for non-java
> applications to Flume
>
> Sebastiano Di Paola wrote:
>
>
> Hi there,
> I'm trying to write a small pice of code using Avro C api and then use
> a Flume avro sink to write the collected and serialized data to hadoop
> HDFS.
>
> I would like to use Avro C api as all the other lib/code I'm using to
> generate/collect my data is in c language and it's already written. So
> I'm adding Avro output to an already existing software.
>
> As I'm reading from avro C API does not support RPC call yet.
>
> So Is there a way to seamless integrate a Flume avro source with avro
> C api?
> Is there any already written example?
>
> Or I have to create my avro file with avro C api (i.e.
> my_data_file.avro) on the file system
> and then use the already provided avro client
> bin/flume-ng avro-client -H localhost -p 41414 -F my_data_file.avro
> in order to have the file "read" through the Flume avro source and
> sent to the Flume sink on hdfs?
>
> Isn't it possible only to configure the Flume Avro source to listen on
> a particulary port and then use Avro C api to send the message?
>
> (I did a quick search on avro C api, but i couln't find any function I
> could call to answer my previous question)
>
> Kind regard.
> Seba
>
>

--047d7bacb2aabe84920501832078
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi,</div>As I&#39;m still planning how to do to trans=
port my data, I can use Thrift as well, but how does Thrift and Avro integr=
ates?<div>Avro seems very interesting to serialize data and also pretty str=
aightforward (define a schema -&gt; transform data -&gt; read data again us=
ing the same schema).</div>
<div>It&#39;s not clear to me the role of Thrift (ok it will generate code =
to make the RPC call) and the integration with Avro.<br></div><div>Can you =
please add more details?</div><div>Regards.</div><div>Seba</div></div><div =
class=3D"gmail_extra">
<br><br><div class=3D"gmail_quote">On Mon, Aug 25, 2014 at 11:26 PM, Hari S=
hreedharan <span dir=3D"ltr">&lt;<a href=3D"mailto:hshreedharan@cloudera.co=
m" target=3D"_blank">hshreedharan@cloudera.com</a>&gt;</span> wrote:<br><bl=
ockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #=
ccc solid;padding-left:1ex">
<div>Can you use thrift? That is the recommended mechanism for non-java app=
lications to Flume<br>
<br>
Sebastiano Di Paola wrote:<br>
<blockquote type=3D"cite"><br>
Hi there,<br>
I&#39;m trying to write a small pice of code using Avro C api and then use =
<br>
a Flume avro sink to write the collected and serialized data to hadoop <br>
HDFS.<br>
<br>
I would like to use Avro C api as all the other lib/code I&#39;m using to <=
br>
generate/collect my data is in c language and it&#39;s already written. So =
<br>
I&#39;m adding Avro output to an already existing software.<br>
<br>
As I&#39;m reading from avro C API does not support RPC call yet.<br>
<br>
So Is there a way to seamless integrate a Flume avro source with avro <br>
C api?<br>
Is there any already written example?<br>
<br>
Or I have to create my avro file with avro C api (i.e. <br>
my_data_file.avro) on the file system<br>
and then use the already provided avro client<br>
bin/flume-ng avro-client -H localhost -p 41414 -F my_data_file.avro<br>
in order to have the file &quot;read&quot; through the Flume avro source an=
d <br>
sent to the Flume sink on hdfs?<br>
<br>
Isn&#39;t it possible only to configure the Flume Avro source to listen on =
<br>
a particulary port and then use Avro C api to send the message?<br>
<br>
(I did a quick search on avro C api, but i couln&#39;t find any function I =
<br>
could call to answer my previous question)<br>
<br>
Kind regard.<br>
Seba<br>
</blockquote></div>
</blockquote></div><br></div>

--047d7bacb2aabe84920501832078--