flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gonzalo Herreros <gherre...@gmail.com>
Subject Re: flume problem
Date Wed, 02 Mar 2016 10:00:42 GMT
The channel serializes the flume event as avro including the headers, the
http headers become event headers
However the sink should only store the content, not the headers

On 2 March 2016 at 09:51, Baris Akgun (Garanti Teknoloji) <
BarisAkgu@garanti.com.tr> wrote:

> No, we send json twitter data but in flume channel ı saw content type word
> for each tweet. Is it normal ? How can ı send just tweets json without any
> content type. I took tweets json from GNIP company.
>
> Thanks
> iPhone'umdan gönderildi
>
> 2 Mar 2016 tarihinde 10:56 saatinde, Gonzalo Herreros <gherreros@gmail.com>
> şunları yazdı:
>
> Could it be that you are serializing avro instead of json?
>
> On 2 March 2016 at 08:25, Baris Akgun (Garanti Teknoloji) <
> BarisAkgu@garanti.com.tr> wrote:
>
>> Hi,
>>
>>
>>
>> When I send json data to flume with using http post, flume adds
>> Co**ntent-Typeapplication/json** for each json post.
>>
>>
>>
>> In my http post java code,  I give the content-type with using
>>
>>
>>
>> **con.setRequestProperty("Content-Type", "application/json");** function.
>>
>>
>>
>>
>>
>> I am using blob handler.
>>
>>
>>
>> **In flume conf file**
>>
>>
>>
>> *tier1.sources.source1.type = org.apache.flume.source.http.HTTPSource
>>
>> tier1.sources.source1.handler =
>> org.apache.flume.sink.solr.morphline.BlobHandler*
>>
>>
>>
>> In flume channel, flume adds content type for each post as you see. After
>> HDFS sink, The content type word causes a problem when ı try to parse json
>> with spark sql or hive serDe.
>>
>>
>>
>> **The flume channel log data**
>>
>>
>>
>> *^LContent-Typeapplication/jsonú{"id":"+ag:_ea_ch.++i++e_.c-
>>
>> ^LContentTypeapplication/json‘{"id":"tag:search.twitter.com
>> ,2005:642913165047648*
>>
>>
>>
>> Is there any idea for that problem?
>>
>>
>>
>> Thank a lot.
>>
>>
>>
>> *Barış Akgün*
>> Analitik Veri Ambarı ve Büyük Veri Yönetimi
>> Uzman
>>
>> Tel
>>
>> :
>>
>> Dahili
>>
>> :
>>
>> Faks
>>
>> :
>>
>>
>> Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir
>> ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis
>> olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune
>> uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde
>> yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da
>> sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin
>> ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan,
>> butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve
>> bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz.
>>
>> This message and attachments are confidential and intended solely for the
>> individual(s) stated in this message. If you received this message although
>> you are not the addressee, you are responsible to keep the message
>> confidential. The sender has no responsibility for the accuracy or
>> correctness of the information in the message and its attachments. Our
>> company shall have no liability for any changes or late receiving, loss of
>> integrity and confidentiality, viruses and any damages caused in anyway to
>> your computer system.
>>
>
> Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir
> ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis
> olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune
> uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde
> yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da
> sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin
> ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan,
> butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve
> bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz.
>
> This message and attachments are confidential and intended solely for the
> individual(s) stated in this message. If you received this message although
> you are not the addressee, you are responsible to keep the message
> confidential. The sender has no responsibility for the accuracy or
> correctness of the information in the message and its attachments. Our
> company shall have no liability for any changes or late receiving, loss of
> integrity and confidentiality, viruses and any damages caused in anyway to
> your computer system.
>

Mime
View raw message