Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5869718302 for ; Wed, 2 Mar 2016 10:05:53 +0000 (UTC) Received: (qmail 1508 invoked by uid 500); 2 Mar 2016 10:05:53 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 1452 invoked by uid 500); 2 Mar 2016 10:05:53 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 1442 invoked by uid 99); 2 Mar 2016 10:05:53 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2016 10:05:53 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 83F921A11F3 for ; Wed, 2 Mar 2016 10:05:52 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id HgmpUw_g5TZq for ; Wed, 2 Mar 2016 10:05:50 +0000 (UTC) Received: from mail-wm0-f41.google.com (mail-wm0-f41.google.com [74.125.82.41]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id D4C885FB14 for ; Wed, 2 Mar 2016 10:05:49 +0000 (UTC) Received: by mail-wm0-f41.google.com with SMTP id l68so70074360wml.1 for ; Wed, 02 Mar 2016 02:05:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to; bh=Riil+sQOowcY/YgiMLIETJZNu/OWaj4v5dVpLLZAUEw=; b=knaHVfoeMSplRKrw08Ll/49rGT4Rx5CAFrVvfNWfio9RvmEdTCbpMx8WAFdCFGGodi B3LX8MSUAx4XFD2yrzW0hsac7mWEVge7JI3Xzi4bp1JkA8OSF2aMrPWrQLdRqStJZuJi jPngrFjOmx0e08bXkfC88lJU4pcAcpTDNBNLfx3v32ak+TMp5ip6IMnW3CssQXka4zDo 9NfkK2Yyn4iEbWmZIreYn3dI9v8SwSNgTO4JcK5aIw6e0fSf0aXHdAvKh/5Z72JmF/5y xaM85c++0ExYBhHyFKyph9y3YhuCl16IcOVtxLxM8PfcPXHG0zqnOVsn/M++rbSoRzMv pMQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to; bh=Riil+sQOowcY/YgiMLIETJZNu/OWaj4v5dVpLLZAUEw=; b=WFNXqTYwE1p5qEKO/12jAuzl3KArkngQjyIyCNDeH1cgfSl7KfSCt6ZrScFI5c1N7q KR1FJ4ZjjbeKxg7Dp0es27ekZSHrinDYv19AmGqZkokFCTOc4syvppZGKrpSI/ZWSr1L Rg96YObrtt+O6644kEqsmOZc5scj7fZZY/pAhUr9VhYw0uLazA2UJjjVWOPjRb4e6dnN b3w+Sm+fLflUn9rSAu/JFeVMhtSb+6gGiCulABBo7qMiv8dAjQVkgFOOEcC9iLUan+Ch 98g7wBoLs1fne/8INciB4V7x4HiQ1Mdp0uYMrbRQR24UjII/eqckvbmG1Pdbf3+HFOWk y9fg== X-Gm-Message-State: AD7BkJIjCtO9M7WSn8owOQg7b+vw2PE0qP+zBWVyBvOrTZ7HWOXS6sH3mHYurQaq12cm1zYaZqeU9rbbwqSRIw== MIME-Version: 1.0 X-Received: by 10.28.183.137 with SMTP id h131mr3796999wmf.100.1456912842382; Wed, 02 Mar 2016 02:00:42 -0800 (PST) Received: by 10.194.29.34 with HTTP; Wed, 2 Mar 2016 02:00:42 -0800 (PST) In-Reply-To: <818E3059-20CE-4D12-A974-DAFEFF2380FF@garanti.com.tr> References: <818E3059-20CE-4D12-A974-DAFEFF2380FF@garanti.com.tr> Date: Wed, 2 Mar 2016 10:00:42 +0000 Message-ID: Subject: Re: flume problem From: Gonzalo Herreros To: user Content-Type: multipart/alternative; boundary=001a1148e2e6706885052d0df485 --001a1148e2e6706885052d0df485 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable The channel serializes the flume event as avro including the headers, the http headers become event headers However the sink should only store the content, not the headers On 2 March 2016 at 09:51, Baris Akgun (Garanti Teknoloji) < BarisAkgu@garanti.com.tr> wrote: > No, we send json twitter data but in flume channel =C4=B1 saw content typ= e word > for each tweet. Is it normal ? How can =C4=B1 send just tweets json witho= ut any > content type. I took tweets json from GNIP company. > > Thanks > iPhone'umdan g=C3=B6nderildi > > 2 Mar 2016 tarihinde 10:56 saatinde, Gonzalo Herreros > =C5=9Funlar=C4=B1 yazd=C4=B1: > > Could it be that you are serializing avro instead of json? > > On 2 March 2016 at 08:25, Baris Akgun (Garanti Teknoloji) < > BarisAkgu@garanti.com.tr> wrote: > >> Hi, >> >> >> >> When I send json data to flume with using http post, flume adds >> Co**ntent-Typeapplication/json** for each json post. >> >> >> >> In my http post java code, I give the content-type with using >> >> >> >> **con.setRequestProperty("Content-Type", "application/json");** function= . >> >> >> >> >> >> I am using blob handler. >> >> >> >> **In flume conf file** >> >> >> >> *tier1.sources.source1.type =3D org.apache.flume.source.http.HTTPSource >> >> tier1.sources.source1.handler =3D >> org.apache.flume.sink.solr.morphline.BlobHandler* >> >> >> >> In flume channel, flume adds content type for each post as you see. Afte= r >> HDFS sink, The content type word causes a problem when =C4=B1 try to par= se json >> with spark sql or hive serDe. >> >> >> >> **The flume channel log data** >> >> >> >> *^LContent-Typeapplication/json=C3=BA{"id":"+ag:_ea_ch.++i++e_.c- >> >> ^LContentTypeapplication/json=E2=80=98{"id":"tag:search.twitter.com >> ,2005:642913165047648* >> >> >> >> Is there any idea for that problem? >> >> >> >> Thank a lot. >> >> >> >> *Bar=C4=B1=C5=9F Akg=C3=BCn* >> Analitik Veri Ambar=C4=B1 ve B=C3=BCy=C3=BCk Veri Y=C3=B6netimi >> Uzman >> >> Tel >> >> : >> >> Dahili >> >> : >> >> Faks >> >> : >> >> >> Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldi= r >> ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis >> olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune >> uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde >> yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da >> sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesaji= n >> ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan, >> butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve >> bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutula= maz. >> >> This message and attachments are confidential and intended solely for th= e >> individual(s) stated in this message. If you received this message altho= ugh >> you are not the addressee, you are responsible to keep the message >> confidential. The sender has no responsibility for the accuracy or >> correctness of the information in the message and its attachments. Our >> company shall have no liability for any changes or late receiving, loss = of >> integrity and confidentiality, viruses and any damages caused in anyway = to >> your computer system. >> > > Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir > ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis > olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune > uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde > yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da > sirketimizin herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin > ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan, > butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve > bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulam= az. > > This message and attachments are confidential and intended solely for the > individual(s) stated in this message. If you received this message althou= gh > you are not the addressee, you are responsible to keep the message > confidential. The sender has no responsibility for the accuracy or > correctness of the information in the message and its attachments. Our > company shall have no liability for any changes or late receiving, loss o= f > integrity and confidentiality, viruses and any damages caused in anyway t= o > your computer system. > --001a1148e2e6706885052d0df485 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
The channel serializes the flume event as avro including t= he headers, the http headers become event headers
However the sink shou= ld only store the content, not the headers

On 2 March 2016 at 09:51, Baris Akgun = (Garanti Teknoloji) <BarisAkgu@garanti.com.tr> wrote:=
No, we send json twitter data but in flume channel =C4=B1 saw content = type word for each tweet. Is it normal ? How can =C4=B1 send just tweets js= on without any content type. I took tweets json from GNIP company.

Thanks
iPhone'umdan g=C3=B6nderildi

2 Mar 2016 tarihinde 10:56 saatinde, Gonzalo Herreros <gherreros@gmail.com> =C5=9Fu= nlar=C4=B1 yazd=C4=B1:

Could it be that you are serializing avro instead of json?=

On 2 March 2016 at 08:25, Baris Akgun (Garanti T= eknoloji) <BarisAkgu@garanti.com.tr> wrote:

Hi,

=C2=A0

When I send json data to flume with using= http post, flume adds Co**ntent-Typeapplication/json** for each json post.=

=C2=A0

In my http post java code,=C2=A0 I give t= he content-type with using

=C2=A0

**con.setRequestProperty("Content-Ty= pe", "application/json");** function.

=C2=A0

=C2=A0

I am using blob handler.

=C2=A0

**In flume conf file**

=C2=A0

*tier1.sources.source1.type =3D org.apach= e.flume.source.http.HTTPSource

tier1.sources.source1.handler =3D org.apa= che.flume.sink.solr.morphline.BlobHandler*

=C2=A0

In flume channel, flume adds content type= for each post as you see. After HDFS sink, The content type word causes a = problem when =C4=B1 try to parse json with spark sql or hive serDe.

=C2=A0

**The flume channel log data**<= /u>

=C2=A0

*^LContent-Typeapplication/json=C3=BA{&qu= ot;id":"+ag:_ea_ch.++i++e_.c-

^LContentTypeapplication/json=E2=80=98{&q= uot;id":"tag:search.twitter.com,2005:642913165047648*

=C2=A0

Is there any idea for that problem?

=C2=A0

Thank a lot.

=C2=A0

Bar=C4=B1=C5=9F Akg=C3=BC= n
Analitik Veri Ambar=C4=B1 ve B=C3=BCy=C3=BCk Veri Y=C3=B6netimi
Uzman

Tel

:

Dahili

:

Faks

:

=C2=A0

Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir v= e gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis olmasi= halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune uyulmasi z= orunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde yer alan bilgilerin dogrulugu ve guncelligi = konusunda gonderenin ya da sirketimizin herhangi bir sorumlulugu bulunmamak= tadir. Sirketimiz mesajin ve bilgilerinin size degisiklige ugrayarak veya g= ec ulasmasindan, butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve bilgisayar sisteminiz= e verebilecegi herhangi bir zarardan sorumlu tutulamaz.

This message and attachments are confidential and intended solely for the i= ndividual(s) stated in this message. If you received this message although = you are not the addressee, you are responsible to keep the message confiden= tial. The sender has no responsibility for the accuracy or correctness of the information in the message and its = attachments. Our company shall have no liability for any changes or late re= ceiving, loss of integrity and confidentiality, viruses and any damages cau= sed in anyway to your computer system.

Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir v= e gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis olmasi= halinde mesaj iceriginin gizliligi ve bu gizlilik yukumlulugune uyulmasi z= orunlulugu tarafiniz icin de soz konusudur. Mesaj ve eklerinde yer alan bil= gilerin dogrulugu ve guncelligi konusunda gonderenin ya da sirketimizin her= hangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin ve bilgilerinin s= ize degisiklige ugrayarak veya gec ulasmasindan, butunlugunun ve gizliligin= in korunamamasindan, virus icermesinden ve bilgisayar sisteminize verebilec= egi herhangi bir zarardan sorumlu tutulamaz.

This message and attach= ments are confidential and intended solely for the individual(s) stated in = this message. If you received this message although you are not the address= ee, you are responsible to keep the message confidential. The sender has no= responsibility for the accuracy or correctness of the information in the m= essage and its attachments. Our company shall have no liability for any cha= nges or late receiving, loss of integrity and confidentiality, viruses and = any damages caused in anyway to your computer system.

--001a1148e2e6706885052d0df485--