Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3FB8518613 for ; Thu, 14 May 2015 03:57:04 +0000 (UTC) Received: (qmail 57838 invoked by uid 500); 14 May 2015 03:57:03 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 57781 invoked by uid 500); 14 May 2015 03:57:03 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 57771 invoked by uid 99); 14 May 2015 03:57:03 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 May 2015 03:57:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 60A3918291E for ; Thu, 14 May 2015 03:57:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.002 X-Spam-Level: *** X-Spam-Status: No, score=3.002 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id W6d3Fy0ShNuC for ; Thu, 14 May 2015 03:56:50 +0000 (UTC) Received: from mail-pa0-f46.google.com (mail-pa0-f46.google.com [209.85.220.46]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 3AC5F21784 for ; Thu, 14 May 2015 03:56:49 +0000 (UTC) Received: by pabtp1 with SMTP id tp1so70836954pab.2 for ; Wed, 13 May 2015 20:55:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:message-id:references:to; bh=qDBucevN4N+bPOQwZqMZv3QLURqLZwr8CwcEYMX6XI8=; b=LFTp7OSOkdVQAai5etgAPUoL8BOYnEnu3/ktg6YfHoLQRLYBCTn2dGaZYMIuMm8nm5 zNIJ9+ZLYa4x0ToUXCyWErL/QezuK1aAw6svh6L2ZPn26d8IMKXd4iiH5+tpmja2GzjF VlUOA9NUSPd9Otp5Qq6qypz3wdzvz1tRmzaZnBG7x8eN00WiJjz0ev5shzLBFrD75mQ6 1V39+F/CM0DDuA7aGwZwz20WO9PZSD2rCxPyRcPXxxFXq1C5Fjx1Uhd4FdGJwWnDz/jn 4yreE02BYLKBqLJM+SF4n8fPHx4fvYK+NnP1M0j0vTJ2VMnK2kLXgdpSSEl+1eErmWrK 9yKQ== X-Gm-Message-State: ALoCoQkMrbzWAc5KjvPjlA5NSMRbwhmdto8i5+M+9tg89fwXfVWw37psTNK1tvE73keyAk1EcQhl X-Received: by 10.66.243.69 with SMTP id ww5mr4052007pac.106.1431575717537; Wed, 13 May 2015 20:55:17 -0700 (PDT) Received: from [192.168.2.63] (c-76-103-147-3.hsd1.ca.comcast.net. [76.103.147.3]) by mx.google.com with ESMTPSA id e5sm20733006pdc.94.2015.05.13.20.55.16 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 13 May 2015 20:55:16 -0700 (PDT) Content-Type: multipart/alternative; boundary="Apple-Mail=_6ABCF8E0-C5AD-4946-A97D-7F2A2B9D6AAF" Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2098\)) Subject: Re: set flume send logs to hdfs error From: Hari Shreedharan X-Priority: 3 In-Reply-To: Date: Wed, 13 May 2015 20:55:15 -0700 Message-Id: <90622FB2-B856-4A25-8CC4-B64240C58513@cloudera.com> References: To: user@flume.apache.org X-Mailer: Apple Mail (2.2098) --Apple-Mail=_6ABCF8E0-C5AD-4946-A97D-7F2A2B9D6AAF Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 You need to use Flume=E2=80=99s client API to send data to the Avro = Source. Use the thrift source, and generate thrift client for Python = using this IDL file: = https://github.com/apache/flume/blob/trunk/flume-ng-sdk/src/main/thrift/fl= ume.thrift = You can use that to send data to the thrift source at that point. Thanks, Hari Shreedharan > On May 13, 2015, at 8:06 PM, =E9=B9=B0 <980548079@qq.com> wrote: >=20 > I send data by python scripts use socket send the code like this: >=20 > import sys=20 > from socket import * >=20 >=20 > HOST =3D '192.168.1.117' = = = =20 > PORT =3D44444 > BUFSIZ =3D 1024 > ADDR =3D (HOST, PORT) >=20 > tcpCliSock =3D socket(AF_INET, SOCK_STREAM) > tcpCliSock.connect(ADDR) > i=3D0 > for x in range(3): > print x, "xx" > n=3DtcpCliSock.send("test datas from flume") > tcpCliSock.close() >=20 >=20 >=20 > ------------------ =E5=8E=9F=E5=A7=8B=E9=82=AE=E4=BB=B6 = ------------------ > =E5=8F=91=E4=BB=B6=E4=BA=BA: "Hari = Shreedharan";; > =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2015=E5=B9=B45=E6=9C=8814=E6=97=A5= (=E6=98=9F=E6=9C=9F=E5=9B=9B) =E4=B8=8A=E5=8D=8810:53 > =E6=94=B6=E4=BB=B6=E4=BA=BA: = "user@flume.apache.org"; > =E4=B8=BB=E9=A2=98: Re: set flume send logs to hdfs error >=20 > How are you sending data to the Avro Source? >=20 >=20 > Thanks, > Hari >=20 > On Wed, May 13, 2015 at 7:38 PM, =E9=B9=B0 <980548079@qq.com = > wrote: > hi all , > i'm want set flume send data to hdfs my configure file is lile this : > tier1.sources=3Dsource1 =20 > tier1.channels=3Dchannel1 =20 > tier1.sinks=3Dsink1 =20 >=20 > tier1.sources.source1.type=3Davro =20 > tier1.sources.source1.bind=3D0.0.0.0 =20 > tier1.sources.source1.port=3D44444 =20 > tier1.sources.source1.channels=3Dchannel1 =20 >=20 > tier1.channels.channel1.type=3Dmemory =20 > tier1.channels.channel1.capacity=3D10000 =20 > tier1.channels.channel1.transactionCapacity=3D1000 =20 > tier1.channels.channel1.keep-alive=3D30 =20 >=20 > tier1.sinks.sink1.type=3Dhdfs =20 > tier1.sinks.sink1.channel=3Dchannel1 =20 > tier1.sinks.sink1.hdfs.path=3Dhdfs://hadoop-home.com:9000/user/hadoop/ = =20 > tier1.sinks.sink1.hdfs.fileType=3DDataStream =20 > tier1.sinks.sink1.hdfs.writeFormat=3DText =20 > tier1.sinks.sink1.hdfs.rollInterval=3D0 =20 > tier1.sinks.sink1.hdfs.rollSize=3D10240 =20 > tier1.sinks.sink1.hdfs.rollCount=3D0 =20 > tier1.sinks.sink1.hdfs.idleTimeout=3D60 =20 >=20 > when I start the flume by this configure file and send data to the = port 44444 I get an error : > org.apache.avro.AvroRuntimeException: Excessively large list = allocation request detected: 154218761 items! Connection closed; > dose anybody can help me ,thanks. >=20 --Apple-Mail=_6ABCF8E0-C5AD-4946-A97D-7F2A2B9D6AAF Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 You need to use Flume=E2=80=99s client API to send data to = the Avro Source. Use the thrift source, and generate thrift client for = Python using this IDL file: https://github.com/apache/flume/blob/trunk/flume-ng-sdk/src/mai= n/thrift/flume.thrift

You can use that to send data to the thrift source at that = point.

Thanks,
Hari = Shreedharan




On May 13, 2015, at 8:06 PM, =E9=B9=B0 <980548079@qq.com> = wrote:

I = send data by python scripts use socket send the code like this:

import sys
from socket import = *


HOST =3D = '192.168.1.117'          = ;            &= nbsp;           &nb= sp;            = ;            &= nbsp;           &nb= sp;            = ;            &= nbsp;           &nb= sp;            = ;            &= nbsp;           &nb= sp;            = ;            &= nbsp;           &nb= sp;            = ;            &= nbsp;        
PORT = =3D44444
BUFSIZ =3D 1024
ADDR =3D (HOST, = PORT)

tcpCliSock =3D socket(AF_INET, = SOCK_STREAM)
tcpCliSock.connect(ADDR)
i=3D0for x in range(3):
    print x, = "xx"
    n=3DtcpCliSock.send("test datas = from flume")
tcpCliSock.close()



------------------ =E5=8E=9F=E5=A7=8B=E9=82=AE=E4=BB=B6&nb= sp;------------------
=E5=8F=91=E4=BB=B6=E4=BA=BA: "Hari = Shreedharan";<hshreedharan@cloudera.com>;
=E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2015=E5=B9=B45=E6= =9C=8814=E6=97=A5(=E6=98=9F=E6=9C=9F=E5=9B=9B) =E4=B8=8A=E5=8D=8810:53
=E6=94=B6=E4=BB=B6=E4=BA=BA: "user@flume.apache.org"<user@flume.apache.org>;
=E4=B8=BB=E9=A2=98:&nbs= p;Re: set flume send logs to hdfs error

How are you sending data to = the Avro Source?


Thanks,
Hari

On Wed, May 13, 2015 at 7:38 = PM, =E9=B9=B0 <980548079@qq.com> wrote:
hi all ,
 i'm want set flume send data to hdfs my configure file = is lile this :
tier1.sources=3Dsource1 
tier1.channels=3Dchannel1 
tier1.sinks=3Dsink1 

tier1.sources.source1.type=3Davro 
tier1.sources.source1.bind=3D0.0.0.0 
tier1.sources.source1.port=3D44444 
tier1.sources.source1.channels=3Dchannel1 

tier1.channels.channel1.type=3Dmemory  =
tier1.channels.channel1.capacity=3D10000 
tier1.channels.channel1.transactionCapacity=3D1000 
tier1.channels.channel1.keep-alive=3D30 

tier1.sinks.sink1.type=3Dhdfs 
tier1.sinks.sink1.channel=3Dchannel1 
tier1.sinks.sink1.hdfs.path=3Dhdfs://hadoop-home.com:9000/user/hadoop/
tier1.sinks.sink1.hdfs.fileType=3DDataStream 
tier1.sinks.sink1.hdfs.writeFormat=3DText 
tier1.sinks.sink1.hdfs.rollInterval=3D0 
tier1.sinks.sink1.hdfs.rollSize=3D10240 
tier1.sinks.sink1.hdfs.rollCount=3D0 
tier1.sinks.sink1.hdfs.idleTimeout=3D60 

when I start the flume by this configure file = and send data to the port 44444 I get an error :
org.apache.avro.AvroRuntimeException: Excessively large list = allocation request detected: 154218761 items! Connection closed;
dose anybody can help me ,thanks.


= --Apple-Mail=_6ABCF8E0-C5AD-4946-A97D-7F2A2B9D6AAF--