Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
Received-SPF: pass (athena.apache.org: domain of otto@wikimedia.org designates
 209.85.212.44 as permitted sender)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\))
Subject: Re: Need for UDP / Multicast Source
From: Andrew Otto <otto@wikimedia.org>
In-Reply-To: <15D9FF84-2273-49C9-BCBD-7C6C747153D6@wikimedia.org>
Date: Thu, 17 Jan 2013 11:26:34 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <098E9C31-4846-4AED-B675-7B87AEABE07C@wikimedia.org>
References: <5654382F-6A12-433F-B375-212663D14CD3@wikimedia.org>
 <30D5941F8B9A4248BEEB50C06687A957@cloudera.com>
 <0598ACC3-FE31-4F72-9AF7-B9B79C16FDF6@gmail.com>
 <5967102B-1710-4472-8A8B-9BA244E92B33@wikimedia.org>
 <A8E07B2B-8560-4542-B3E0-2693695EF62E@wikimedia.org>
 <9BAF63F3-1CFF-46EA-A420-301E1CCEFBA2@wikimedia.org>
 <CAFukC=53hqcawtSjS89NzsJUZXiWhAHdyfu9-a_RKq==BDahcg@mail.gmail.com>
 <B02B9813-8BD6-44ED-9335-CBD527D9978B@wikimedia.org>
 <CAFukC=7dE3KxGOSGrJ_XRwhp9pB9PimV5tK4exaT=Oz=S8BTeA@mail.gmail.com>
 <B9EBC5B131D246F181C3193300C2145D@cloudera.com>
 <029748F1-8B14-4F6E-B68E-17671E8BC4E7@wikimedia.org>
 <A81B7424F0454AEAA066360E376D3325@cloudera.com>
 <CAA8wT0mEK8X0DD1w8-xL5LvaiLXKm3Hr-q1RP8Kp1bcx4gsRsA@mail.gmail.com>
 <15D9FF84-2273-49C9-BCBD-7C6C747153D6@wikimedia.org>
To: user@flume.apache.org

Ok, I'm still struggling with this a bit.  Here's what I've currently =
got going.

In order to make it easier to check what I am and am not receiving, I've =
narrowed the logs that I store in HDFS down to those originating from a =
single host (cp1044.wikimedia.org).  Each host generates contiguous =
sequence numbers for each log line.  I can use the sequence number to =
make sure I'm not missing lines from a host.

On another nearby node, I started a process to store all of the log =
lines originating from this cp1044.  I then started the Flume agent and =
waited a 3 minutes for it to roll files 3 times.  I currently have 4 =
HDFS sinks going, so this created a total of 12 files.  I got the files =
out of HDFS, and then sorted on their sequence numbers to gain the first =
and last sequence number in this set of files. =20

I took those two border sequence numbers and extracted all of the log =
lines generated by cp1044 on the nearby host (not using Flume).  I =
should be able to compare the number of lines here with the number of =
lines in the 12 files I extracted from HDFS and Flume.  If they are the =
same, then Flume and UDPSource is working!

Flume saved 19451 events to HDFS, and the number of raw events recorded =
outside of Flume and HDFS was 78176.  I'm up to about 25% of data!  =
Better but still not good enough. :(

This was for about 3 minutes of data, so for a single host, this =
shouldn't be more than 500 events per second.  I must be doing something =
really wrong on the Flume tweaky side of things, eh?  Any more ideas?

Thanks!

P.S.  YOU GUYS ARE SO HELPFUL.  Thanks so much for everything thus far.


On Jan 17, 2013, at 10:34 AM, Andrew Otto <otto@wikimedia.org> wrote:

>> with UDP there's no guaranty that the data will reach destination.
>=20
> True, but I'm experimenting with using Flume as a replacement for a =
system that is already in place.  I actually got the numbers I listed =
below by grabbing data directly off of the UDP stream and saving them to =
a file on local disk.  Its possible that UDP data is getting lost in the =
network somewhere, but if that were the case I wouldn't know about it.  =
I am comparing Flume's performance to a single process writing to a =
local disk.
>=20
>=20