Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1D03B1004D for ; Sun, 7 Apr 2013 22:15:19 +0000 (UTC) Received: (qmail 4656 invoked by uid 500); 7 Apr 2013 22:15:18 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 4600 invoked by uid 500); 7 Apr 2013 22:15:18 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 4592 invoked by uid 99); 7 Apr 2013 22:15:18 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 07 Apr 2013 22:15:18 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of sandeepbaldawa@gmail.com designates 209.85.214.51 as permitted sender) Received: from [209.85.214.51] (HELO mail-bk0-f51.google.com) (209.85.214.51) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 07 Apr 2013 22:15:14 +0000 Received: by mail-bk0-f51.google.com with SMTP id y8so2771498bkt.24 for ; Sun, 07 Apr 2013 15:14:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=RgNJFpYePlJ42Y+SUs5W4T/pVi4MwEorg+M1zOah+rY=; b=qZ4jdp9i5/QHaNj9iTLev0CWFpKhPUUl6UfDqRwIJI8vs703WbPmxWLcPihFA5nOWY oQm3MHVtgkXFndlvkdC/nqSjN5pqn5UMm/ds1YlAC7dwdScuymnK4ckgcLOKEO1DejZE l/O2qtY3Aqugtzj2xQp7/OrZH8eTXOG6FiioCkTk1BJyNxhNb+JgGjgSbYrh35el0Ckf 4kKUtPgQch/70p05hihbvJlBX5sVuFpZpJcBzjiLYJ8qNiutp0vXxUucit+DDWK8Z+j9 QDew4mvKN5X+OI5xbQvk8GrEfT5oUFjYY+t97G6swf9Egba8Bsr5tHlmWApERe2mrEOL qXkg== MIME-Version: 1.0 X-Received: by 10.205.0.193 with SMTP id nn1mr10240593bkb.94.1365372893299; Sun, 07 Apr 2013 15:14:53 -0700 (PDT) Received: by 10.204.148.143 with HTTP; Sun, 7 Apr 2013 15:14:53 -0700 (PDT) In-Reply-To: References: Date: Sun, 7 Apr 2013 15:14:53 -0700 Message-ID: Subject: Re: Best way for a newbie to learn flume From: Sandeep Baldawa To: user Content-Type: multipart/alternative; boundary=20cf30223b514ab60f04d9cca695 X-Virus-Checked: Checked by ClamAV on apache.org --20cf30223b514ab60f04d9cca695 Content-Type: text/plain; charset=ISO-8859-1 Thanks again for all the details. I will follow the steps described by you. On Sun, Apr 7, 2013 at 2:58 PM, Israel Ekpo wrote: > Also the link below is associated with Flume OG and not Flume NG > > http://archive.cloudera.com/cdh/3/flume/UserGuide/ > > The architecture and features have changed significantly since that version > > > On 7 April 2013 17:54, Israel Ekpo wrote: > >> Sandeep, >> >> So Flume currently has two tracks: >> >> Flume OG (not actively supported) >> https://cwiki.apache.org/confluence/display/FLUME/Flume+OG+%28pre+1.0%29 >> Flume NG (currently active) >> https://cwiki.apache.org/confluence/display/FLUME/Flume+NG >> >> The latest stable version for Flume NG is 1.3.1 >> >> The NG stands for Next Generation and it is the current active >> development track. >> >> The OG refers to the Original Generation of Flume. This includes releases >> before the 1.0.0 release. >> >> New comers and existing users of the OG track are encouraged to migrate >> over to the NG track. >> >> You can download Flume NG 1.3.1 here >> >> http://flume.apache.org/download.html >> >> Regarding "Getting Started", in the next couple of weeks, additional >> information will be added to Wiki to make the on-boarding process easier >> for new comers. >> >> In the time being, please bear with us. >> >> I would recommend you download and install the latest version of Java 1.6. >> >> Then download Flume and extract it to folder in your directory. >> >> Then you can use the following sources, channels and sinks to get started. >> >> This is the best way for you to learn and understand the various pieces. >> >> SOURCE: Spooling Directory Source >> CHANNEL: File Channel (more reliable) or Memory Channel (faster) >> SINK: File Roll Sink >> >> You can create a directory that you will be spooling and dump a couple of >> log files in there. Make sure the files a new-line delimited. >> >> Each line will represent an event in the log files. >> >> Then configure the file channel and the file roll sink using guidelines >> and examples available in the user guide. >> >> http://flume.apache.org/FlumeUserGuide.html >> >> That will give you a feel for how flume works. >> >> Once you have that set up then you can run the agent and see what happens. >> >> Once you start getting the hang of things you can try other sources and >> sinks or maybe even create a few of your own custom sources, channels or >> sinks. >> >> >> >> On 7 April 2013 17:10, Sandeep Baldawa wrote: >> >>> >>> Thanks for the detailed reply. >>> >>> Awesome questions and I should have added these details in my question, >>> am learning flume more as a hobby, learning experience and a tech >>> enthusiastic(heard pretty good things about flume). >>> >>> Thanks again for the instructions. Just one question about setting >>> things up, are instructions at >>> http://archive.cloudera.com/cdh/3/flume/UserGuide/ relevant with the >>> latest build?, I liked the documentation in this link which has a quick >>> start guide too. >>> >>> >>> On Sun, Apr 7, 2013 at 1:28 PM, Israel Ekpo wrote: >>> >>>> Sandeep, >>>> >>>> Excellent questions. >>>> >>>> You asked "what problem Flume is trying to solve?". >>>> >>>> It think the more appropriate question is what problem you are trying >>>> to solve? >>>> >>>> This will go a long way in helping us understand which components of >>>> Flume you may need and how you need to set it up. >>>> >>>> Are you using Flume as part of your job or personal hubby? Are you >>>> using Flume for a course at school or part of an academic project? >>>> >>>> Going back to your original question, in the simplest terms, and for >>>> most use cases, Flume is a system designed for collecting and transporting >>>> large amounts of data and events from one or more sources and then >>>> aggregating the collected data in a centralized data store or for onward >>>> propagation to subsequent sources. >>>> >>>> You can use it for aggregating data from log files, network traffic, >>>> click streams, twitter and any other source that can generate events. >>>> >>>> Spend more time to review the user guide and you will find a lot of >>>> information and answers to prospective questions. >>>> >>>> http://flume.apache.org/FlumeUserGuide.html >>>> >>>> To install flume you will need to set up Java 1.6 and then make sure >>>> that it is available in your PATH and then download the latest version of >>>> Flume and decompress the tarball or zip file. >>>> >>>> You will need to set up the configuration file(s) for the agents based >>>> on the sources, channels and sinks you choose to use. >>>> >>>> I would recommend that you go ahead and get started with setting it up >>>> and let us know if you run into any issues. >>>> >>>> If you can share which use case and what problem you are trying to >>>> solve, someone can point you in the right direction. >>>> >>>> >>>> >>>> On 7 April 2013 14:45, Sandeep Baldawa wrote: >>>> >>>>> Hello, >>>>> >>>>> I am a new user for flume and was wondering what was the best way to >>>>> learn about flume from a user's perspective. >>>>> >>>>> I am interested more in getting answers for the following questions >>>>> - What problem is flume trying to solve. >>>>> - How to install flume in the most simple way to understand the >>>>> concepts. >>>>> - Am trying to follow http://flume.apache.org/FlumeUserGuide.html#and am not sure if I can find a quick start guide here, can someone point >>>>> me to the correct link if possible. >>>>> >>>>> Thanks, >>>>> Sandeep >>>>> >>>> >>>> >>> >> > --20cf30223b514ab60f04d9cca695 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Thanks again for all the details. I will follow the steps = described by you.


On Sun, Apr 7, 2013 at 2:58 PM, Israel Ekpo <israel@aicer.org= > wrote:
Also the link below is associated with Flume= OG and not Flume NG


The architecture and features have changed significantly since that version=


On 7 April 2013 17:54, Israel Ekpo <israel@aicer.org> wrote:
Sandeep,

So Flume current= ly has two tracks:


The latest st= able version for Flume NG is 1.3.1

The NG stands for Next Generation and it is the current= active development track.

The OG refers to the Or= iginal Generation of Flume. This includes releases before the 1.0.0 release= .

New comers and existing users of the OG track are encou= raged to migrate over to the NG track.

You can dow= nload Flume NG 1.3.1 here


Regarding "Getting Started", in the next coup= le of weeks, additional information will be added to Wiki to make the on-bo= arding process easier for new comers.

In the time = being, please bear with us.

I would recommend you download and install the latest v= ersion of Java 1.6.

Then download Flume and extrac= t it to folder in your directory.

Then you can use= the following sources, channels and sinks to get started.

This is the best way for you to learn and understand th= e various pieces.

SOURCE: Spooling Directory Sourc= e
CHANNEL: File Channel (more reliable) or Memory Channel (faster= )
SINK: File Roll Sink

You can create a directo= ry that you will be spooling and dump a couple of log files in there. Make = sure the files a new-line delimited.

Each line wil= l represent an event in the log files.

Then configure the file channel and the file roll sink = using guidelines and examples available in the user guide.


That will give you a feel for how flume works.

Once you have that set up then you can run the agent and s= ee what happens.

Once you start getting the hang o= f things you can try other sources and sinks or maybe even create a few of = your own custom sources, channels or sinks.



On 7 April 2013 17:10, Sandeep Baldawa <sandeepbaldawa@gmail.com<= /a>> wrote:


On Sun, Apr 7, 2013 at 1:28 PM, Israel Ekpo <israel@aicer.org> wrote:
Sandeep,

Excellent questions.

Y= ou asked "what problem Flume is trying to solve?".

=
It think the more appropriate question is what problem you are t= rying to solve?

This will go a long way in helping us understand which = components of Flume you may need and how you need to set it up.
<= br>
Are you using Flume as part of your job or personal hubby? Ar= e you using Flume for a course at school or part of an academic project?

Going back to your original question, in the simplest t= erms, and for most use cases, Flume is a system designed for collecting and= transporting large amounts of data and events from one or more sources and= then aggregating the collected data in a centralized data store or for onw= ard propagation to subsequent sources.

You can use it for aggregating data from log files, net= work traffic, click streams, twitter and any other source that can generate= events.

Spend more time to review the user guide = and you will find a lot of information and answers to prospective questions= .


To install flume you will need to set up Java 1.6 and then= make sure that it is available in your PATH and then download the latest v= ersion of Flume and decompress the tarball or zip file.

You will need to set up the configuration file(s) for t= he agents based on the sources, channels and sinks you choose to use.
=

I would recommend that you go ahead and get started wit= h setting it up and let us know if you run into any issues.

If you can share which use case and what problem you ar= e trying to solve, someone can point you in the right direction.





--20cf30223b514ab60f04d9cca695--