flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammad Tariq <donta...@gmail.com>
Subject Re: Newbee question about flume 1.2 set up
Date Wed, 13 Jun 2012 20:57:14 GMT
Good luck :)
Regards,
    Mohammad Tariq


On Thu, Jun 14, 2012 at 2:07 AM, Bhaskar <bmarthi@gmail.com> wrote:
> Thank you both.  I am going to try it with both as POC.
>
> Bhaskar
>
>
> On Wed, Jun 13, 2012 at 4:28 PM, Mohammad Tariq <dontariq@gmail.com> wrote:
>>
>> As said by Ralph, Hdfs or Local fs are not the only options..you can
>> dump your data into any store like Cassandra, Hbase etc etc
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Thu, Jun 14, 2012 at 1:54 AM, Ralph Goers <ralph.goers@dslextreme.com>
>> wrote:
>> > FWIW, Hadoop is not the only target repository. We are using Cassandra.
>> >
>> > Ralph
>> >
>> > On Jun 13, 2012, at 1:19 PM, Mohammad Tariq wrote:
>> >
>> >> it's absolutely ok for initial learning, but not feasible for
>> >> production or if you want to evaluate hadoop ecosystem properly..I
>> >> would suggest to setup a hadoop cluster in pseudo mode on your pc
>> >> first..or if you do not require hadoopat all then there is no problem.
>> >>
>> >> Regards,
>> >>     Mohammad Tariq
>> >>
>> >>
>> >> On Thu, Jun 14, 2012 at 1:08 AM, Bhaskar <bmarthi@gmail.com> wrote:
>> >>> Thank you Mohammad for prompt response.  I built it from source and
>> >>> have
>> >>> tried few combinations so far.  I do not have HDFS set up yet.  I
was
>> >>> trying
>> >>> to use local file system as sink.  Is that a feasible option?
>> >>>
>> >>> Bhaskar
>> >>>
>> >>>
>> >>> On Wed, Jun 13, 2012 at 1:45 PM, Mohammad Tariq <dontariq@gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> Hello Bhaskar,
>> >>>>
>> >>>>         The very first step would be to build flume-ng from
>> >>>> trunk..you can use following commands to do that -
>> >>>>
>> >>>> $ svn co https://svn.apache.org/repos/asf/incubator/flume/trunk
flume
>> >>>> $ cd flume
>> >>>> $ mvn3 install -DskipTests
>> >>>>
>> >>>> I would suggest to use Maven 3.0.3, as you may find some problems
>> >>>> with
>> >>>> Maven2..Once you are done with your build, you need to write the
>> >>>> configuration files for your agents..For example, to collect apache
>> >>>> web server logs into  the Hdfs, the basic configuration would be
>> >>>> something like this -
>> >>>>
>> >>>> agent1.sources = tail
>> >>>> agent1.channels = MemoryChannel-2
>> >>>> agent1.sinks = HDFS
>> >>>>
>> >>>> agent1.sources.tail.type = exec
>> >>>> agent1.sources.tail.command = tail -F /var/log/apache2/access.log
>> >>>> agent1.sources.tail.channels = MemoryChannel-2
>> >>>>
>> >>>> agent1.sinks.HDFS.channel = MemoryChannel-2
>> >>>> agent1.sinks.HDFS.type = hdfs
>> >>>> agent1.sinks.HDFS.hdfs.path = hdfs://localhost:9000/flume
>> >>>> agent1.sinks.HDFS.hdfs.file.Type = DataStream
>> >>>>
>> >>>> agent1.channels.MemoryChannel-2.type = memory
>> >>>>
>> >>>> Save this file as agent1.conf inside your flume-ng/conf directory
and
>> >>>> start your agent using -
>> >>>>
>> >>>> $ bin/flume-ng agent -n agent1 -f conf/agent1.conf
>> >>>>
>> >>>>
>> >>>> Regards,
>> >>>>     Mohammad Tariq
>> >>>>
>> >>>>
>> >>>> On Wed, Jun 13, 2012 at 10:45 PM, Bhaskar <bmarthi@gmail.com>
wrote:
>> >>>>> Good Afternoon,
>> >>>>> I am a newbee to flume and read thru limited documentation
>> >>>>> available.  I
>> >>>>> would like to set up the following to test out.
>> >>>>>
>> >>>>> 1.  Read apache access logs (as source)
>> >>>>> 2.  Use memory channel
>> >>>>> 3.  Write it to a NFS (or even local) file system
>> >>>>>
>> >>>>> Can some one help me with the necessary configuration.  I am
having
>> >>>>> difficult time to glean that information from available
>> >>>>> documentation.
>> >>>>>  I am
>> >>>>> sure someone has done such test before and i appreciate if you
can
>> >>>>> pass
>> >>>>> on
>> >>>>> that information.  Secondly, I also would like to stream the
logs to
>> >>>>> a
>> >>>>> remote server.  Is that a log4j configuration or do i need
to run an
>> >>>>> agent
>> >>>>> on each host to do so?  Any configuration examples would be
of great
>> >>>>> help.
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Bhaskar
>> >>>
>> >>>
>> >
>
>

Mime
View raw message