flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore alajangi <alajangikish...@gmail.com>
Subject Re: copy to hdfs
Date Mon, 16 Jun 2014 07:04:03 GMT
could anybody help me ?


On Mon, Jun 16, 2014 at 10:27 AM, kishore alajangi <
alajangikishore@gmail.com> wrote:

> Instead just mentioning hdsfs.path = /flume/messages/, do i need to
> mention something else?
>
>
> On Mon, Jun 16, 2014 at 10:25 AM, kishore alajangi <
> alajangikishore@gmail.com> wrote:
>
>> I created the /flume/messages directories, but still nothing is written
>> with flume in those directories. please help me.
>>
>>
>> On Mon, Jun 16, 2014 at 10:15 AM, kishore alajangi <
>> alajangikishore@gmail.com> wrote:
>>
>>> Do I need to create the /flume/messages/ directories?
>>>
>>>
>>>
>>> On Mon, Jun 16, 2014 at 10:14 AM, kishore alajangi <
>>> alajangikishore@gmail.com> wrote:
>>>
>>>> checked, nothing is written in hdfs.
>>>>
>>>>
>>>> On Mon, Jun 16, 2014 at 10:10 AM, Sharninder <sharninder@gmail.com>
>>>> wrote:
>>>>
>>>>> That just means source has done its work and is waiting for more data
>>>>> to read. Did you check hdfs to see if all data has been written?
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jun 16, 2014 at 11:34 AM, kishore alajangi <
>>>>> alajangikishore@gmail.com> wrote:
>>>>>
>>>>>> Hi Mohit and sharminder,
>>>>>>
>>>>>> Thanks for reply, after I called with -n tier, source is not
>>>>>> directory error came, I changed the source to /tmp/ and hdfs.path
to
>>>>>> /flume/messages/ in config file, and run the command, the INFO i
am getting
>>>>>> now is "spooling directory source runner has shutdown"
>>>>>> what could be the problem, please help me.
>>>>>>
>>>>>>
>>>>>> On Sun, Jun 15, 2014 at 10:21 PM, Mohit Durgapal <
>>>>>> durgapalmohit@gmail.com> wrote:
>>>>>>
>>>>>>> Replace -n agent with -n tier1
>>>>>>>
>>>>>>>
>>>>>>> On Sunday, June 15, 2014, kishore alajangi <
>>>>>>> alajangikishore@gmail.com> wrote:
>>>>>>>
>>>>>>>> Dear Sharminder,
>>>>>>>>
>>>>>>>> Thanks for your reply, yes I am playing with flume, as you
>>>>>>>> suggested i am using spool directory source, My configuration
file looks
>>>>>>>> like
>>>>>>>>
>>>>>>>> *tier1.sources  = source1
>>>>>>>> tier1.channels = channel1
>>>>>>>> tier1.sinks    = sink1
>>>>>>>>
>>>>>>>> tier1.sources.source1.type     = spooldir
>>>>>>>> tier1.sources.source1.spoolDir = /var/log/messages
>>>>>>>> tier1.sources.source1.channels = channel1
>>>>>>>> tier1.channels.channel1.type   = memory
>>>>>>>>
>>>>>>>> tier1.sinks.sink1.type         = hdfs
>>>>>>>> tier1.sinks.sink1.hdfs.path = hdfs://localhost:8020/flume/messages
>>>>>>>> tier1.sinks.sink1.hdfs.fileType = SequenceFile
>>>>>>>> tier1.sinks.sink1.hdfs.filePrefix = data
>>>>>>>> tier1.sinks.sink1.hdfs.fileSuffix = .seq
>>>>>>>>
>>>>>>>> # Roll based on the block size only
>>>>>>>> tier1.sinks.sink1.hdfs.rollCount=0
>>>>>>>> tier1.sinks.sink1.hdfs.rollInterval=0
>>>>>>>> tier1.sinks.sink1.hdfs.rollSize = 120000000
>>>>>>>> # seconds to wait before closing the file.
>>>>>>>> tier1.sinks.sink1.hdfs.idleTimeout = 60
>>>>>>>> tier1.sinks.sink1.channel      = channel1
>>>>>>>>
>>>>>>>> tier1.channels.channel1.capacity = 100000*
>>>>>>>> tier1.sources.source1.deserializer.maxLineLength = 32768
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> the command I used is
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ./flume-ng agent --conf ./conf/ -f bin/example.conf -Dflume.root.logger=DEBUG,console
-n agent
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> it gives warn after created sources, channels, sinks for
tier1 agent is
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> no configuration found for this host:agent
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> any help?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Jun 15, 2014 at 11:18 AM, Sharninder <sharninder@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I want to copy my local data to hdfs using flume in a
single
>>>>>>>>>> machine which isrunning hadoop, How can I do that,
please help me.
>>>>>>>>>>
>>>>>>>>>> What is this "local data" ?
>>>>>>>>>
>>>>>>>>> If it's just files, why not use the hadoop fs copy command
>>>>>>>>> instead? If you want to play around with flume, take
a look at the spool
>>>>>>>>> directory source or the exec source and you should be
able to put something
>>>>>>>>> together that'll push data through flume to hadoop.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Sharninder
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks,
>>>>>>>> Kishore.
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks,
>>>>>> Kishore.
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Kishore.
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Kishore.
>>>
>>
>>
>>
>> --
>> Thanks,
>> Kishore.
>>
>
>
>
> --
> Thanks,
> Kishore.
>



-- 
Thanks,
Kishore.

Mime
View raw message