flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore alajangi <alajangikish...@gmail.com>
Subject Re: copy to hdfs
Date Mon, 16 Jun 2014 06:04:25 GMT
Hi Mohit and sharminder,

Thanks for reply, after I called with -n tier, source is not directory
error came, I changed the source to /tmp/ and hdfs.path to /flume/messages/
in config file, and run the command, the INFO i am getting now is "spooling
directory source runner has shutdown"
what could be the problem, please help me.


On Sun, Jun 15, 2014 at 10:21 PM, Mohit Durgapal <durgapalmohit@gmail.com>
wrote:

> Replace -n agent with -n tier1
>
>
> On Sunday, June 15, 2014, kishore alajangi <alajangikishore@gmail.com>
> wrote:
>
>> Dear Sharminder,
>>
>> Thanks for your reply, yes I am playing with flume, as you suggested i am
>> using spool directory source, My configuration file looks like
>>
>> *tier1.sources  = source1
>> tier1.channels = channel1
>> tier1.sinks    = sink1
>>
>> tier1.sources.source1.type     = spooldir
>> tier1.sources.source1.spoolDir = /var/log/messages
>> tier1.sources.source1.channels = channel1
>> tier1.channels.channel1.type   = memory
>>
>> tier1.sinks.sink1.type         = hdfs
>> tier1.sinks.sink1.hdfs.path = hdfs://localhost:8020/flume/messages
>> tier1.sinks.sink1.hdfs.fileType = SequenceFile
>> tier1.sinks.sink1.hdfs.filePrefix = data
>> tier1.sinks.sink1.hdfs.fileSuffix = .seq
>>
>> # Roll based on the block size only
>> tier1.sinks.sink1.hdfs.rollCount=0
>> tier1.sinks.sink1.hdfs.rollInterval=0
>> tier1.sinks.sink1.hdfs.rollSize = 120000000
>> # seconds to wait before closing the file.
>> tier1.sinks.sink1.hdfs.idleTimeout = 60
>> tier1.sinks.sink1.channel      = channel1
>>
>> tier1.channels.channel1.capacity = 100000*
>> tier1.sources.source1.deserializer.maxLineLength = 32768
>>
>>
>> the command I used is
>>
>>
>> ./flume-ng agent --conf ./conf/ -f bin/example.conf -Dflume.root.logger=DEBUG,console
-n agent
>>
>>
>> it gives warn after created sources, channels, sinks for tier1 agent is
>>
>>
>> no configuration found for this host:agent
>>
>>
>> any help?
>>
>>
>>
>>
>> On Sun, Jun 15, 2014 at 11:18 AM, Sharninder <sharninder@gmail.com>
>> wrote:
>>
>>>
>>> I want to copy my local data to hdfs using flume in a single machine
>>>> which isrunning hadoop, How can I do that, please help me.
>>>>
>>>> What is this "local data" ?
>>>
>>> If it's just files, why not use the hadoop fs copy command instead? If
>>> you want to play around with flume, take a look at the spool directory
>>> source or the exec source and you should be able to put something together
>>> that'll push data through flume to hadoop.
>>>
>>> --
>>> Sharninder
>>>
>>>
>>
>>
>> --
>> Thanks,
>> Kishore.
>>
>


-- 
Thanks,
Kishore.

Mime
View raw message