flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sharninder <sharnin...@gmail.com>
Subject Re: copy to hdfs
Date Mon, 16 Jun 2014 06:10:19 GMT
That just means source has done its work and is waiting for more data to
read. Did you check hdfs to see if all data has been written?



On Mon, Jun 16, 2014 at 11:34 AM, kishore alajangi <
alajangikishore@gmail.com> wrote:

> Hi Mohit and sharminder,
>
> Thanks for reply, after I called with -n tier, source is not directory
> error came, I changed the source to /tmp/ and hdfs.path to /flume/messages/
> in config file, and run the command, the INFO i am getting now is "spooling
> directory source runner has shutdown"
> what could be the problem, please help me.
>
>
> On Sun, Jun 15, 2014 at 10:21 PM, Mohit Durgapal <durgapalmohit@gmail.com>
> wrote:
>
>> Replace -n agent with -n tier1
>>
>>
>> On Sunday, June 15, 2014, kishore alajangi <alajangikishore@gmail.com>
>> wrote:
>>
>>> Dear Sharminder,
>>>
>>> Thanks for your reply, yes I am playing with flume, as you suggested i
>>> am using spool directory source, My configuration file looks like
>>>
>>> *tier1.sources  = source1
>>> tier1.channels = channel1
>>> tier1.sinks    = sink1
>>>
>>> tier1.sources.source1.type     = spooldir
>>> tier1.sources.source1.spoolDir = /var/log/messages
>>> tier1.sources.source1.channels = channel1
>>> tier1.channels.channel1.type   = memory
>>>
>>> tier1.sinks.sink1.type         = hdfs
>>> tier1.sinks.sink1.hdfs.path = hdfs://localhost:8020/flume/messages
>>> tier1.sinks.sink1.hdfs.fileType = SequenceFile
>>> tier1.sinks.sink1.hdfs.filePrefix = data
>>> tier1.sinks.sink1.hdfs.fileSuffix = .seq
>>>
>>> # Roll based on the block size only
>>> tier1.sinks.sink1.hdfs.rollCount=0
>>> tier1.sinks.sink1.hdfs.rollInterval=0
>>> tier1.sinks.sink1.hdfs.rollSize = 120000000
>>> # seconds to wait before closing the file.
>>> tier1.sinks.sink1.hdfs.idleTimeout = 60
>>> tier1.sinks.sink1.channel      = channel1
>>>
>>> tier1.channels.channel1.capacity = 100000*
>>> tier1.sources.source1.deserializer.maxLineLength = 32768
>>>
>>>
>>>
>>> the command I used is
>>>
>>>
>>>
>>> ./flume-ng agent --conf ./conf/ -f bin/example.conf -Dflume.root.logger=DEBUG,console
-n agent
>>>
>>>
>>>
>>> it gives warn after created sources, channels, sinks for tier1 agent is
>>>
>>>
>>>
>>> no configuration found for this host:agent
>>>
>>>
>>>
>>> any help?
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Jun 15, 2014 at 11:18 AM, Sharninder <sharninder@gmail.com>
>>> wrote:
>>>
>>>>
>>>> I want to copy my local data to hdfs using flume in a single machine
>>>>> which isrunning hadoop, How can I do that, please help me.
>>>>>
>>>>> What is this "local data" ?
>>>>
>>>> If it's just files, why not use the hadoop fs copy command instead? If
>>>> you want to play around with flume, take a look at the spool directory
>>>> source or the exec source and you should be able to put something together
>>>> that'll push data through flume to hadoop.
>>>>
>>>> --
>>>> Sharninder
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Kishore.
>>>
>>
>
>
> --
> Thanks,
> Kishore.
>

Mime
View raw message