flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Will McQueen <w...@cloudera.com>
Subject Re: How to upload the SEQ data into hdfs
Date Tue, 24 Jul 2012 02:46:20 GMT
Or as Brock said, you can refer to the link he posted and use the example
from the user guide instead, then you'll need to include this:

agent.sources = avro-AppSrv-sourceagent.sinks =
hdfs-Cluster1-sinkagent.channels = mem-channel-1

... but that example uses an Avro source so you'll likely need to start an
avro-client to test (or use Flume SDK). Or just change the source type to
SEQ.

Cheers,
Will

On Mon, Jul 23, 2012 at 6:07 PM, mardan Khan <mardan8310@gmail.com> wrote:

>
>
>
> Thanks Brocks,
>
> I have just gone through the posted link and just copy past the one of
> configuration file  and change the hdfs path as below:
>
>
>
> # properties of avro-AppSrv-source
> agent.sources.avro-AppSrv-source.type = avro
> agent.sources.avro-AppSrv-source.bind = localhost
> agent.sources.avro-AppSrv-source.port = 10000
>
> # properties of mem-channel-1
> agent.channels.mem-channel-1.type = memory
> agent.channels.mem-channel-1.capacity = 1000
> agent.channels.mem-channel-1.transactionCapacity = 100
>
> # properties of hdfs-Cluster1-sink
> agent.sinks.hdfs-Cluster1-sink.type = hdfs
> agent.sinks.hdfs-Cluster1-sink.hdfs.path = hdfs://
> 134.83.35.24/user/mardan/flume/
>
>
> apply the following command:
>
> $  /usr/bin/flume-ng agent -n agent -c conf -f
> /usr/lib/flume-ng/conf/flume.conf
>
>
> and got the following error. Most of the time of getting this error
>
> 12/07/24 01:54:43 ERROR properties.PropertiesFileConfigurationProvider:
> Failed to load configuration data. Exception follows.
> java.lang.NullPointerException
>     at
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSources(PropertiesFileConfigurationProvider.java:324)
>     at
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:222)
>     at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
>     at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
>     at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
>     at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>     at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>     at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>     at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>     at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:662)
>
> I think some thing wrong in the configuration file. I am using flume1.x
> version and installed in /usr/lib/flume-ng/
>
> Could you please check the command and configuration file.
>
> Thanks
>
>
>
>
>
>
>
> On Tue, Jul 24, 2012 at 1:33 AM, Brock Noland <brock@cloudera.com> wrote:
>
>> Yes, you can do that. In fact that is the most common case. The
>> documents which should help you do so are here:
>>
>> https://cwiki.apache.org/confluence/display/FLUME/Flume+1.x+Documentation
>>
>> Brock
>>
>> On Mon, Jul 23, 2012 at 7:26 PM, mardan Khan <mardan8310@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I am just doing testing. I am generating the sequence and want to upload
>> > into hdfs. My configuration file as:
>> >
>> > agent2.channels = c1
>> > agent2.sources = r1
>> > agent2.sinks = k1
>> >
>> > agent2.channels.c1.type = MEMORY
>> >
>> > agent2.sources.r1.channels = c1
>> > agent2.sources.r1.type = SEQ
>> >
>> > agent2.sinks.k1.channel = c1
>> > agent2.sinks.k1.type = LOGGER
>> >
>> >
>> > Is it possible to upload into hdfs, if possible then how I can make the
>> > changes in configuration file.
>> >
>> >
>> > Many thanks
>> >
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>>
>
>

Mime
View raw message