flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Handa <amithand...@gmail.com>
Subject flume ng error while going for hdfs sink
Date Thu, 05 Jul 2012 07:28:38 GMT
HI All,

While trying to run Flume ng using HDFS SInk, and using avro Client.. i am
getting IOException. Kindly help in resolving this issue

Exception log is as follows:
2012-07-05 12:01:32,789 (conf-file-poller-0) [INFO -
org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)]
Creating instance of sink HDFS typehdfs
2012-07-05 12:01:32,816 (conf-file-poller-0) [DEBUG -
org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)]
java.io.IOException: config()
    at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)
    at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:214)
    at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187)
    at
org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239)
    at
org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:516)
    at
org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:239)
    at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
    at
org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSinks(PropertiesFileConfigurationProvider.java:373)
    at
org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:223)
    at
org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
    at
org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
    at
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)


Flume.conf file, contents are :
# Define a memory channel called ch1 on agent1
agent1.channels.ch1.type = memory

# Define an Avro source called avro-source1 on agent1 and tell it
# to bind to 0.0.0.0:41414. Connect it to channel ch1.
agent1.sources.avro-source1.channels = ch1
agent1.sources.avro-source1.type = avro
agent1.sources.avro-source1.bind = 0.0.0.0
agent1.sources.avro-source1.port = 41414

# Define a hdfs sink that simply logs all events it receives
# and connect it to the other end of the same channel.
agent1.sinks.HDFS.channel = ch1
agent1.sinks.HDFS.type = hdfs
agent1.sinks.HDFS.hdfs.path = hdfs://localhost:54310/ (in my setup hdfs is
using port 54310
agent1.sinks.HDFS.hdfs.file.Type = DataStream


# Finally, now that we've defined all of our components, tell
# agent1 which ones we want to activate.
agent1.channels = ch1
agent1.sources = avro-source1
#agent1.sinks = fileroll_sink1
agent1.sinks = HDFS

Flume NG Command
bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1

Mime
View raw message