flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Lord <jeffrey.l...@gmail.com>
Subject Re: issue with hdfs sink config
Date Thu, 13 Sep 2012 16:28:08 GMT
Raymond,

I don't think you need the port 50070 on your hdfs path.
Try setting this to the fqdn of your hdfs namenode, leave off the port
50070 (this is just for the webui)

-Jeff

On Thu, Sep 13, 2012 at 9:04 AM, Raymond Ng <raymondair@gmail.com> wrote:

> Hi
>
> I'm baffled as to what the problem is with my hdfs config below
>
> agent.sinks.probeHdfsSink2.type = hdfs
> agent.sinks.probeHdfsSink2.hdfs.path =
> hdfs://master:50070/user/test/data/flume/%{client}
> agent.sinks.probeHdfsSink2.hdfs.filePrefix = %{site}_sessionData_%Y%m%d
> agent.sinks.probeHdfsSink2.channel = probeFileChannel2
> agent.sinks.probeHdfsSink2.hdfs.rollInterval = 120
> agent.sinks.probeHdfsSink2.hdfs.rollSize = 209715200
> agent.sinks.probeHdfsSink2.hdfs.rollCount = 0
> agent.sinks.probeHdfsSink2.hdfs.batchSize = 100000
> agent.sinks.probeHdfsSink2.hdfs.writeFormat = Text
> agent.sinks.probeHdfsSink2.hdfs.fileType = DataStream
> the flume log below does not provide enough info to help identify the
> problem, FlumeConfiguration.AgentConfiguration holds a list of
> <FlumeConfigurationError> but none of them get displayed in the log.
>
> If I replace hdfs sink with a file sink then everything works, so the
> problem is definitely in my hdfs configs but I can't see where?  any advice
> is appreciated
>
> 2012-09-13 16:49:43,145  INFO [main] LifecycleSupervisor.java - Starting
> lifecycle supervisor 1
> 2012-09-13 16:49:43,151  INFO [main] FlumeNode.java - Flume node starting
> - agent
> 2012-09-13 16:49:43,156  INFO [lifecycleSupervisor-1-0]
> DefaultLogicalNodeManager.java - Node manager starting
> 2012-09-13 16:49:43,156  INFO [lifecycleSupervisor-1-2]
> AbstractFileConfigurationProvider.java - Configuration provider starting
> 2012-09-13 16:49:43,157  INFO [lifecycleSupervisor-1-0]
> LifecycleSupervisor.java - Starting lifecycle supervisor 11
> 2012-09-13 16:49:43,158 DEBUG [lifecycleSupervisor-1-0]
> DefaultLogicalNodeManager.java - Node manager started
> 2012-09-13 16:49:43,160 DEBUG [lifecycleSupervisor-1-2]
> AbstractFileConfigurationProvider.java - Configuration provider started
> 2012-09-13 16:49:43,160 DEBUG [conf-file-poller-0]
> AbstractFileConfigurationProvider.java - Checking
> file:/usr/local/lib/flume-ng/conf/acme-agent.conf for changes
> 2012-09-13 16:49:43,160  INFO [conf-file-poller-0]
> AbstractFileConfigurationProvider.java - Reloading configuration
> file:/usr/local/lib/flume-ng/conf/acme-agent.conf
> 2012-09-13 16:49:43,170  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Added sinks: probeHdfsSink2 Agent: agent
> 2012-09-13 16:49:43,171  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeHdfsSink2
> 2012-09-13 16:49:43,171 DEBUG [conf-file-poller-0] FlumeConfiguration.java
> - Created context for probeHdfsSink2: hdfs.rollSize
> 2012-09-13 16:49:43,172  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeHdfsSink2
> 2012-09-13 16:49:43,172  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeHdfsSink2
> 2012-09-13 16:49:43,172  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeFileSink
> 2012-09-13 16:49:43,172 DEBUG [conf-file-poller-0] FlumeConfiguration.java
> - Created context for probeFileSink: sink.rollInterval
> 2012-09-13 16:49:43,176  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeHdfsSink2
> 2012-09-13 16:49:43,176  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeHdfsSink2
> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeHdfsSink2
> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeFileSink
> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeFileSink
> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeHdfsSink2
> 2012-09-13 16:49:43,178  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeFileSink
> 2012-09-13 16:49:43,178  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeHdfsSink2
> 2012-09-13 16:49:43,178  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeHdfsSink2
> 2012-09-13 16:49:43,178  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeHdfsSink2
> 2012-09-13 16:49:43,179  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Processing:probeFileSink
> 2012-09-13 16:49:43,179 DEBUG [conf-file-poller-0] FlumeConfiguration.java
> - Starting validation of configuration for agent: agent,
> initial-configuration: AgentConfiguration[agent]
> SOURCES: {probeAvroSrc2={ parameters:{port=5556,
> channels=probeFileChannel2, type=avro, bind=192.168.200.170} }}
> CHANNELS: {probeFileChannel2={
> parameters:{checkpointDir=/home/test/flume-ng/filechannel2/checkpoint,
> dataDirs=/home/test/flume-ng/filechannel2/data, write-timeout=3,
> transactionCapacity=200000, checkpointInterval=10, capacity=1000000,
> keep-alive=3, maxFileSize=1073741824, type=FILE} }}
> SINKS: {probeHdfsSink2={ parameters:{hdfs.fileType=DataStream,
> hdfs.path=hdfs://master:50070/user/test/data/flume/%{client},
> hdfs.batchSize=100000, hdfs.rollInterval=120, hdfs.rollSize=209715200,
> hdfs.filePrefix=%{site}_sessionData_%Y%m%d, hdfs.writeFormat=Text,
> type=hdfs, channel=probeFileChannel2, hdfs.rollCount=0} }, probeFileSink={
> parameters:{sink.serializer=TEXT, sink.rollInterval=0, type=FILE_ROLL,
> channel=probeFileChannel2, sink.directory=/home/test/flume/filesink} }}
> 2012-09-13 16:49:43,188 DEBUG [conf-file-poller-0] FlumeConfiguration.java
> - Created channel probeFileChannel2
> 2012-09-13 16:49:43,214 DEBUG [conf-file-poller-0] FlumeConfiguration.java
> - Creating sink: probeHdfsSink2 using HDFS
> 2012-09-13 16:49:43,218 DEBUG [conf-file-poller-0] FlumeConfiguration.java
> - Post validation configuration for agent
> AgentConfiguration created without Configuration stubs for which only
> basic syntactical validation was performed[agent]
> SOURCES: {probeAvroSrc2={ parameters:{port=5556,
> channels=probeFileChannel2, type=avro, bind=192.168.200.170} }}
> CHANNELS: {probeFileChannel2={
> parameters:{checkpointDir=/home/test/flume-ng/filechannel2/checkpoint,
> dataDirs=/home/test/flume-ng/filechannel2/data, write-timeout=3,
> transactionCapacity=200000, checkpointInterval=10, capacity=1000000,
> keep-alive=3, maxFileSize=1073741824, type=FILE} }}
> SINKS: {probeHdfsSink2={ parameters:{hdfs.fileType=DataStream,
> hdfs.path=hdfs://master:50070/user/test/data/flume/%{client},
> hdfs.batchSize=100000, hdfs.rollInterval=120, hdfs.rollSize=209715200,
> hdfs.filePrefix=%{site}_sessionData_%Y%m%d, hdfs.writeFormat=Text,
> type=hdfs, channel=probeFileChannel2, hdfs.rollCount=0} }}
> 2012-09-13 16:49:43,218 DEBUG [conf-file-poller-0] FlumeConfiguration.java
> - Channels:probeFileChannel2
> 2012-09-13 16:49:43,219 DEBUG [conf-file-poller-0] FlumeConfiguration.java
> - Sinks probeHdfsSink2
> 2012-09-13 16:49:43,219 DEBUG [conf-file-poller-0] FlumeConfiguration.java
> - Sources probeAvroSrc2
> 2012-09-13 16:49:43,219  INFO [conf-file-poller-0] FlumeConfiguration.java
> - Post-validation flume configuration contains configuration  for agents:
> [agent]
> 2012-09-13 16:49:43,219  INFO [conf-file-poller-0]
> PropertiesFileConfigurationProvider.java - Creating channels
> 2012-09-13 16:49:43,220 DEBUG [conf-file-poller-0]
> DefaultChannelFactory.java - Creating instance of channel probeFileChannel2
> type FILE
> 2012-09-13 16:49:43,236  INFO [conf-file-poller-0]
> MonitoredCounterGroup.java - Monitoried counter group for type: CHANNEL,
> name: probeFileChannel2, registered successfully.
> 2012-09-13 16:49:43,236  INFO [conf-file-poller-0]
> PropertiesFileConfigurationProvider.java - created channel probeFileChannel2
> 2012-09-13 16:49:43,236 DEBUG [conf-file-poller-0]
> DefaultSourceFactory.java - Creating instance of source probeAvroSrc2, type
> avro
> 2012-09-13 16:49:43,256  INFO [conf-file-poller-0]
> MonitoredCounterGroup.java - Monitoried counter group for type: SOURCE,
> name: probeAvroSrc2, registered successfully.
> 2012-09-13 16:49:43,277  INFO [conf-file-poller-0] DefaultSinkFactory.java
> - Creating instance of sink: probeHdfsSink2, type: hdfs
>
>
> --
> Rgds
> Ray
>

Mime
View raw message