flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Lord <jeffrey.l...@gmail.com>
Subject Re: issue with hdfs sink config
Date Fri, 14 Sep 2012 21:16:04 GMT
Let's backup a step.
Do you have the hadoop libs installed?
In this version you need to have the hadoop libs installed in order for
flume to write to hdfs sink.


On Fri, Sep 14, 2012 at 12:35 AM, Raymond Ng <raymondair@gmail.com> wrote:

> Hi Jeff
>
> removing the port number didn't work
>
> what I should have mentioned was that I'm using flume 1.2, the same config
> I mentioned below works for flume 1.1, so I'm not sure what has changed in
> 1.2 for the configs and I need to use 1.2 because of the File channel
>  I did see HDFSSinkConfiguration mentioned in SinkConfiguration.java but
> I can't find this class in the flume 1.2 jars, and ClassNotFoundException
> is being wrapped up in ConfigurationException which doesn't get reported in
> the log
>
>
> Ray
>
>
> On Thu, Sep 13, 2012 at 5:28 PM, Jeff Lord <jeffrey.lord@gmail.com> wrote:
>
>> Raymond,
>>
>> I don't think you need the port 50070 on your hdfs path.
>> Try setting this to the fqdn of your hdfs namenode, leave off the port
>> 50070 (this is just for the webui)
>>
>> -Jeff
>>
>> On Thu, Sep 13, 2012 at 9:04 AM, Raymond Ng <raymondair@gmail.com> wrote:
>>
>>> Hi
>>>
>>> I'm baffled as to what the problem is with my hdfs config below
>>>
>>> agent.sinks.probeHdfsSink2.type = hdfs
>>> agent.sinks.probeHdfsSink2.hdfs.path =
>>> hdfs://master:50070/user/test/data/flume/%{client}
>>> agent.sinks.probeHdfsSink2.hdfs.filePrefix = %{site}_sessionData_%Y%m%d
>>> agent.sinks.probeHdfsSink2.channel = probeFileChannel2
>>> agent.sinks.probeHdfsSink2.hdfs.rollInterval = 120
>>> agent.sinks.probeHdfsSink2.hdfs.rollSize = 209715200
>>> agent.sinks.probeHdfsSink2.hdfs.rollCount = 0
>>> agent.sinks.probeHdfsSink2.hdfs.batchSize = 100000
>>> agent.sinks.probeHdfsSink2.hdfs.writeFormat = Text
>>> agent.sinks.probeHdfsSink2.hdfs.fileType = DataStream
>>> the flume log below does not provide enough info to help identify the
>>> problem, FlumeConfiguration.AgentConfiguration holds a list of
>>> <FlumeConfigurationError> but none of them get displayed in the log.
>>>
>>> If I replace hdfs sink with a file sink then everything works, so the
>>> problem is definitely in my hdfs configs but I can't see where?  any advice
>>> is appreciated
>>>
>>> 2012-09-13 16:49:43,145  INFO [main] LifecycleSupervisor.java - Starting
>>> lifecycle supervisor 1
>>> 2012-09-13 16:49:43,151  INFO [main] FlumeNode.java - Flume node
>>> starting - agent
>>> 2012-09-13 16:49:43,156  INFO [lifecycleSupervisor-1-0]
>>> DefaultLogicalNodeManager.java - Node manager starting
>>> 2012-09-13 16:49:43,156  INFO [lifecycleSupervisor-1-2]
>>> AbstractFileConfigurationProvider.java - Configuration provider starting
>>> 2012-09-13 16:49:43,157  INFO [lifecycleSupervisor-1-0]
>>> LifecycleSupervisor.java - Starting lifecycle supervisor 11
>>> 2012-09-13 16:49:43,158 DEBUG [lifecycleSupervisor-1-0]
>>> DefaultLogicalNodeManager.java - Node manager started
>>> 2012-09-13 16:49:43,160 DEBUG [lifecycleSupervisor-1-2]
>>> AbstractFileConfigurationProvider.java - Configuration provider started
>>> 2012-09-13 16:49:43,160 DEBUG [conf-file-poller-0]
>>> AbstractFileConfigurationProvider.java - Checking
>>> file:/usr/local/lib/flume-ng/conf/acme-agent.conf for changes
>>> 2012-09-13 16:49:43,160  INFO [conf-file-poller-0]
>>> AbstractFileConfigurationProvider.java - Reloading configuration
>>> file:/usr/local/lib/flume-ng/conf/acme-agent.conf
>>> 2012-09-13 16:49:43,170  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Added sinks: probeHdfsSink2 Agent: agent
>>> 2012-09-13 16:49:43,171  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>> 2012-09-13 16:49:43,171 DEBUG [conf-file-poller-0]
>>> FlumeConfiguration.java - Created context for probeHdfsSink2: hdfs.rollSize
>>> 2012-09-13 16:49:43,172  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>> 2012-09-13 16:49:43,172  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>> 2012-09-13 16:49:43,172  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Processing:probeFileSink
>>> 2012-09-13 16:49:43,172 DEBUG [conf-file-poller-0]
>>> FlumeConfiguration.java - Created context for probeFileSink:
>>> sink.rollInterval
>>> 2012-09-13 16:49:43,176  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>> 2012-09-13 16:49:43,176  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Processing:probeFileSink
>>> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Processing:probeFileSink
>>> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>> 2012-09-13 16:49:43,178  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Processing:probeFileSink
>>> 2012-09-13 16:49:43,178  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>> 2012-09-13 16:49:43,178  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>> 2012-09-13 16:49:43,178  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>> 2012-09-13 16:49:43,179  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Processing:probeFileSink
>>> 2012-09-13 16:49:43,179 DEBUG [conf-file-poller-0]
>>> FlumeConfiguration.java - Starting validation of configuration for agent:
>>> agent, initial-configuration: AgentConfiguration[agent]
>>> SOURCES: {probeAvroSrc2={ parameters:{port=5556,
>>> channels=probeFileChannel2, type=avro, bind=192.168.200.170} }}
>>> CHANNELS: {probeFileChannel2={
>>> parameters:{checkpointDir=/home/test/flume-ng/filechannel2/checkpoint,
>>> dataDirs=/home/test/flume-ng/filechannel2/data, write-timeout=3,
>>> transactionCapacity=200000, checkpointInterval=10, capacity=1000000,
>>> keep-alive=3, maxFileSize=1073741824, type=FILE} }}
>>> SINKS: {probeHdfsSink2={ parameters:{hdfs.fileType=DataStream,
>>> hdfs.path=hdfs://master:50070/user/test/data/flume/%{client},
>>> hdfs.batchSize=100000, hdfs.rollInterval=120, hdfs.rollSize=209715200,
>>> hdfs.filePrefix=%{site}_sessionData_%Y%m%d, hdfs.writeFormat=Text,
>>> type=hdfs, channel=probeFileChannel2, hdfs.rollCount=0} }, probeFileSink={
>>> parameters:{sink.serializer=TEXT, sink.rollInterval=0, type=FILE_ROLL,
>>> channel=probeFileChannel2, sink.directory=/home/test/flume/filesink} }}
>>> 2012-09-13 16:49:43,188 DEBUG [conf-file-poller-0]
>>> FlumeConfiguration.java - Created channel probeFileChannel2
>>> 2012-09-13 16:49:43,214 DEBUG [conf-file-poller-0]
>>> FlumeConfiguration.java - Creating sink: probeHdfsSink2 using HDFS
>>> 2012-09-13 16:49:43,218 DEBUG [conf-file-poller-0]
>>> FlumeConfiguration.java - Post validation configuration for agent
>>> AgentConfiguration created without Configuration stubs for which only
>>> basic syntactical validation was performed[agent]
>>> SOURCES: {probeAvroSrc2={ parameters:{port=5556,
>>> channels=probeFileChannel2, type=avro, bind=192.168.200.170} }}
>>> CHANNELS: {probeFileChannel2={
>>> parameters:{checkpointDir=/home/test/flume-ng/filechannel2/checkpoint,
>>> dataDirs=/home/test/flume-ng/filechannel2/data, write-timeout=3,
>>> transactionCapacity=200000, checkpointInterval=10, capacity=1000000,
>>> keep-alive=3, maxFileSize=1073741824, type=FILE} }}
>>> SINKS: {probeHdfsSink2={ parameters:{hdfs.fileType=DataStream,
>>> hdfs.path=hdfs://master:50070/user/test/data/flume/%{client},
>>> hdfs.batchSize=100000, hdfs.rollInterval=120, hdfs.rollSize=209715200,
>>> hdfs.filePrefix=%{site}_sessionData_%Y%m%d, hdfs.writeFormat=Text,
>>> type=hdfs, channel=probeFileChannel2, hdfs.rollCount=0} }}
>>> 2012-09-13 16:49:43,218 DEBUG [conf-file-poller-0]
>>> FlumeConfiguration.java - Channels:probeFileChannel2
>>> 2012-09-13 16:49:43,219 DEBUG [conf-file-poller-0]
>>> FlumeConfiguration.java - Sinks probeHdfsSink2
>>> 2012-09-13 16:49:43,219 DEBUG [conf-file-poller-0]
>>> FlumeConfiguration.java - Sources probeAvroSrc2
>>> 2012-09-13 16:49:43,219  INFO [conf-file-poller-0]
>>> FlumeConfiguration.java - Post-validation flume configuration contains
>>> configuration  for agents: [agent]
>>> 2012-09-13 16:49:43,219  INFO [conf-file-poller-0]
>>> PropertiesFileConfigurationProvider.java - Creating channels
>>> 2012-09-13 16:49:43,220 DEBUG [conf-file-poller-0]
>>> DefaultChannelFactory.java - Creating instance of channel probeFileChannel2
>>> type FILE
>>> 2012-09-13 16:49:43,236  INFO [conf-file-poller-0]
>>> MonitoredCounterGroup.java - Monitoried counter group for type: CHANNEL,
>>> name: probeFileChannel2, registered successfully.
>>> 2012-09-13 16:49:43,236  INFO [conf-file-poller-0]
>>> PropertiesFileConfigurationProvider.java - created channel probeFileChannel2
>>> 2012-09-13 16:49:43,236 DEBUG [conf-file-poller-0]
>>> DefaultSourceFactory.java - Creating instance of source probeAvroSrc2, type
>>> avro
>>> 2012-09-13 16:49:43,256  INFO [conf-file-poller-0]
>>> MonitoredCounterGroup.java - Monitoried counter group for type: SOURCE,
>>> name: probeAvroSrc2, registered successfully.
>>> 2012-09-13 16:49:43,277  INFO [conf-file-poller-0]
>>> DefaultSinkFactory.java - Creating instance of sink: probeHdfsSink2, type:
>>> hdfs
>>>
>>>
>>> --
>>> Rgds
>>> Ray
>>>
>>
>>
>
>
> --
> Rgds
> Ray
>

Mime
View raw message