flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raymond Ng <raymond...@gmail.com>
Subject Re: issue with hdfs sink config
Date Sat, 15 Sep 2012 06:28:58 GMT
you are right jeff. i was using a old version of hadoop jar. problem solved
now. thanks
On Sep 14, 2012 10:17 PM, "Jeff Lord" <jeffrey.lord@gmail.com> wrote:

> Let's backup a step.
> Do you have the hadoop libs installed?
> In this version you need to have the hadoop libs installed in order for
> flume to write to hdfs sink.
>
>
> On Fri, Sep 14, 2012 at 12:35 AM, Raymond Ng <raymondair@gmail.com> wrote:
>
>> Hi Jeff
>>
>> removing the port number didn't work
>>
>> what I should have mentioned was that I'm using flume 1.2, the same
>> config I mentioned below works for flume 1.1, so I'm not sure what has
>> changed in 1.2 for the configs and I need to use 1.2 because of the File
>> channel
>>  I did see HDFSSinkConfiguration mentioned in SinkConfiguration.java but
>> I can't find this class in the flume 1.2 jars, and ClassNotFoundException
>> is being wrapped up in ConfigurationException which doesn't get reported in
>> the log
>>
>>
>> Ray
>>
>>
>> On Thu, Sep 13, 2012 at 5:28 PM, Jeff Lord <jeffrey.lord@gmail.com>wrote:
>>
>>> Raymond,
>>>
>>> I don't think you need the port 50070 on your hdfs path.
>>> Try setting this to the fqdn of your hdfs namenode, leave off the port
>>> 50070 (this is just for the webui)
>>>
>>> -Jeff
>>>
>>> On Thu, Sep 13, 2012 at 9:04 AM, Raymond Ng <raymondair@gmail.com>wrote:
>>>
>>>> Hi
>>>>
>>>> I'm baffled as to what the problem is with my hdfs config below
>>>>
>>>> agent.sinks.probeHdfsSink2.type = hdfs
>>>> agent.sinks.probeHdfsSink2.hdfs.path =
>>>> hdfs://master:50070/user/test/data/flume/%{client}
>>>> agent.sinks.probeHdfsSink2.hdfs.filePrefix = %{site}_sessionData_%Y%m%d
>>>> agent.sinks.probeHdfsSink2.channel = probeFileChannel2
>>>> agent.sinks.probeHdfsSink2.hdfs.rollInterval = 120
>>>> agent.sinks.probeHdfsSink2.hdfs.rollSize = 209715200
>>>> agent.sinks.probeHdfsSink2.hdfs.rollCount = 0
>>>> agent.sinks.probeHdfsSink2.hdfs.batchSize = 100000
>>>> agent.sinks.probeHdfsSink2.hdfs.writeFormat = Text
>>>> agent.sinks.probeHdfsSink2.hdfs.fileType = DataStream
>>>> the flume log below does not provide enough info to help identify the
>>>> problem, FlumeConfiguration.AgentConfiguration holds a list of
>>>> <FlumeConfigurationError> but none of them get displayed in the log.
>>>>
>>>> If I replace hdfs sink with a file sink then everything works, so the
>>>> problem is definitely in my hdfs configs but I can't see where?  any advice
>>>> is appreciated
>>>>
>>>> 2012-09-13 16:49:43,145  INFO [main] LifecycleSupervisor.java -
>>>> Starting lifecycle supervisor 1
>>>> 2012-09-13 16:49:43,151  INFO [main] FlumeNode.java - Flume node
>>>> starting - agent
>>>> 2012-09-13 16:49:43,156  INFO [lifecycleSupervisor-1-0]
>>>> DefaultLogicalNodeManager.java - Node manager starting
>>>> 2012-09-13 16:49:43,156  INFO [lifecycleSupervisor-1-2]
>>>> AbstractFileConfigurationProvider.java - Configuration provider starting
>>>> 2012-09-13 16:49:43,157  INFO [lifecycleSupervisor-1-0]
>>>> LifecycleSupervisor.java - Starting lifecycle supervisor 11
>>>> 2012-09-13 16:49:43,158 DEBUG [lifecycleSupervisor-1-0]
>>>> DefaultLogicalNodeManager.java - Node manager started
>>>> 2012-09-13 16:49:43,160 DEBUG [lifecycleSupervisor-1-2]
>>>> AbstractFileConfigurationProvider.java - Configuration provider started
>>>> 2012-09-13 16:49:43,160 DEBUG [conf-file-poller-0]
>>>> AbstractFileConfigurationProvider.java - Checking
>>>> file:/usr/local/lib/flume-ng/conf/acme-agent.conf for changes
>>>> 2012-09-13 16:49:43,160  INFO [conf-file-poller-0]
>>>> AbstractFileConfigurationProvider.java - Reloading configuration
>>>> file:/usr/local/lib/flume-ng/conf/acme-agent.conf
>>>> 2012-09-13 16:49:43,170  INFO [conf-file-poller-0]
>>>> FlumeConfiguration.java - Added sinks: probeHdfsSink2 Agent: agent
>>>> 2012-09-13 16:49:43,171  INFO [conf-file-poller-0]
>>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>>> 2012-09-13 16:49:43,171 DEBUG [conf-file-poller-0]
>>>> FlumeConfiguration.java - Created context for probeHdfsSink2: hdfs.rollSize
>>>> 2012-09-13 16:49:43,172  INFO [conf-file-poller-0]
>>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>>> 2012-09-13 16:49:43,172  INFO [conf-file-poller-0]
>>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>>> 2012-09-13 16:49:43,172  INFO [conf-file-poller-0]
>>>> FlumeConfiguration.java - Processing:probeFileSink
>>>> 2012-09-13 16:49:43,172 DEBUG [conf-file-poller-0]
>>>> FlumeConfiguration.java - Created context for probeFileSink:
>>>> sink.rollInterval
>>>> 2012-09-13 16:49:43,176  INFO [conf-file-poller-0]
>>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>>> 2012-09-13 16:49:43,176  INFO [conf-file-poller-0]
>>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>>> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0]
>>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>>> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0]
>>>> FlumeConfiguration.java - Processing:probeFileSink
>>>> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0]
>>>> FlumeConfiguration.java - Processing:probeFileSink
>>>> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0]
>>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>>> 2012-09-13 16:49:43,178  INFO [conf-file-poller-0]
>>>> FlumeConfiguration.java - Processing:probeFileSink
>>>> 2012-09-13 16:49:43,178  INFO [conf-file-poller-0]
>>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>>> 2012-09-13 16:49:43,178  INFO [conf-file-poller-0]
>>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>>> 2012-09-13 16:49:43,178  INFO [conf-file-poller-0]
>>>> FlumeConfiguration.java - Processing:probeHdfsSink2
>>>> 2012-09-13 16:49:43,179  INFO [conf-file-poller-0]
>>>> FlumeConfiguration.java - Processing:probeFileSink
>>>> 2012-09-13 16:49:43,179 DEBUG [conf-file-poller-0]
>>>> FlumeConfiguration.java - Starting validation of configuration for agent:
>>>> agent, initial-configuration: AgentConfiguration[agent]
>>>> SOURCES: {probeAvroSrc2={ parameters:{port=5556,
>>>> channels=probeFileChannel2, type=avro, bind=192.168.200.170} }}
>>>> CHANNELS: {probeFileChannel2={
>>>> parameters:{checkpointDir=/home/test/flume-ng/filechannel2/checkpoint,
>>>> dataDirs=/home/test/flume-ng/filechannel2/data, write-timeout=3,
>>>> transactionCapacity=200000, checkpointInterval=10, capacity=1000000,
>>>> keep-alive=3, maxFileSize=1073741824, type=FILE} }}
>>>> SINKS: {probeHdfsSink2={ parameters:{hdfs.fileType=DataStream,
>>>> hdfs.path=hdfs://master:50070/user/test/data/flume/%{client},
>>>> hdfs.batchSize=100000, hdfs.rollInterval=120, hdfs.rollSize=209715200,
>>>> hdfs.filePrefix=%{site}_sessionData_%Y%m%d, hdfs.writeFormat=Text,
>>>> type=hdfs, channel=probeFileChannel2, hdfs.rollCount=0} }, probeFileSink={
>>>> parameters:{sink.serializer=TEXT, sink.rollInterval=0, type=FILE_ROLL,
>>>> channel=probeFileChannel2, sink.directory=/home/test/flume/filesink} }}
>>>> 2012-09-13 16:49:43,188 DEBUG [conf-file-poller-0]
>>>> FlumeConfiguration.java - Created channel probeFileChannel2
>>>> 2012-09-13 16:49:43,214 DEBUG [conf-file-poller-0]
>>>> FlumeConfiguration.java - Creating sink: probeHdfsSink2 using HDFS
>>>> 2012-09-13 16:49:43,218 DEBUG [conf-file-poller-0]
>>>> FlumeConfiguration.java - Post validation configuration for agent
>>>> AgentConfiguration created without Configuration stubs for which only
>>>> basic syntactical validation was performed[agent]
>>>> SOURCES: {probeAvroSrc2={ parameters:{port=5556,
>>>> channels=probeFileChannel2, type=avro, bind=192.168.200.170} }}
>>>> CHANNELS: {probeFileChannel2={
>>>> parameters:{checkpointDir=/home/test/flume-ng/filechannel2/checkpoint,
>>>> dataDirs=/home/test/flume-ng/filechannel2/data, write-timeout=3,
>>>> transactionCapacity=200000, checkpointInterval=10, capacity=1000000,
>>>> keep-alive=3, maxFileSize=1073741824, type=FILE} }}
>>>> SINKS: {probeHdfsSink2={ parameters:{hdfs.fileType=DataStream,
>>>> hdfs.path=hdfs://master:50070/user/test/data/flume/%{client},
>>>> hdfs.batchSize=100000, hdfs.rollInterval=120, hdfs.rollSize=209715200,
>>>> hdfs.filePrefix=%{site}_sessionData_%Y%m%d, hdfs.writeFormat=Text,
>>>> type=hdfs, channel=probeFileChannel2, hdfs.rollCount=0} }}
>>>> 2012-09-13 16:49:43,218 DEBUG [conf-file-poller-0]
>>>> FlumeConfiguration.java - Channels:probeFileChannel2
>>>> 2012-09-13 16:49:43,219 DEBUG [conf-file-poller-0]
>>>> FlumeConfiguration.java - Sinks probeHdfsSink2
>>>> 2012-09-13 16:49:43,219 DEBUG [conf-file-poller-0]
>>>> FlumeConfiguration.java - Sources probeAvroSrc2
>>>> 2012-09-13 16:49:43,219  INFO [conf-file-poller-0]
>>>> FlumeConfiguration.java - Post-validation flume configuration contains
>>>> configuration  for agents: [agent]
>>>> 2012-09-13 16:49:43,219  INFO [conf-file-poller-0]
>>>> PropertiesFileConfigurationProvider.java - Creating channels
>>>> 2012-09-13 16:49:43,220 DEBUG [conf-file-poller-0]
>>>> DefaultChannelFactory.java - Creating instance of channel probeFileChannel2
>>>> type FILE
>>>> 2012-09-13 16:49:43,236  INFO [conf-file-poller-0]
>>>> MonitoredCounterGroup.java - Monitoried counter group for type: CHANNEL,
>>>> name: probeFileChannel2, registered successfully.
>>>> 2012-09-13 16:49:43,236  INFO [conf-file-poller-0]
>>>> PropertiesFileConfigurationProvider.java - created channel probeFileChannel2
>>>> 2012-09-13 16:49:43,236 DEBUG [conf-file-poller-0]
>>>> DefaultSourceFactory.java - Creating instance of source probeAvroSrc2, type
>>>> avro
>>>> 2012-09-13 16:49:43,256  INFO [conf-file-poller-0]
>>>> MonitoredCounterGroup.java - Monitoried counter group for type: SOURCE,
>>>> name: probeAvroSrc2, registered successfully.
>>>> 2012-09-13 16:49:43,277  INFO [conf-file-poller-0]
>>>> DefaultSinkFactory.java - Creating instance of sink: probeHdfsSink2, type:
>>>> hdfs
>>>>
>>>>
>>>> --
>>>> Rgds
>>>> Ray
>>>>
>>>
>>>
>>
>>
>> --
>> Rgds
>> Ray
>>
>
>

Mime
View raw message