flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Chavez <pcha...@verticalsearchworks.com>
Subject spoolDir source problem
Date Thu, 11 Apr 2013 22:15:25 GMT
Hello,

I've run into a problem with the spoolDir source, on Windows, and am not sure how to proceed.

The agent starts fine and the source is created without issue and is apparently ready. After
agent start a .flumespool directory is created in the path the source is watching. This directory
remains empty as long as the agent is idle.

However, as soon as I drop a file into the spool directory (parent to the .flumespool directory),
I get a series of errors in the flume log and a file named '.flumespool-main.meta<string
of numbers>.tmp' is created in that .flumespool directory at the rate of one per second.
The file in the spool directory is never touched as far as I can tell and the /metrics web
page shows no movement on the channel or sink. A possibly related note is that the sources
don't show in the metrics page, even though the logs say the source(s) are started.

All I have done so far is set the directory security to be 'Everyone/Full Control', basically
the windows version of 'chmod 777'

Any help is appreciated!

thanks,
Paul

Here's what the log shows.
11 Apr 2013 15:11:48,092 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:184)
 - Starting Source spool_WebLogs
11 Apr 2013 15:11:48,092 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:184)
 - Starting Source http_Default
11 Apr 2013 15:11:48,092 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.source.SpoolDirectorySource.start:66)
 - SpoolDirectorySource source starting with directory: c:\flume_data\spool\web
11 Apr 2013 15:11:48,124 INFO  [conf-file-poller-0] (org.mortbay.log.Slf4jLog.info:67)  -
Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
11 Apr 2013 15:11:48,139 INFO  [conf-file-poller-0] (org.mortbay.log.Slf4jLog.info:67)  -
jetty-6.1.26
11 Apr 2013 15:11:48,155 INFO  [conf-file-poller-0] (org.mortbay.log.Slf4jLog.info:67)  -
Started SocketConnector@0.0.0.0:41414<mailto:SocketConnector@0.0.0.0:41414>
11 Apr 2013 15:11:48,202 INFO  [lifecycleSupervisor-1-2] (org.mortbay.log.Slf4jLog.info:67)
 - jetty-6.1.26
11 Apr 2013 15:11:48,217 INFO  [lifecycleSupervisor-1-2] (org.mortbay.log.Slf4jLog.info:67)
 - Started SocketConnector@0.0.0.0:6240<mailto:SocketConnector@0.0.0.0:6240>
11 Apr 2013 15:11:48,389 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.sink.AvroSink.start:253)
 - Avro sink avro_Default started.
11 Apr 2013 15:11:48,404 ERROR [pool-6-thread-1] (org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile:442)
 - Exception opening file: c:\flume_data\spool\web\u_ex130411.log-201304111500.log
java.io.IOException: Unable to delete existing meta file c:\flume_data\spool\web\.flumespool\.flumespool-main.meta
 at org.apache.flume.serialization.DurablePositionTracker.getInstance(DurablePositionTracker.java:96)
 at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile(ReliableSpoolingFileEventReader.java:423)
 at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:212)
 at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:154)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
 at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
11 Apr 2013 15:11:48,919 ERROR [pool-6-thread-1] (org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile:442)
 - Exception opening file: c:\flume_data\spool\web\u_ex130411.log-201304111500.log
java.io.IOException: Unable to delete existing meta file c:\flume_data\spool\web\.flumespool\.flumespool-main.meta
 at org.apache.flume.serialization.DurablePositionTracker.getInstance(DurablePositionTracker.java:96)
 at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile(ReliableSpoolingFileEventReader.java:417)
 at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:212)
 at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:154)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
 at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)

Mime
View raw message