flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Chavez <pcha...@verticalsearchworks.com>
Subject RE: spoolDir source problem
Date Fri, 12 Apr 2013 18:41:08 GMT
Anyone have any ideas on this? I can't even find the class throwing the exception to try and
see what it is doing. I would really like to use this on Windows, but would like to know at
least if there's some compatibility issue so I can move on.

thanks,
Paul


________________________________
From: Paul Chavez [mailto:pchavez@verticalsearchworks.com]
Sent: Thursday, April 11, 2013 3:15 PM
To: user@flume.apache.org
Subject: spoolDir source problem

Hello,

I've run into a problem with the spoolDir source, on Windows, and am not sure how to proceed.

The agent starts fine and the source is created without issue and is apparently ready. After
agent start a .flumespool directory is created in the path the source is watching. This directory
remains empty as long as the agent is idle.

However, as soon as I drop a file into the spool directory (parent to the .flumespool directory),
I get a series of errors in the flume log and a file named '.flumespool-main.meta<string
of numbers>.tmp' is created in that .flumespool directory at the rate of one per second.
The file in the spool directory is never touched as far as I can tell and the /metrics web
page shows no movement on the channel or sink. A possibly related note is that the sources
don't show in the metrics page, even though the logs say the source(s) are started.

All I have done so far is set the directory security to be 'Everyone/Full Control', basically
the windows version of 'chmod 777'

Any help is appreciated!

thanks,
Paul

Here's what the log shows.
11 Apr 2013 15:11:48,092 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:184)
 - Starting Source spool_WebLogs
11 Apr 2013 15:11:48,092 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:184)
 - Starting Source http_Default
11 Apr 2013 15:11:48,092 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.source.SpoolDirectorySource.start:66)
 - SpoolDirectorySource source starting with directory: c:\flume_data\spool\web
11 Apr 2013 15:11:48,124 INFO  [conf-file-poller-0] (org.mortbay.log.Slf4jLog.info:67)  -
Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
11 Apr 2013 15:11:48,139 INFO  [conf-file-poller-0] (org.mortbay.log.Slf4jLog.info:67)  -
jetty-6.1.26
11 Apr 2013 15:11:48,155 INFO  [conf-file-poller-0] (org.mortbay.log.Slf4jLog.info:67)  -
Started SocketConnector@0.0.0.0:41414<mailto:SocketConnector@0.0.0.0:41414>
11 Apr 2013 15:11:48,202 INFO  [lifecycleSupervisor-1-2] (org.mortbay.log.Slf4jLog.info:67)
 - jetty-6.1.26
11 Apr 2013 15:11:48,217 INFO  [lifecycleSupervisor-1-2] (org.mortbay.log.Slf4jLog.info:67)
 - Started SocketConnector@0.0.0.0:6240<mailto:SocketConnector@0.0.0.0:6240>
11 Apr 2013 15:11:48,389 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.sink.AvroSink.start:253)
 - Avro sink avro_Default started.
11 Apr 2013 15:11:48,404 ERROR [pool-6-thread-1] (org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile:442)
 - Exception opening file: c:\flume_data\spool\web\u_ex130411.log-201304111500.log
java.io.IOException: Unable to delete existing meta file c:\flume_data\spool\web\.flumespool\.flumespool-main.meta
 at org.apache.flume.serialization.DurablePositionTracker.getInstance(DurablePositionTracker.java:96)
 at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile(ReliableSpoolingFileEventReader.java:423)
 at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:212)
 at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:154)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
 at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
11 Apr 2013 15:11:48,919 ERROR [pool-6-thread-1] (org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile:442)
 - Exception opening file: c:\flume_data\spool\web\u_ex130411.log-201304111500.log
java.io.IOException: Unable to delete existing meta file c:\flume_data\spool\web\.flumespool\.flumespool-main.meta
 at org.apache.flume.serialization.DurablePositionTracker.getInstance(DurablePositionTracker.java:96)
 at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile(ReliableSpoolingFileEventReader.java:417)
 at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:212)
 at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:154)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
 at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)

Mime
View raw message