Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D1F3ADA4E for ; Sat, 17 Nov 2012 16:02:20 +0000 (UTC) Received: (qmail 59246 invoked by uid 500); 17 Nov 2012 16:02:20 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 58974 invoked by uid 500); 17 Nov 2012 16:02:20 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 58953 invoked by uid 99); 17 Nov 2012 16:02:19 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Nov 2012 16:02:19 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of danoyoung@gmail.com designates 209.85.223.179 as permitted sender) Received: from [209.85.223.179] (HELO mail-ie0-f179.google.com) (209.85.223.179) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Nov 2012 16:02:14 +0000 Received: by mail-ie0-f179.google.com with SMTP id c10so4692452ieb.38 for ; Sat, 17 Nov 2012 08:01:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=3yCEVb87tYXuwKfiTLzEOmjXexxAayd/45jLAo4+7jc=; b=g0FX2kEOzTPgKfiB99fq7mHCk44alVvSRGx/7IFhNtrijiBWPfmunLNWZ2Z/ay9yRf KjO6QublIvUpNyn0cVZlADcX+3QTGpI8vCu9djv36/3bf46fUy7CbrZ4r1u80tn3tIwj McLFvY9OpV8sYJYIjTSA5Le86POlHNisYg6rAZ+LyE+nye/EmfTpwiDneudEf9R3CQc+ nA6EZom+HZiwKcSNZyeni4rjznAIjWH8/JgW/j9LMoxuEn7QzmX4WvE2qZ5XYqCjdkf+ JTIEiorQiiiHtu12btGUp3DzgoUlTmgPhWzmmtDiIDXBqJSDNkLW0LQCzenwp26UJLkH xIJA== MIME-Version: 1.0 Received: by 10.50.149.225 with SMTP id ud1mr2113464igb.32.1353168113954; Sat, 17 Nov 2012 08:01:53 -0800 (PST) Received: by 10.231.71.77 with HTTP; Sat, 17 Nov 2012 08:01:53 -0800 (PST) Received: by 10.231.71.77 with HTTP; Sat, 17 Nov 2012 08:01:53 -0800 (PST) In-Reply-To: References: Date: Sat, 17 Nov 2012 09:01:53 -0700 Message-ID: Subject: Re: .SpoolingFileLineReader warning.... From: Dan Young To: user@flume.apache.org Content-Type: multipart/alternative; boundary=e89a8f3baf03c0ef0704ceb30067 X-Virus-Checked: Checked by ClamAV on apache.org --e89a8f3baf03c0ef0704ceb30067 Content-Type: text/plain; charset=ISO-8859-1 Hey Brock, No I have not modified the conf while the agent was running. /mnt/flume is local. Note that this is running on an ec2 instance and the disk is the ephemeral drive, not EBS. Regards , Dano On Nov 17, 2012 8:58 AM, "Brock Noland" wrote: > Hi, > > I highly doubt it's related to > (https://issues.apache.org/jira/browse/FLUME-1721) but have you > modified the configuration file since starting the agent? If so, can > you restart the agent and see if the error continues? > > Also, is /mnt/flume local disk or NAS? > > Brock > > On Sat, Nov 17, 2012 at 9:02 AM, Dan Young wrote: > > First a bit of context, I'm using logrotate to monitor and copy (cp -p) > log > > files to a flume spooling directory source. So every hour, logrotate > checks > > for and copies a file from the source to the flume destination. I see the > > following warning message in the flume logs. > > > > > > 17 Nov 2012 14:47:07,682 WARN [pool-10-thread-1] > > (org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile:328) - > > Could not find file: > > /mnt/flume/clickstream/clickstream.log-2012-11-17-1353163623 > > java.io.FileNotFoundException: > > /mnt/flume/clickstream/clickstream.log-2012-11-17-1353163623 (Permission > > denied) > > at java.io.FileInputStream.open(Native Method) > > at java.io.FileInputStream.(FileInputStream.java:138) > > at java.io.FileReader.(FileReader.java:72) > > at > > > org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile(SpoolingFileLineReader.java:322) > > at > > > org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:172) > > at > > > org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > at > > > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:722) > > > > > > Although it appears that Flume processes the log, I'm curious why I''m > > seeing this and if I have anything with permissions incorrect? > > > > > > > > Here's the permissions: > > > > source log directory under /var/log: > > drwxrwxr-x 2 ubuntu ubuntu 4096 Nov 17 14:47 clickstream > > > > source files: > > -rw-rw-r-- 1 ubuntu ubuntu 9055750 Nov 17 13:29 > > clickstream.log-2012-11-17-1353158953.gz > > -rw-rw-r-- 1 ubuntu ubuntu 13583565 Nov 17 14:17 > > clickstream.log-2012-11-17-1353161821.gz > > -rw-rw-r-- 1 ubuntu ubuntu 131296672 Nov 17 14:47 > > clickstream.log-2012-11-17-1353163623 > > -rw-rw-r-- 1 ubuntu ubuntu 65648336 Nov 17 14:52 clickstream.log > > > > flume source directory under /mnt/flume: > > drwxrwxr-x 2 ubuntu ubuntu 4096 Nov 17 14:48 clickstream > > > > flume source files: > > -rw-rw-r-- 1 ubuntu ubuntu 131296672 Nov 17 13:29 > > clickstream.log-2012-11-17-1353158953.COMPLETED > > -rw-rw-r-- 1 ubuntu ubuntu 196945008 Nov 17 14:17 > > clickstream.log-2012-11-17-1353161821.COMPLETED > > -rw-rw-r-- 1 ubuntu ubuntu 131296672 Nov 17 14:47 > > clickstream.log-2012-11-17-1353163623.COMPLETED > > > > Any insight would be appreciated. > > > > Regards, > > > > Dan > > > > -- > Apache MRUnit - Unit testing MapReduce - > http://incubator.apache.org/mrunit/ > --e89a8f3baf03c0ef0704ceb30067 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Hey Brock,

No I have not modified the conf while the agent was running.=

/mnt/flume is local. Note that this is running on an ec2 ins= tance and the disk is the ephemeral drive, not EBS.

Regards ,

Dano

On Nov 17, 2012 8:58 AM, "Brock Noland"= ; <brock@cloudera.com> wrot= e:
Hi,

I highly doubt it's related to
(https://issues.apache.org/jira/browse/FLUME-1721) but have you
modified the configuration file since starting the agent? =A0If so, can
you restart the agent and see if the error continues?

Also, is /mnt/flume local disk or NAS?

Brock

On Sat, Nov 17, 2012 at 9:02 AM, Dan Young <danoyoung@gmail.com> wrote:
> First a bit of context, I'm using logrotate to monitor and copy (c= p -p) log
> files to a flume spooling directory source. =A0So every hour, logrotat= e checks
> for and copies a file from the source to the flume destination. I see = the
> following warning message in the flume logs.
>
>
> 17 Nov 2012 14:47:07,682 WARN =A0[pool-10-thread-1]
> (org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile:328) = =A0-
> Could not find file:
> /mnt/flume/clickstream/clickstream.log-2012-11-17-1353163623
> java.io.FileNotFoundException:
> /mnt/flume/clickstream/clickstream.log-2012-11-17-1353163623 (Permissi= on
> denied)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.<init>(FileInputStream.java:138)
> at java.io.FileReader.<init>(FileReader.java:72)
> at
> org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile(Spooli= ngFileLineReader.java:322)
> at
> org.apache.flume.client.avro.SpoolingFileLineReader.readLines(Spooling= FileLineReader.java:172)
> at
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.ru= n(SpoolDirectorySource.java:135)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:= 471)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:= 351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) > at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.a= ccess$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.r= un(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j= ava:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.= java:603)
> at java.lang.Thread.run(Thread.java:722)
>
>
> Although it appears that Flume processes the log, I'm curious why = I''m
> seeing this and if I have anything with permissions incorrect?
>
>
>
> Here's the permissions:
>
> source log directory under /var/log:
> drwxrwxr-x 2 ubuntu =A0 =A0ubuntu =A0 4096 Nov 17 14:47 clickstream >
> source files:
> -rw-rw-r-- 1 ubuntu ubuntu =A0 9055750 Nov 17 13:29
> clickstream.log-2012-11-17-1353158953.gz
> -rw-rw-r-- 1 ubuntu ubuntu =A013583565 Nov 17 14:17
> clickstream.log-2012-11-17-1353161821.gz
> -rw-rw-r-- 1 ubuntu ubuntu 131296672 Nov 17 14:47
> clickstream.log-2012-11-17-1353163623
> -rw-rw-r-- 1 ubuntu ubuntu =A065648336 Nov 17 14:52 clickstream.log >
> flume source directory under /mnt/flume:
> drwxrwxr-x 2 ubuntu ubuntu 4096 Nov 17 14:48 clickstream
>
> flume source files:
> -rw-rw-r-- 1 ubuntu ubuntu 131296672 Nov 17 13:29
> clickstream.log-2012-11-17-1353158953.COMPLETED
> -rw-rw-r-- 1 ubuntu ubuntu 196945008 Nov 17 14:17
> clickstream.log-2012-11-17-1353161821.COMPLETED
> -rw-rw-r-- 1 ubuntu ubuntu 131296672 Nov 17 14:47
> clickstream.log-2012-11-17-1353163623.COMPLETED
>
> Any insight would be appreciated.
>
> Regards,
>
> Dan



--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
--e89a8f3baf03c0ef0704ceb30067--