Return-Path: X-Original-To: apmail-falcon-dev-archive@minotaur.apache.org Delivered-To: apmail-falcon-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 500FEF2FC for ; Tue, 13 Aug 2013 08:38:14 +0000 (UTC) Received: (qmail 32389 invoked by uid 500); 13 Aug 2013 08:38:13 -0000 Delivered-To: apmail-falcon-dev-archive@falcon.apache.org Received: (qmail 32336 invoked by uid 500); 13 Aug 2013 08:38:12 -0000 Mailing-List: contact dev-help@falcon.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@falcon.incubator.apache.org Delivered-To: mailing list dev@falcon.incubator.apache.org Received: (qmail 32323 invoked by uid 99); 13 Aug 2013 08:38:12 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Aug 2013 08:38:12 +0000 X-ASF-Spam-Status: No, hits=-2002.7 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 13 Aug 2013 08:38:09 +0000 Received: (qmail 32242 invoked by uid 99); 13 Aug 2013 08:37:47 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Aug 2013 08:37:47 +0000 Date: Tue, 13 Aug 2013 08:37:47 +0000 (UTC) From: "Shwetha G S (JIRA)" To: dev@falcon.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (FALCON-81) globStatus in feed retention fails in some cases MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/FALCON-81?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shwetha G S updated FALCON-81: ------------------------------ Description: {noformat} If the data path is /data/${YEAR}/${MONTH}/${DAY}/${HOUR}/SomeData, feed eviction checks for pattern /data/*/*/*/*/SomeData which should be directory. Glob status on this fails if there is a file /data/2013/01/01/01/file: Failing Oozie Launcher, Main class [org.apache.ivory.retention.FeedEvictor], main() threw exception, org.apache.hadoop.security.AccessControlException: Permission denied: user=, access=EXECUTE, inode="/data/2013/01/01/01/file"::supergroup:-rw-r--r-- org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=dataload, access=EXECUTE, inode="/data/2013/01/01/01/file"::supergroup:-rw-r--r-- at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:914) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:523) at org.apache.hadoop.fs.FileSystem.getFileStatus(FileSystem.java:1534) at org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1096) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1038) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1016) at org.apache.ivory.retention.FeedEvictor.findFilesForFeed(FeedEvictor.java:204) at org.apache.ivory.retention.FeedEvictor.discoverInstanceToDelete(FeedEvictor.java:166) at org.apache.ivory.retention.FeedEvictor.run(FeedEvictor.java:112) This is because of an issue in glob status. Feed evictor should search with pattern /data/*/*/*/*/*/{SomeData} {noformat} was: {noformat} If the data path is /data/${YEAR}/${MONTH}/${DAY}/${HOUR}/SomeData, feed eviction checks for pattern /data/*/*/*/*/SomeData which should be directory. Glob status on this fails if there is a file /data/2013/01/01/01/file: Failing Oozie Launcher, Main class [org.apache.ivory.retention.FeedEvictor], main() threw exception, org.apache.hadoop.security.AccessControlException: Permission denied: user=, access=EXECUTE, inode="/data/2013/01/01/01/file"::supergroup:-rw-r--r-- org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=dataload, access=EXECUTE, inode="/data/2013/01/01/01/file"::supergroup:-rw-r--r-- at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:914) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:523) at org.apache.hadoop.fs.FileSystem.getFileStatus(FileSystem.java:1534) at org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1096) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1038) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1016) at org.apache.ivory.retention.FeedEvictor.findFilesForFeed(FeedEvictor.java:204) at org.apache.ivory.retention.FeedEvictor.discoverInstanceToDelete(FeedEvictor.java:166) at org.apache.ivory.retention.FeedEvictor.run(FeedEvictor.java:112) This is because of an issue in glob status. Feed evictor should search with pattern /data/*/*/*/*/*/{SomeData} {noformat} > globStatus in feed retention fails in some cases > ------------------------------------------------ > > Key: FALCON-81 > URL: https://issues.apache.org/jira/browse/FALCON-81 > Project: Falcon > Issue Type: Bug > Reporter: Shwetha G S > > {noformat} > If the data path is /data/${YEAR}/${MONTH}/${DAY}/${HOUR}/SomeData, feed eviction checks for pattern /data/*/*/*/*/SomeData which should be directory. > Glob status on this fails if there is a file /data/2013/01/01/01/file: > Failing Oozie Launcher, Main class [org.apache.ivory.retention.FeedEvictor], main() threw exception, org.apache.hadoop.security.AccessControlException: Permission denied: user=, access=EXECUTE, inode="/data/2013/01/01/01/file"::supergroup:-rw-r--r-- > org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=dataload, access=EXECUTE, inode="/data/2013/01/01/01/file"::supergroup:-rw-r--r-- > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) > at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57) > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:914) > at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:523) > at org.apache.hadoop.fs.FileSystem.getFileStatus(FileSystem.java:1534) > at org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1096) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1038) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1016) > at org.apache.ivory.retention.FeedEvictor.findFilesForFeed(FeedEvictor.java:204) > at org.apache.ivory.retention.FeedEvictor.discoverInstanceToDelete(FeedEvictor.java:166) > at org.apache.ivory.retention.FeedEvictor.run(FeedEvictor.java:112) > This is because of an issue in glob status. Feed evictor should search with pattern /data/*/*/*/*/*/{SomeData} > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira