falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peeyush Bishnoi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1146) feed retention policy deleted everything all the way up to the root
Date Wed, 08 Apr 2015 19:29:12 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485859#comment-14485859
] 

Peeyush Bishnoi commented on FALCON-1146:
-----------------------------------------

I am able to replicate this issue in which entire feed path deleted up to the root, if path
is empty. Currently feedBasePath (/adis/test) is not fully qualified when passed to deleteInstance
and to deleteParentIfEmpty function. But instance directory path is fully qualified (hdfs://champlain.hortonworks.local:8020/adis/test/2015/04/02/09/28)
which has to be deleted. When check was performed whether feedBasePath (/adis/test) matches
to parent feed instance directory path (hdfs://champlain.hortonworks.local:8020/adis/test)
to stop deletion, it did not match and thus deletion up to the root level happened (if path
is empty) as there is recursive call for deleteParentIfEmpty function.

{code:java}
private void deleteParentIfEmpty(FileSystem fs, Path parent, Path feedBasePath) throws IOException
{
        if (feedBasePath.equals(parent)) {
            LOG.info("Not deleting feed base path: {}", parent);
        } else {
            FileStatus[] files = fs.listStatus(parent);
            if (files != null && files.length == 0) {
                LOG.info("Parent path: {} is empty, deleting path", parent);
                if (fs.delete(parent, true)) {
                    LOG.info("Deleted empty dir: {}", parent);
                } else {
                    throw new IOException("Unable to delete parent path:" + parent);
                }
                deleteParentIfEmpty(fs, parent.getParent(), feedBasePath);
            }
        }
    }
{code}

After feedBasePath has been made fully qualified, it matches to the parent directory path
appropriately, thus it stop deletion up to the feedBasePath directory only. [~sriksun] I will
provide the patch for this issue so as to fix in 0.6.* itself.

> feed retention policy deleted everything all the way up to the root
> -------------------------------------------------------------------
>
>                 Key: FALCON-1146
>                 URL: https://issues.apache.org/jira/browse/FALCON-1146
>             Project: Falcon
>          Issue Type: Bug
>          Components: common, retention
>    Affects Versions: 0.6
>            Reporter: Peeyush Bishnoi
>            Assignee: Peeyush Bishnoi
>            Priority: Critical
>             Fix For: 0.6.1
>
>
> For the below feed definition xml,  feed is set to delete every minute but once it run
it deleted everything all the way up and even the root directory.
> {code}
> <?xml version="1.0" encoding="UTF-8"?>
> <feed description="ALU EVDO retention feed" name="alu-evdo-retention" xmlns="uri:falcon:feed:0.1">
>     <frequency>minutes(30)</frequency>
>     <clusters>
>         <cluster name="primary-cluster" type="source">
>             <validity start="2014-03-01T00:00Z" end="2016-01-01T00:00Z"/>
>             <retention limit="minutes(1)" action="delete"/>
>         </cluster>
>     </clusters>
>     <locations>
>         <location type="data" path="/adis/test/${YEAR}-${MONTH}-${DAY}/${HOUR}/${MINUTE}"/>
>     </locations>
>     <ACL owner="flume" group="landing" permission="0x777"/>
>     <schema location="/none" provider="none" />
> </feed>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message