falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peeyush Bishnoi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1146) feed retention policy deleted everything all the way up to the root
Date Wed, 08 Apr 2015 19:29:12 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485859#comment-14485859

Peeyush Bishnoi commented on FALCON-1146:

I am able to replicate this issue in which entire feed path deleted up to the root, if path
is empty. Currently feedBasePath (/adis/test) is not fully qualified when passed to deleteInstance
and to deleteParentIfEmpty function. But instance directory path is fully qualified (hdfs://champlain.hortonworks.local:8020/adis/test/2015/04/02/09/28)
which has to be deleted. When check was performed whether feedBasePath (/adis/test) matches
to parent feed instance directory path (hdfs://champlain.hortonworks.local:8020/adis/test)
to stop deletion, it did not match and thus deletion up to the root level happened (if path
is empty) as there is recursive call for deleteParentIfEmpty function.

private void deleteParentIfEmpty(FileSystem fs, Path parent, Path feedBasePath) throws IOException
        if (feedBasePath.equals(parent)) {
            LOG.info("Not deleting feed base path: {}", parent);
        } else {
            FileStatus[] files = fs.listStatus(parent);
            if (files != null && files.length == 0) {
                LOG.info("Parent path: {} is empty, deleting path", parent);
                if (fs.delete(parent, true)) {
                    LOG.info("Deleted empty dir: {}", parent);
                } else {
                    throw new IOException("Unable to delete parent path:" + parent);
                deleteParentIfEmpty(fs, parent.getParent(), feedBasePath);

After feedBasePath has been made fully qualified, it matches to the parent directory path
appropriately, thus it stop deletion up to the feedBasePath directory only. [~sriksun] I will
provide the patch for this issue so as to fix in 0.6.* itself.

> feed retention policy deleted everything all the way up to the root
> -------------------------------------------------------------------
>                 Key: FALCON-1146
>                 URL: https://issues.apache.org/jira/browse/FALCON-1146
>             Project: Falcon
>          Issue Type: Bug
>          Components: common, retention
>    Affects Versions: 0.6
>            Reporter: Peeyush Bishnoi
>            Assignee: Peeyush Bishnoi
>            Priority: Critical
>             Fix For: 0.6.1
> For the below feed definition xml,  feed is set to delete every minute but once it run
it deleted everything all the way up and even the root directory.
> {code}
> <?xml version="1.0" encoding="UTF-8"?>
> <feed description="ALU EVDO retention feed" name="alu-evdo-retention" xmlns="uri:falcon:feed:0.1">
>     <frequency>minutes(30)</frequency>
>     <clusters>
>         <cluster name="primary-cluster" type="source">
>             <validity start="2014-03-01T00:00Z" end="2016-01-01T00:00Z"/>
>             <retention limit="minutes(1)" action="delete"/>
>         </cluster>
>     </clusters>
>     <locations>
>         <location type="data" path="/adis/test/${YEAR}-${MONTH}-${DAY}/${HOUR}/${MINUTE}"/>
>     </locations>
>     <ACL owner="flume" group="landing" permission="0x777"/>
>     <schema location="/none" provider="none" />
> </feed>
> {code}

This message was sent by Atlassian JIRA

View raw message