Mailing-List: contact dev-help@falcon.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@falcon.apache.org
Date: Fri, 11 Mar 2016 11:35:42 +0000 (UTC)
From: "Pallavi Rao (JIRA)" <jira@apache.org>
To: dev@falcon.incubator.apache.org
Message-ID: <JIRA.12949056.1457694259000.714.1457696142298@Atlassian.JIRA>
In-Reply-To: <JIRA.12949056.1457694259000@Atlassian.JIRA>
References: <JIRA.12949056.1457694259000@Atlassian.JIRA>
 <JIRA.12949056.1457694259398@arcas>
Subject: [jira] [Commented] (FALCON-1852) Optional Input for a process not
 truly optional
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/FALCON-1852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190822#comment-15190822 ] 

Pallavi Rao commented on FALCON-1852:
-------------------------------------

The change, if made will be backward incompatible. Any historical reason/opinion on why this change should not be made?

> Optional Input for a process not truly optional
> -----------------------------------------------
>
>                 Key: FALCON-1852
>                 URL: https://issues.apache.org/jira/browse/FALCON-1852
>             Project: Falcon
>          Issue Type: Bug
>            Reporter: Pallavi Rao
>            Assignee: Pallavi Rao
>
> Currently, when a feed input is marked as optional, we do not add it to the coordinator definition's datasets. This means we do not wait for all instances (for a given data window) to arrive. Instead, we just resolve the paths for a data window and pass it as a parameter.
> For example:
> {noformat}
> <inputs>
>         <!-- In the workflow, the input paths will be available in a variable 'inpaths' -->
>         <input name="inpaths" feed="in" start="now(0,-5)" end="now(0,-1)"/>
>         <input name="in2paths" feed="in2" start="now(0,-5)" end="now(0,-1)" optional="true"/>
>     </inputs>
> {noformat}
> For a process instance 2013-01-01T00:00Z, the optional input, in2paths, will be resolved as below:
> {noformat}
>  <property>
>     <name>in2paths</name>
>     <value>hdfs://localhost:9000/data/in2/2013/11/15/00/04,hdfs://localhost:9000/data/in2/2013/11/15/00/03,hdfs://localhost:9000/data/in2/2013/11/15/00/02,hdfs://localhost:9000/data/in2/2013/11/15/00/01,hdfs://localhost:9000/data/in2/2013/11/15/00/00</value>
>   </property>
> {noformat}
> If one of the instance of in2paths (example, hdfs://localhost:9000/data/in2/2013/11/15/00/04) is missing, the workflow will fail anyway.
> Hence, input, in2paths is not truly optional. Only that the triggering of instance is not gated on it.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)