falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shwetha GS <shwetha...@inmobi.com>
Subject Re: Using properties in processes
Date Thu, 10 Jul 2014 14:50:53 GMT
Falcon internally uses oozie. Looks like oozie doesn't support this. 

Will check oozie tomorrow and see if it can be fixed in oozie

Sent from my iPhone

> On Jul 10, 2014, at 5:59 PM, Josh Clum <joshclum@gmail.com> wrote:
> 
> It took me forever, but I finally found this error in the bundle logs. If I
> reference a property in my process xml file,
> 
> ie <property name="indexSelect" value="Select .... where
> ${falcon_input_partition_filter_hive}" />
> 
> then I get:
> 
> at
> org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:696)
> at
> org.apache.oozie.command.coord.CoordSubmitXCommand.resolveTagContents(CoordSubmitXCommand.java:895)
> ... 10 more
> Caused by: javax.servlet.jsp.el.ELException: variable
> [falcon_input_partition_filter_hive] cannot be resolved
> at
> org.apache.oozie.util.ELEvaluator$Context.resolveVariable(ELEvaluator.java:106)
> at org.apache.commons.el.NamedValue.evaluate(NamedValue.java:124)
> at
> org.apache.commons.el.ExpressionString.evaluate(ExpressionString.java:114)
> at
> org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:274)
> at
> org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190)
> at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:203)
> at
> org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:687)
> ... 11 more
> 
> If I try and use the oozie el functions in my process xml
> 
> ie <property name="indexSelect" value="Select .... where
> ${wf:conf('falcon_input_partition_filter_hive')}" />
> 
> then I get:
> 
> at
> org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:696)
> at
> org.apache.oozie.command.coord.CoordSubmitXCommand.resolveTagContents(CoordSubmitXCommand.java:895)
> ... 10 more
> Caused by: javax.servlet.jsp.el.ELException: No function is mapped to the
> name "wf:conf"
> at org.apache.commons.el.Logger.logError(Logger.java:481)
> at org.apache.commons.el.Logger.logError(Logger.java:498)
> at org.apache.commons.el.Logger.logError(Logger.java:525)
> at
> org.apache.commons.el.FunctionInvocation.evaluate(FunctionInvocation.java:150)
> at
> org.apache.commons.el.ExpressionString.evaluate(ExpressionString.java:114)
> at
> org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:274)
> at
> org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190)
> at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:203)
> at
> org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:687)
> ... 11 more
> 
> I also tried defining a property earlier in my process xml and then using
> it later with no luck.
> 
> ie
> 
> <properties>
> 
> ...
> 
> <property name="foo" value=":)" />
> ...
> <property name="bar" value="${foo}" />
> 
> ...
> 
> </properties>
> 
> which still gives me the ELException: variable
> [falcon_input_partition_filter_hive] cannot be resolved.
> 
> So it would seem that there is no way to access configurations within the
> process xml itself. Is this true?
> 
> Thanks,
> 
> Josh
> 
> 
> On Wed, Jul 9, 2014 at 11:05 AM, Seetharam Venkatesh <
> venkatesh@innerzeal.com> wrote:
> 
>> Josh,
>> 
>> These variables are exposed to user. The convention is:
>> 
>> falcon_${input_feed_name}_partition_filter_pig
>> falcon_${input_feed_name}_partition_filter_hive
>> falcon_${input_feed_name}_partition_filter_java
>> falcon_${input_feed_name}_datain_partitions_hive
>> 
>> You should be able to use any of these.
>> 
>> If you are having issues with oozie starting the jobs, its good to look at
>> oozie.log and see if there an error with any EL.
>> 
>> Thanks!
>> 
>> 
>> On Wed, Jul 9, 2014 at 12:57 PM, Srikanth Sundarrajan <sriksun@hotmail.com
>> wrote:
>> 
>>> In general it is not a good practice to refer to Falcon internal
>>> variables, but this might be a public one. Will let @Venkatesh confirm
>>> this. If it is indeed public, this ought to work, but this may also lead
>> to
>>> the possibility of cyclic referencing of properties. A peek into the
>> Oozie
>>> logs might give ideas on what might be the reason for coord not starting.
>>> 
>>> Regards
>>> Srikanth Sundarrajan
>>> 
>>> 
>>> ----------------------------------------
>>>> From: joshclum@gmail.com
>>>> Date: Tue, 8 Jul 2014 15:47:10 -0400
>>>> Subject: Using properties in processes
>>>> To: dev@falcon.incubator.apache.org
>>>> 
>>>> Hi,
>>>> 
>>>> Is it possible to reference properties like
>>> falcon_input_partition_filter_hive
>>>> inside of the process xml definition itself?
>>>> 
>>>> For instance:
>>>> 
>>>> <properties>
>>>> ...
>>>> <property name="indexSelect" value="Select .... where
>>>> ${falcon_input_partition_filter_hive}" />
>>>> 
>>>> ...
>>>> 
>>>> </properties>
>>>> 
>>>> 
>>>> When I try to do this, I don't get any errors, but my process never
>>> starts
>>>> up either. Looked in the falcon logs with no luck. Ideas?
>>>> 
>>>> Thanks,
>>>> Josh
>> 
>> 
>> 
>> --
>> Regards,
>> Venkatesh
>> 
>> “Perfection (in design) is achieved not when there is nothing more to add,
>> but rather when there is nothing more to take away.”
>> - Antoine de Saint-Exupéry
>> 

-- 
_____________________________________________________________
The information contained in this communication is intended solely for the 
use of the individual or entity to whom it is addressed and others 
authorized to receive it. It may contain confidential or legally privileged 
information. If you are not the intended recipient you are hereby notified 
that any disclosure, copying, distribution or taking any action in reliance 
on the contents of this information is strictly prohibited and may be 
unlawful. If you have received this communication in error, please notify 
us immediately by responding to this email and then delete it from your 
system. The firm is neither liable for the proper and complete transmission 
of the information contained in this communication nor for any delay in its 
receipt.

Mime
View raw message