falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venkat Ranganathan <vranganat...@hortonworks.com>
Subject Re: [DISCUSS]: Moving recipe cooking to server
Date Fri, 13 Mar 2015 02:47:37 GMT
It looks like the initial thought was to have the recipe processing a 
client side feature because of isolation issues.  It has its merits 
(primarily in terms of allowing Falcon server insulated from the recipe 
processing), client side recipes make the ability to catalog, query, reuse 
and management of the recipes a little difficult.  

I agree that we don’t want JSPs.  I don’t think it is a good idea to 
expect Java applet capability in the UI.  It brings its own list of 
security concerns.

Using oozie seems like a natural approach given Falcon uses Oozie a lot.  
But the operational issue with this approach is said to be the overall 
apparatus needed to monitor and manage the recipe-cooking worfklows.   
Oozie scheduling can bring arbitrary delays (as we have seen internally 
also).   Granted we can design around the limitations and make use of the 
strengths of the approach but it seems something we can avoid if we can.

Option 2 is to fork a new process in falcon server for Option 2 and may be 
we throttle how many processes we spawn at a time.       We may have some 
work in secure clusters, but this will give the necessary control over the 
recipe cooking while not needing additional operational work.   We will 
write this in such a way that if we choose to use the oozie apprach, the 
code can be made as part of a Oozie java action with some minimal effort.


Thanks

Venkat

On 3/12/15, 12:53 PM, "Arpit Gupta" <arpit@hortonworks.com> wrote:

>I fail to understand why is it best to keep recipe client side.
>
>Also if the user is allowed to run any arbitrary java code for recipe 
>cooking and it runs in the browser how would we handle if the user code 
>wanted to check something on hdfs? If the cluster was secure? Nothing 
>over rpc would work. Then that means they cant write any arbitrary code. 
>May be i am missing something in what kind of user code are we expecting.
>
>for option #2 will falcon handle forking off another process? 
>--
>Arpit Gupta
>Hortonworks Inc.
>http://hortonworks.com/
>
>> On Mar 12, 2015, at 11:46 AM, Ying Zheng <yzheng@hortonworks.com> wrote:
>> 
>> The first question is whether we need to move recipe to server side or
>> not, no matter which webpage we use (JSP or HTML5). As far as I
>> understand, it is best to keep it on client side. Yes? If we choose 
>>HTML5,
>> one way is to use JAVA applet in HTML5.
>> 
>> 
>> On 3/12/15, 9:31 AM, "Siva Thumma" <sivatumma@gmail.com> wrote:
>> 
>>> +1.
>>> 
>>>> On 12-Mar-2015, at 8:58 pm, Arpit Gupta <arpit@hortonworks.com> wrote:
>>>> 
>>>> Are people not moving away from JSP's? I know all UI's in Hadoop moved
>>>> away from JSP recently to html 5
>>>> https://issues.apache.org/jira/browse/HADOOP-10563
>>>> 
>>>> --
>>>> Arpit Gupta
>>>> Hortonworks Inc.
>>>> http://hortonworks.com/
>>>> 
>>>> On Mar 12, 2015, at 8:13 AM, Ying Zheng
>>>> <yzheng@hortonworks.com<mailto:yzheng@hortonworks.com>> wrote:
>>>> 
>>>> Sort of, but not a completely new server. We don't need to write a new
>>>> server. As long as we have Tomcat installed and running, we just need 
>>>>to
>>>> copy front end JSP files to a folder under tomcat. In this way, we
>>>> separate front end and back end.
>>>> 
>>>> On Mar 12, 2015 7:39 AM, Srikanth Sundarrajan
>>>> <sriksun@hotmail.com<mailto:sriksun@hotmail.com>> wrote:
>>>> Are you suggesting a completely new server for cooking recipe?
>>>> 
>>>> Sent from my iPhone
>>>> 
>>>> On 12-Mar-2015, at 8:03 pm, "Ying Zheng"
>>>> <yzheng@hortonworks.com<mailto:yzheng@hortonworks.com>> wrote:
>>>> 
>>>> Perhaps I don't understand your question. JSP is one way to generate
>>>> webpages. Why does it need to run in Falcon server? It can be isolated
>>>> from Falcon server, and developed and run in Apache Tomcat server.
>>>> 
>>>> Thanks,
>>>> Ying
>>>> 
>>>> On Mar 12, 2015 12:53 AM, Srikanth Sundarrajan
>>>> <sriksun@hotmail.com<mailto:sriksun@hotmail.com>> wrote:
>>>> Hi Ying,
>>>> JSP would still run in Falcon server JVM, so not sure if that is any
>>>> different from implementing this natively inside the falcon server.
>>>> 
>>>> Regards
>>>> Srikanth Sundarrajan
>>>> 
>>>> ----------------------------------------
>>>> Subject: Re: [DISCUSS]: Moving recipe cooking to server
>>>> From: yzheng@hortonworks.com<mailto:yzheng@hortonworks.com>
>>>> To: dev@falcon.apache.org<mailto:dev@falcon.apache.org>
>>>> Date: Thu, 12 Mar 2015 07:43:28 +0000
>>>> 
>>>> Hi Sowmya, Srikanth, Venkatesh and Venkat,
>>>> 
>>>> As far as I understand, recipe helps the user generate the process
>>>> entity
>>>> xml and it looks best to keep it a client side logic, as what you did
>>>> before. The current question is how to run recipe logic on DR UI. To
>>>> avoid
>>>> repeating all the recipe logic in JS, one choice you can consider is
>>>> JSP.
>>>> JSP can be supported by Apache Tomcat server. If you package the 
>>>>recipe
>>>> code into JAR, it can be reused in JSP. JS and CSS can also be 
>>>>included
>>>> in
>>>> the JSP page.
>>>> 
>>>> Cheers,
>>>> Ying
>>>> 
>>>> 
>>>> On 3/11/15, 11:51 PM, "Srikanth Sundarrajan"
>>>> <sriksun@hotmail.com<mailto:sriksun@hotmail.com>> wrote:
>>>> 
>>>> Thanks Sowmya for capturing the discussion and also initiating this
>>>> conversation to drive consensus around the design. I am inclined 
>>>>towards
>>>> Approach 2.
>>>> 
>>>> Regards
>>>> Srikanth Sundarrajan
>>>> 
>>>> ----------------------------------------
>>>> Subject: [DISCUSS]: Moving recipe cooking to server
>>>> From: sramesh@hortonworks.com<mailto:sramesh@hortonworks.com>
>>>> To: dev@falcon.apache.org<mailto:dev@falcon.apache.org>
>>>> Date: Thu, 12 Mar 2015 05:17:16 +0000
>>>> 
>>>> I had a discussion with Srikanth, Venkatesh and Venkat regarding 
>>>>making
>>>> recipe processing server side concept. I am sending out the summary of
>>>> the meeting and opening it for further discussion.
>>>> 
>>>> Today Recipe cooking is a client side logic. Recipe also supports
>>>> extensions i.e. user can cook his/her own custom recipes.
>>>> Decision to make it client side logic was for the following reasons
>>>> 
>>>> * Keep it isolated from falcon server
>>>> 
>>>> * As custom recipe cooking is supported, user recipes can introduce
>>>> security vulnerabilities and also can bring down the falcon server
>>>> 
>>>> Today, falcon provides HDFS DR recipe out of the box. There is a plan
>>>> to add UI support for DR in Falcon.
>>>> Rest API support cannot be added for recipe as it is client side
>>>> processing.
>>>> If the UI is pure java script[JS] then all the recipe cooking logic 
>>>>has
>>>> to be repeated in JS. This is not a feasible solution - if more 
>>>>recipes
>>>> are added say DR for hive, hbase and others, UI won't be extensible.
>>>> 
>>>> For the above mentioned reasons Recipe should me made a server side
>>>> logic.
>>>> Provided recipes [recipes provided out of the box] can run as Falcon
>>>> process. Recipe cooking will be done in a new process if its custom
>>>> recipe [user code].
>>>> 
>>>> For cooking of custom recipes, design proposed should consider 
>>>>handling
>>>> security implications, handling the issues where the custom user code
>>>> can bring down the Falcon server (trapping System.exit), handling 
>>>>class
>>>> path isolation.
>>>> Also it shouldn't in anyway destabilize the Falcon system.
>>>> 
>>>> There are couple of approaches which was discussed
>>>> 
>>>> Approach 1:
>>>> Custom Recipe cooking can be carried out separately in another Oozie
>>>> WF, this will ensure isolation. Oozie already has the ability to
>>>> schedule jobs as a user and handles all the security aspects of it.
>>>> 
>>>> Pros:
>>>> - Provides isolation
>>>> - Piggyback on Oozie as it already provides the required functionality
>>>> 
>>>> Cons:
>>>> - As recipe processing is done in different WF, from operations point
>>>> of view user cannot figure out recipe processing status and thus adds 
>>>>to
>>>> the operational pain.
>>>> 
>>>> Approach 2:
>>>> Custom recipe cooking is done on the server side in a separate
>>>> independent process than Falcon process I.e. It runs in a different 
>>>>JVM.
>>>> Throttling should be added for how many recipe cooking processes can 
>>>>be
>>>> launched keeping in mind the machine configuration.
>>>> 
>>>> Pros:
>>>> - Provides isolation as recipe cooking is done in a independent 
>>>>process
>>>> 
>>>> Cons:
>>>> - Performance overhead as new process is launched for custom recipe
>>>> cooking
>>>> - Adds more complexity to the system
>>>> 
>>>> Thoughts?
>>>> 
>>>> Thanks,
>>>> Sowmya
>>>> 
>>>> 
>> 
>
Mime
View raw message