falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Balu Vellanki (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-141) Support cluster updates
Date Thu, 03 Dec 2015 22:15:11 GMT

    [ https://issues.apache.org/jira/browse/FALCON-141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038706#comment-15038706

Balu Vellanki commented on FALCON-141:

[~sriksun] and [~venkatnrangan] : Here is the summary of our internal discussion. Based on
the summary of all discussion and meetings, I will create subtasks for this jira. 

In most real-world scenarios, updating a cluster from non-HA to HA will require the following
1. Shutdown falcon.
2. Update hadoop cluster from non-HA to HA (or non-secure to secure).
3. Update falcon configs like startup.properties
4. Start falcon. 

The initial solution was to update cluster entity after step-4 in a manner similar to updating
a feed/process entity. This approach will not work because,
- When falcon is started, SharedLibraryHostingService needs to connect to hdfs of each cluster
and copy relevant jars to <cluster_working_dir>/lib/.  Since the cluster entity has
old values for read/write endpoints, Falcon will fail in this step and will not start.  The
solution is to start falcon in a safe-mode, where falcon can start without having to access
cluster-entity's hdfs location. 

Proposed Solution :
1. Shutdown falcon.
2. Update hadoop cluster from non-HA to HA (or non-secure to secure).
3. Start Falcon in safe-mode. In this mode 
   - Falcon starts without starting SharedLibraryHostingService. 
   - SuperUsers of falcon (as specified in startup.properties) can update falcon cluster-entity
   - No other write-operations are allowed in safe-mode. 
4. Update cluster-entity in Falcon
   - In this step, a SuperUser can update the existing cluster entity. The following fields
can be updated by SuperUser
        a) Cluster description, tags : Falcon will update the graphDB with new description
and tags
        b) All interfaces : The underlying read/write and workflow interfaces should be the
same even if the url:port has changed
        c) All locations : Falcon will validate new locations.
        d) Properties
   - Get a lock on cluster entity in the ConfigStore.
   - If any interface, location or property is updated, the feeds and processes dependent
on this cluster should be updated in the workflow engine. This should be done once Falcon
is started in normal mode. So add a "requireUpdate" flag to each entity in the ConfigStore
to specify that the feed/process entity should be updated in workflow engine. 
   - Commit the new cluster entity to ConfigStore after validating entity.
5. Re-start Falcon in normal mode. This requires following tasks to be added to existing falcon
   - start SharedLibraryHostingService along with other services.
   - Update coordinator/bundle of all feed/process entities that are flagged during cluster
entity update. The user of coord/bundle should be same as the prev owner of coord/bundle.

Handling failures in cluster update :
1. If update cluster entity fails, throw an exception to SuperUser with the reason. The user
will have to fix and restart cluster entity.
2. If update of coord/bundle of dependent entity fail,
    - Retain the requireUpdate flag on the entity.
    - Continue updating bundle/coord of remaining feed/process entities. 
    - At the end of Falcon start, show warning to user with list of entities that could not
be updated. 
    - Provide ability to update bundle/coord of flagged entities individually. 

> Support cluster updates
> -----------------------
>                 Key: FALCON-141
>                 URL: https://issues.apache.org/jira/browse/FALCON-141
>             Project: Falcon
>          Issue Type: Bug
>            Reporter: Shwetha G S
>            Assignee: Ajay Yadava

This message was sent by Atlassian JIRA

View raw message