falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pallavi Rao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1368) Improve Falcon server restart time
Date Thu, 06 Aug 2015 05:00:08 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659492#comment-14659492

Pallavi Rao commented on FALCON-1368:

[~sandeep.samudrala], one thing we'll need to consider when loading the config store in parallel
is that, there are a bunch of existing listeners that get invoked when each of the entity
gets reloaded. Now that we are loading the entities in parallel, the listeners would have
to be thread safe too.  We'll have to ensure with enough tests to ensure we don't have any
unnecessary side effects.

> Improve Falcon server restart  time
> -----------------------------------
>                 Key: FALCON-1368
>                 URL: https://issues.apache.org/jira/browse/FALCON-1368
>             Project: Falcon
>          Issue Type: Improvement
>            Reporter: Ajay Yadava
>            Assignee: sandeep samudrala
>         Attachments: FALCON-1368-V1.patch, FALCON-1368.patch
> Currently on restart, Falcon server loads all the entities from HDFS one by one. In a
large set up like the one at Inmobi, where we have several thousands of feeds and processes,
this takes several minutes to increase the start up time.
> Since this is an IO intensive task(reading a file from HDFS into memory), having multiple
threads to load entities in parallel will improve the start up time of the server. 
> Two points need to be taken care of:
> 1. Only a single type of entities should be load in parallel to preserve the order of
load of entities.
> 2. Currently Falcon server fails to start if there is an error in loading any entity,
it will be slightly tricky but we should preserve the same behaviour in case of parallel uploads
via threads.

This message was sent by Atlassian JIRA

View raw message