falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ajay Yadava (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FALCON-1368) Improve Falcon server restart time
Date Wed, 05 Aug 2015 03:10:04 GMT
Ajay Yadava created FALCON-1368:
-----------------------------------

             Summary: Improve Falcon server restart  time
                 Key: FALCON-1368
                 URL: https://issues.apache.org/jira/browse/FALCON-1368
             Project: Falcon
          Issue Type: Improvement
            Reporter: Ajay Yadava


Currently on restart, Falcon server loads all the entities from HDFS one by one. In a large
set up like the one at Inmobi, where we have several thousands of feeds and processes, this
takes several minutes to increase the start up time.

Since this is an IO intensive task(reading a file from HDFS into memory), having multiple
threads to load entities in parallel will improve the start up time of the server. 

Two points need to be taken care of:
1. Only a single type of entities should be load in parallel to preserve the order of load
of entities.

2. Currently Falcon server fails to start if there is an error in loading any entity, it will
be slightly tricky but we should preserve the same behaviour in case of parallel uploads via
threads.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message