falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pallavi Rao" <pallavi....@inmobi.com>
Subject Re: Review Request 35724: Base framework of the native scheduler
Date Mon, 19 Oct 2015 07:15:36 GMT


> On Sept. 14, 2015, 6:43 p.m., Srikanth Sundarrajan wrote:
> > scheduler/pom.xml, line 57
> > <https://reviews.apache.org/r/35724/diff/2/?file=1023467#file1023467line57>
> >
> >     Why would we have oozie-adaptor dependency in Scheduler ?

The Builders are still part of that package and hence the dependency. Will take up the refactor
a little later.


> On Sept. 14, 2015, 6:43 p.m., Srikanth Sundarrajan wrote:
> > scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java, line
104
> > <https://reviews.apache.org/r/35724/diff/2/?file=1023473#file1023473line104>
> >
> >     How do we handle re-runs in this world ? Will ExecutionInstance be different
for different runs ?

Yes. The plan is to have have multiple instances. So, attempt will be part of an instance's
identity. Will be addressed in FALCON-1512.


> On Sept. 14, 2015, 6:43 p.m., Srikanth Sundarrajan wrote:
> > scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java,
line 120
> > <https://reviews.apache.org/r/35724/diff/2/?file=1023476#file1023476line120>
> >
> >     DI ?

The services are actually registered as services via the startup properties. The NotificationServicesRegistry
is more like a convenience class.


> On Sept. 14, 2015, 6:43 p.m., Srikanth Sundarrajan wrote:
> > scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java,
line 121
> > <https://reviews.apache.org/r/35724/diff/2/?file=1023476#file1023476line121>
> >
> >     Can't quite get where the feed EL's being evaluated and paths being resolved.

That will come as part of FALCON-1230


> On Sept. 14, 2015, 6:43 p.m., Srikanth Sundarrajan wrote:
> > scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java,
line 126
> > <https://reviews.apache.org/r/35724/diff/2/?file=1023476#file1023476line126>
> >
> >     Not sure why data availability notification is being registered against ServicesRegistry.

The services are actually registered as services via the startup properties. The NotificationServicesRegistry
is more like a convenience class.


> On Sept. 14, 2015, 6:43 p.m., Srikanth Sundarrajan wrote:
> > scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java, line 141
> > <https://reviews.apache.org/r/35724/diff/2/?file=1023477#file1023477line141>
> >
> >     Would this not be common for both feed & process executors ?

Methods may be similar, but, objects used may be different. Will pull these methods up as
and when we add support for Feed.


> On Sept. 14, 2015, 6:43 p.m., Srikanth Sundarrajan wrote:
> > scheduler/src/main/java/org/apache/falcon/notification/service/FalconNotificationService.java,
line 43
> > <https://reviews.apache.org/r/35724/diff/2/?file=1023478#file1023478line43>
> >
> >     Why is unregister() is on a different entity other than NotificationRequest

Don't want to be keeping the bulky Request object around as many fields aren't really required
for unregister.


> On Sept. 14, 2015, 6:43 p.m., Srikanth Sundarrajan wrote:
> > scheduler/src/main/java/org/apache/falcon/notification/service/impl/JobCompletionService.java,
line 72
> > <https://reviews.apache.org/r/35724/diff/2/?file=1023486#file1023486line72>
> >
> >     If the job is not complete, is the notification request ignored ? Can't see
any polling to check for job completion. Is this dependent on job end notifications ? Are
we expecting that to be resilient to failures ?

This class uses WorkflowJobEndService and listens to notifications from Oozie for job status
change (checked in as part of FALCON-1231). Hence, no polling required.


> On Sept. 14, 2015, 6:43 p.m., Srikanth Sundarrajan wrote:
> > scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java,
line 220
> > <https://reviews.apache.org/r/35724/diff/2/?file=1023487#file1023487line220>
> >
> >     Is there a possiblity of removing running instances when the cache runs out
of space. Am unclear on what the behavior would be

The cache is a loading cache.. so, even if instances are evicted, any attempt to retrieve
them will retrieve it from state store.


> On Sept. 14, 2015, 6:43 p.m., Srikanth Sundarrajan wrote:
> > scheduler/src/main/java/org/apache/falcon/notification/service/ServicesRegistry.java,
line 63
> > <https://reviews.apache.org/r/35724/diff/2/?file=1023479#file1023479line63>
> >
> >     Can the NotificationService be registered with the general Services framework
and accessed through Services::get()

The notification services are already registered with the general Services framework. The
NotificationServicesRegistry is more a convenience class to access just the Notification Services.


- Pallavi


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35724/#review98194
-----------------------------------------------------------


On Oct. 13, 2015, 11:53 a.m., Pallavi Rao wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35724/
> -----------------------------------------------------------
> 
> (Updated Oct. 13, 2015, 11:53 a.m.)
> 
> 
> Review request for Falcon and Srikanth Sundarrajan.
> 
> 
> Bugs: FALCON-1213
>     https://issues.apache.org/jira/browse/FALCON-1213
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> The patch has the basic framework for the scheduler. Each of the individual service needs
to be implemented completely and will be done as separate JIRAs. The intention of the patch
is ensure the base framework satisfies all use cases and get any early feedback in terms of
course correction.
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/falcon/entity/EntityUtil.java 3ab9339 
>   pom.xml 54e6cd1 
>   scheduler/pom.xml PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/exception/DAGEngineException.java PRE-CREATION

>   scheduler/src/main/java/org/apache/falcon/exception/InvalidStateTransitionException.java
PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/exception/ServiceException.java PRE-CREATION

>   scheduler/src/main/java/org/apache/falcon/exception/StateStoreException.java PRE-CREATION

>   scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java PRE-CREATION

>   scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java PRE-CREATION

>   scheduler/src/main/java/org/apache/falcon/execution/FalconExecutionService.java PRE-CREATION

>   scheduler/src/main/java/org/apache/falcon/execution/NotificationHandler.java PRE-CREATION

>   scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java PRE-CREATION

>   scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java PRE-CREATION

>   scheduler/src/main/java/org/apache/falcon/execution/SchedulerUtil.java PRE-CREATION

>   scheduler/src/main/java/org/apache/falcon/notification/service/FalconNotificationService.java
PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/notification/service/NotificationServicesRegistry.java
PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/notification/service/event/DataEvent.java
PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/notification/service/event/Event.java PRE-CREATION

>   scheduler/src/main/java/org/apache/falcon/notification/service/event/JobCompletedEvent.java
PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/notification/service/event/JobScheduledEvent.java
PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/notification/service/event/TimeElapsedEvent.java
PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/notification/service/impl/AlarmService.java
PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/notification/service/impl/DataAvailabilityService.java
PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/notification/service/impl/JobCompletionService.java
PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java
PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/notification/service/request/AlarmRequest.java
PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/notification/service/request/DataNotificationRequest.java
PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/notification/service/request/JobCompletionNotificationRequest.java
PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/notification/service/request/JobScheduleNotificationRequest.java
PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/notification/service/request/NotificationRequest.java
PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/state/EntityState.java PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/state/EntityStateChangeHandler.java PRE-CREATION

>   scheduler/src/main/java/org/apache/falcon/state/ID.java PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/state/InstanceState.java PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java PRE-CREATION

>   scheduler/src/main/java/org/apache/falcon/state/StateMachine.java PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/state/StateService.java PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/state/store/AbstractStateStore.java PRE-CREATION

>   scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java PRE-CREATION

>   scheduler/src/main/java/org/apache/falcon/state/store/StateStore.java PRE-CREATION

>   scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java PRE-CREATION

>   scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngineFactory.java PRE-CREATION

>   scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java
PRE-CREATION 
>   scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java PRE-CREATION

>   scheduler/src/test/java/org/apache/falcon/execution/FalconExecutionServiceTest.java
PRE-CREATION 
>   scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java PRE-CREATION

>   scheduler/src/test/java/org/apache/falcon/execution/SchedulerUtilTest.java PRE-CREATION

>   scheduler/src/test/java/org/apache/falcon/notification/service/AlarmServiceTest.java
PRE-CREATION 
>   scheduler/src/test/java/org/apache/falcon/notification/service/SchedulerServiceTest.java
PRE-CREATION 
>   scheduler/src/test/java/org/apache/falcon/predicate/PredicateTest.java PRE-CREATION

>   scheduler/src/test/java/org/apache/falcon/state/EntityStateServiceTest.java PRE-CREATION

>   scheduler/src/test/java/org/apache/falcon/state/InstanceStateServiceTest.java PRE-CREATION

>   scheduler/src/test/resources/config/cluster/cluster-0.1.xml PRE-CREATION 
>   scheduler/src/test/resources/config/feed/feed-0.1.xml PRE-CREATION 
>   scheduler/src/test/resources/config/process/process-0.1.xml PRE-CREATION 
>   webapp/pom.xml e63aa44 
> 
> Diff: https://reviews.apache.org/r/35724/diff/
> 
> 
> Testing
> -------
> 
> New UTs have been added.
> 
> 
> Thanks,
> 
> Pallavi Rao
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message