falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srikanth Sundarrajan <srik...@hotmail.com>
Subject RE: Falcon with Spark ?
Date Tue, 03 Feb 2015 17:30:21 GMT
Yes support for Spark was specifically added through https://issues.apache.org/jira/browse/OOZIE-1983,
to allow users of Oozie or Falcon to run Spark jobs. Moving the retention job to Spark would
create a heavy dependency on spark within Falcon. 

With https://issues.apache.org/jira/browse/FALCON-965, it should be possible to create an
alternate implementation of eviction.

Regards
Srikanth Sundarrajan

> Date: Tue, 3 Feb 2015 15:51:38 +0100
> From: jb@nanthrax.net
> To: dev@falcon.incubator.apache.org
> Subject: Falcon with Spark ?
> 
> Hi all,
> 
> I'm working (and finally resuming my work ;)) on some Falcon features:
> - Update and improvements on the ActiveMQ broker
> - Complete CDC support of diff/gap storage
> - Support of more workflow entities (mapreduce directly instead of Oozie 
> workflow.xml, etc)
> 
> For the workflow entities, I would like to evaluate the "direct" support 
> of Spark.
> Generally speaking, I wonder if "oppositionally", we couldn't leverage 
> Spark for some internal Falcon processes (like eviction, etc).
> 
> WDYT ?
> 
> Regards
> JB
> -- 
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message