Return-Path: 
 <issues-return-29235-apmail-camel-issues-archive=camel.apache.org@camel.apache.org>
X-Original-To: apmail-camel-issues-archive@minotaur.apache.org
Delivered-To: apmail-camel-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 95E69180C4
	for <apmail-camel-issues-archive@minotaur.apache.org>;
 Wed,  2 Dec 2015 10:32:17 +0000 (UTC)
Received: (qmail 93816 invoked by uid 500); 2 Dec 2015 10:32:11 -0000
Delivered-To: apmail-camel-issues-archive@camel.apache.org
Received: (qmail 93492 invoked by uid 500); 2 Dec 2015 10:32:11 -0000
Mailing-List: contact issues-help@camel.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:issues-help@camel.apache.org>
List-Unsubscribe: <mailto:issues-unsubscribe@camel.apache.org>
List-Post: <mailto:issues@camel.apache.org>
List-Id: <issues.camel.apache.org>
Reply-To: dev@camel.apache.org
Delivered-To: mailing list issues@camel.apache.org
Received: (qmail 93477 invoked by uid 99); 2 Dec 2015 10:32:11 -0000
Received: from arcas.apache.org (HELO arcas) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Dec 2015 10:32:11 +0000
Received: from arcas.apache.org (localhost [127.0.0.1])
	by arcas (Postfix) with ESMTP id F3BE72C1F6C
	for <issues@camel.apache.org>; Wed,  2 Dec 2015 10:32:10 +0000 (UTC)
Date: Wed, 2 Dec 2015 10:32:10 +0000 (UTC)
From: "Henryk Konsek (JIRA)" <jira@apache.org>
To: issues@camel.apache.org
Message-ID: <JIRA.12917538.1449052307000.244001.1449052330995@Atlassian.JIRA>
In-Reply-To: <JIRA.12917538.1449052307000@Atlassian.JIRA>
References: <JIRA.12917538.1449052307000@Atlassian.JIRA>
 <JIRA.12917538.1449052307070@arcas>
Subject: [jira] [Created] (CAMEL-9385) Create Apache Spark component
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Henryk Konsek created CAMEL-9385:
------------------------------------

             Summary: Create Apache Spark component
                 Key: CAMEL-9385
                 URL: https://issues.apache.org/jira/browse/CAMEL-9385
             Project: Camel
          Issue Type: New Feature
            Reporter: Henryk Konsek
            Assignee: Henryk Konsek
             Fix For: 2.17.0


As a part of the the IoT project I'm working on, I have created a Spark component (1) to make it easier to handle analytics requests from devices. I would like to donate this code to the ASF Camel and extend it here, as I guess that there would be many people interested in using Spark from Camel.

The URI looks like {{spark:rdd/rddName/rddCallback}} or {{spark:dataframe/frameName/frameCallback}} depending if you would like to work with RDDs or DataFrames.

The idea here is that Camel route acts as a driver application. You specify RDD/DataFrames definitions (and callbacks to act against those) in a registry (for example as Spring beans or OSGi services). Then you send a parameters for the computations as a body of a message.

For example in Spring Boot you specify RDD+callback as:

{code}
@Bean
JavaRDD myRdd(SparkContext sparkContext) {
  return sparkContext.textFile("foo.txt");
}

@Bean
class MyAnalytics {

  @RddCallback
  long countLines(JavaRDD<String> textFile, long argument) {
     return rdd.count() * argument;
  }

}
{code}

Then you ask for the results of computations:

{code}
long results = producerTemplate.requestBody("spark:rdd/myRdd/MyAnalytics", 10, long.class);
{code}

Such setup is extremely useful for bridging Spark computations via different transports.

(1) https://github.com/rhiot/rhiot/tree/master/components/camel-spark



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)