Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 60FCA200D26 for ; Fri, 6 Oct 2017 07:21:38 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 5F5DD160BDC; Fri, 6 Oct 2017 05:21:38 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 32A5D1609E2 for ; Fri, 6 Oct 2017 07:21:36 +0200 (CEST) Received: (qmail 28538 invoked by uid 500); 6 Oct 2017 05:21:35 -0000 Mailing-List: contact commits-help@predictionio.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@predictionio.incubator.apache.org Delivered-To: mailing list commits@predictionio.incubator.apache.org Received: (qmail 28528 invoked by uid 99); 6 Oct 2017 05:21:35 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Oct 2017 05:21:35 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 6D86CC59B4 for ; Fri, 6 Oct 2017 05:21:34 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -3.722 X-Spam-Level: X-Spam-Status: No, score=-3.722 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id SgRCnReZq3sU for ; Fri, 6 Oct 2017 05:21:17 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with SMTP id 216B6611C3 for ; Fri, 6 Oct 2017 05:20:58 +0000 (UTC) Received: (qmail 25757 invoked by uid 99); 6 Oct 2017 05:20:57 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Oct 2017 05:20:57 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 95957F5CD0; Fri, 6 Oct 2017 05:20:56 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: git-site-role@apache.org To: commits@predictionio.incubator.apache.org Date: Fri, 06 Oct 2017 05:21:41 -0000 Message-Id: <8781bac2ff6b4a79be0b489fcbd0720e@git.apache.org> In-Reply-To: <167c9556ba8e45c0916f21f48ca1b242@git.apache.org> References: <167c9556ba8e45c0916f21f48ca1b242@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [47/51] [abbrv] [partial] incubator-predictionio-site git commit: Documentation based on apache/incubator-predictionio#f92d2ac5d39a99565f07f2ffb64f17dfb8ae2d30 archived-at: Fri, 06 Oct 2017 05:21:38 -0000 http://git-wip-us.apache.org/repos/asf/incubator-predictionio-site/blob/4e16ed29/community/contribute-webhook/index.html ---------------------------------------------------------------------- diff --git a/community/contribute-webhook/index.html b/community/contribute-webhook/index.html new file mode 100644 index 0000000..26425b1 --- /dev/null +++ b/community/contribute-webhook/index.html @@ -0,0 +1,255 @@ +Contribute a Webhooks Connector
On this page
Edit this page

Please check out the latest develop branch.

Event server can collect data from other third-party sites or software th rough their webhooks services (for example, SegmentIO, MailChimp). To support that, a Webhooks Connector for the third-party data is needed to be integrated into Event Server. The job of the Webhooks Connector is as simply as converting the third-party data into Event JSON. You can find an example below.

Currently we support two types of connectors: JsonConnector and FormConnector, which is responsible for accepting JSON data and Form-submission data, respectively.

JsonConnector:

1
+2
+3
+4
+5
+6
+7
+8
+9
+10
+11
+12
+13
package org.apache.predictionio.data.webhooks
+
+/** Connector for Webhooks connection */
+private[predictionio] trait JsonConnector {
+
+  /** Convert from original JObject to Event JObject
+    * @param data original JObject recevived through webhooks
+    * @return Event JObject
+   */
+  def toEventJson(data: JObject): JObject
+
+}
+
+

The EventServer URL path to collect webhooks JSON data:

1
http://<EVENT SERVER URL>/webhooks/<CONNECTOR_NAME>.json?accessKey=<YOUR_ACCESS_KEY>&channel=<CHANNEL_NAME>
+

Note that you may collect Webhooks data into default channel (without the channel parameter in the URL) but it's highly recommended to create dedicated Channel to collect specific Webhooks data (e.g. create one channel "segmentio" for SegmentIO and another channel "mailchimp" for Mailchimp data) because it allows you to manage and query data more easily, and the Webhooks data won't be mixed with your other normal app data.

FormConnector:

1
+2
+3
+4
+5
+6
+7
+8
+9
+10
+11
+12
+13
+14
package org.apache.predictionio.data.webhooks
+
+/** Connector for Webhooks connection with Form submission data format
+  */
+private[predictionio] trait FormConnector {
+
+  /** Convert from original Form submission data to Event JObject
+    * @param data Map of key-value pairs in String type received through webhooks
+    * @return Event JObject
+   */
+  def toEventJson(data: Map[String, String]): JObject
+
+}
+
+

The EventServer URL path to collect webhooks form-subimssion data (no .json):

1
http://<EVENT SERVER URL>/webhooks/<CONNECTOR_NAME>?accessKey=<YOUR_ACCESS_KEY>&channel=<CHANNEL_NAME>
+

Note that you may collect Webhooks data into default channel (without the channel parameter in the URL) but it's highly recommended to create dedicated Channel to collect specific Webhooks data (e.g. create one channel "segmentio" for SegmentIO and another channel "mailchimp" for Mailchimp data) because it allows you to manage and query data more easily, and the Webhooks data won't be mixed with your other normal app data.

Example

For example, let's say there is a third-party website (say, it is named "ExampleJson") which can send the following JSON data through its webhooks service and we would like to collect it into Event Store.

UserActionItem:

1
+2
+3
+4
+5
+6
+7
+8
+9
+10
+11
+12
+13
+14
{
+  "type": "userActionItem",
+  "userId": "as34smg4",
+  "event": "do_something_on",
+  "itemId": "kfjd312bc",
+  "context": {
+    "ip": "1.23.4.56",
+    "prop1": 2.345,
+    "prop2": "value1"
+  },
+  "anotherPropertyA": 4.567,
+  "anotherPropertyB": false,
+  "timestamp": "2015-01-15T04:20:23.567Z"
+}
+

1. Implement Webhooks Connector

Because the data sent by this third-party "ExampleJson" site is in JSON format, we implement an object ExampleJsonConnector which extends JsonConnector:

1
+2
+3
+4
+5
+6
+7
+8
+9
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19
+20
+21
+22
+23
+24
+25
+26
+27
+28
+29
+30
+31
+32
+33
+34
+35
+36
+37
+38
+39
+40
+41
+42
+43
+44
+45
+46
+47
+48
+49
+50
+51
+52
+53
+54
+55
+56
+57
+58
+59
+60
+61
+62
+63
+64
+65
+66
private[predictionio] object ExampleJsonConnector extends JsonConnector {
+
+  implicit val json4sFormats: Formats = DefaultFormats
+
+  override def toEventJson(data: JObject): JObject = {
+    val common = try {
+      data.extract[Common]
+    } catch {
+      case e: Exception => throw new ConnectorException(
+        s"Cannot extract Common field from ${data}. ${e.getMessage()}", e)
+    }
+
+    val json = try {
+      common.`type` match {
+        case "userActionItem" =>
+          toEventJson(common = common, userActionItem = data.extract[UserActionItem])
+        case x: String =>
+          throw new ConnectorException(
+            s"Cannot convert unknown type '${x}' to Event JSON.")
+      }
+    } catch {
+      case e: ConnectorException => throw e
+      case e: Exception => throw new ConnectorException(
+        s"Cannot convert ${data} to eventJson. ${e.getMessage()}", e)
+    }
+
+    json
+  }
+
+  // Convert the UserActionItem JSON to Event JSON
+  def toEventJson(common: Common, userActionItem: UserActionItem): JObject = {
+    import org.json4s.JsonDSL._
+
+    // map to EventAPI JSON
+    val json =
+      ("event" -> userActionItem.event) ~
+      ("entityType" -> "user") ~
+      ("entityId" -> userActionItem.userId) ~
+      ("targetEntityType" -> "item") ~
+      ("targetEntityId" -> userActionItem.itemId) ~
+      ("eventTime" -> userActionItem.timestamp) ~
+      ("properties" -> (
+        ("context" -> userActionItem.context) ~
+        ("anotherPropertyA" -> userActionItem.anotherPropertyA) ~
+        ("anotherPropertyB" -> userActionItem.anotherPropertyB)
+      ))
+    json
+  }
+
+  // Common required fields
+  case class Common(
+    `type`: String
+  )
+
+  // UserActionItem fields
+  case class UserActionItem (
+    userId: String,
+    event: String,
+    itemId: String,
+    context: JObject,
+    anotherPropertyA: Option[Double],
+    anotherPropertyB: Option[Boolean],
+    timestamp: String
+  )
+
+}
+

You can find the complete example in the GitHub repo and how to write tests for the connector.

Please put the connector code in a separate directory for each site. For example, code for segmentio connector should be in

1
data/src/main/scala/org.apache.predictionio/data/webhooks/segmentio/
+

and tests should be in

1
data/src/test/scala/org.apache.predictionio/data/webhooks/segmentio/
+

For form-submission data, you can find the comple example the GitHub repo and how to write tests for the connector.

2. Integrate the Connector into Event Server

Once we have the connector implemented, we can add this to the EventServer so we can collect real-time data.

Add the connector to WebhooksConnectors object:

1
+2
+3
+4
+5
+6
+7
+8
+9
+10
+11
+12
+13
+14
+15
+16
+17
+import org.apache.predictionio.data.webhooks.examplejson.ExampleJsonConnector // ADDED
+
+private[predictionio] object WebhooksConnectors {
+
+  // Map of Connector Name to Connector
+  val json: Map[String, JsonConnector] = Map(
+    "segmentio" -> SegmentIOConnector,
+    "examplejson" -> ExampleJsonConnector // ADDED
+  )
+
+  // Map of Connector Name to Connector
+  val form: Map[String, FormConnector] = Map(
+    "mailchimp" -> MailChimpConnector
+  )
+
+}
+

Note that the name of the connectors (e.g. "examplejson", "segmentio") will be used as the webhooks URL. In this example, the event server URL to collect data from "ExampleJson" would be:

1
http://<EVENT SERVER URL>/webhooks/examplejson.json?accessKey=<YOUR_ACCESS_KEY>&channel=<CHANNEL_NAME>
+

For FormConnector, the URL doesn't have .json. For example,

1
http://<EVENT SERVER URL>/webhooks/mailchimp?accessKey=<YOUR_ACCESS_KEY>&channel=<CHANNEL_NAME>
+

That's it. Once you re-compile Apache PredictionIO (incubating), you can send the ExampleJson data to the following URL and the data will be stored to the App of the corresponding Access Key.

Community

Contribute

Apache IncubatorApache PredictionIO is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making p rocess have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
\ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-predictionio-site/blob/4e16ed29/community/contribute-webhook/index.html.gz ---------------------------------------------------------------------- diff --git a/community/contribute-webhook/index.html.gz b/community/contribute-webhook/index.html.gz new file mode 100644 index 0000000..583002e Binary files /dev/null and b/community/contribute-webhook/index.html.gz differ http://git-wip-us.apache.org/repos/asf/incubator-predictionio-site/blob/4e16ed29/community/index.html ---------------------------------------------------------------------- diff --git a/community/index.html b/community/index.html new file mode 100644 index 0000000..17efa5f --- /dev/null +++ b/community/index.html @@ -0,0 +1,6 @@ +Community Page

PredictionIO Docs

Community Page

On this page
Edit this page

User Mailing List

This list is for users of Apache PredictionIO (incubating) to ask questions, share knowledge, and discuss issues. Do send mail to this list with usage and configuration questions and problems. Also, please send questions to this list to verify your problem before filing issues in JIRA.

Subscribe to our User Mailing List. Unsubscribe from our User Mailing List.

Twitter

Follow us on Twitter @predictionio.

Facebook Page

Like us on Facebook at https://www.facebook.com/predictionio.

GitHub

View our code on GitHub at https://github.com/apache/incubator-predictionio.

Community

Contribute

Apache IncubatorApache PredictionIO is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
\ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-predictionio-site/blob/4e16ed29/community/index.html.gz ---------------------------------------------------------------------- diff --git a/community/index.html.gz b/community/index.html.gz new file mode 100644 index 0000000..ed927bd Binary files /dev/null and b/community/index.html.gz differ