Return-Path: X-Original-To: apmail-streams-dev-archive@minotaur.apache.org Delivered-To: apmail-streams-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 408471015F for ; Sun, 16 Nov 2014 23:02:14 +0000 (UTC) Received: (qmail 5763 invoked by uid 500); 16 Nov 2014 23:02:14 -0000 Delivered-To: apmail-streams-dev-archive@streams.apache.org Received: (qmail 5719 invoked by uid 500); 16 Nov 2014 23:02:14 -0000 Mailing-List: contact dev-help@streams.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@streams.incubator.apache.org Delivered-To: mailing list dev@streams.incubator.apache.org Received: (qmail 5707 invoked by uid 99); 16 Nov 2014 23:02:14 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Nov 2014 23:02:14 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Sun, 16 Nov 2014 23:01:51 +0000 Received: (qmail 1971 invoked by uid 99); 16 Nov 2014 23:00:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Nov 2014 23:00:34 +0000 Date: Sun, 16 Nov 2014 23:00:34 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: dev@streams.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (STREAMS-147) Platform-level type conversion MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/STREAMS-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14214125#comment-14214125 ] ASF GitHub Bot commented on STREAMS-147: ---------------------------------------- GitHub user steveblackmon opened a pull request: https://github.com/apache/incubator-streams/pull/131 Streams 68,218 This is a big PR, because STREAMS-147/STREAMS-218 is tackling a big problem: the proliferation of classes and code snippets that perform simple or advanced conversion between typical document types String, ObjectNode, Activity, and pojos generated from schemas matching provider data schemas. The goal is to have platform-level utilities and processors that perform these functions with a 'plug-in' architecture where individual modules can aid those processors in conversion. Thus the responsibility of the module with regard to conversion would be only to define schemas, classify documents that align with those schemas, and provide classes the generic processors can use to convert specific classified documents. A major benefit of this approach are that the code footprint of each module can be smaller - because generic type converters are available and easy to use, components can set preconditions about what type of documents they will accept. Stream pipelines will be able to use a less diverse set of processors to accomplish the same work, so we can focus on making these utility components solid and all modules and streams will benefit. Additionally use cases involving in-bound queues of many types of documents from multiple providers are now simple to implement, whereas without generic converters those use cases would have been hard and ugly. Look forward to your comments and seeing this approach or some variation become the standard. I'm open to changes to the new interfaces or to the converter resolution/instantiation lifecycle. Also please note that these generic classes currently do not support a parallelism hint > 1, due to object serialization errors. Still looking into this and will add tests to ensure they can be parallelized. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/incubator-streams STREAMS-68,218 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-streams/pull/131.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #131 ---- commit aad0e88742b9156e6e11d1bf06d23133ec6ec961 Author: sblackmon Date: 2014-11-13T23:08:21Z omni-bus update primarily addresses STREAMS-218 (Generic Activity Serializer Processor) addresses STREAMS-68 (ActivitySerializer should be ActivityConverter) - was touching related classes anyway some additional refactoring of class and method names is advisable but this is a big chunk resolves STREAMS-219 (.gitignore files generated by maven plugins) needed to knock this out to avoid committing these files along with other changes commit 7a067ce133288fb79e41aed2fcc7ccd389ff1284 Author: sblackmon Date: 2014-11-13T23:35:41Z fixed pig runtime test commit 8737bdadfe595f06dd47b6652d538eec3e3d7f36 Author: sblackmon Date: 2014-11-14T00:15:44Z assessed additional provider modules either refactored for compatibility with streams-converters and/or added comments/todos regarding deprecation etc... commit 93026a7c2110df4ff441d2732f68085ebf933d53 Author: Steve Blackmon Date: 2014-11-15T18:51:11Z improved error handling serialize twitter type support classes commit a24b3be723dbc7ccca7f34cc0f58ae8bb8838470 Author: Steve Blackmon Date: 2014-11-16T00:34:32Z improved configuration scheme commit df648a80d17c36220cba95a77331c1bbdcfd3532 Author: Steve Blackmon Date: 2014-11-16T00:55:54Z Merge branch 'master' into STREAMS-68,218 Conflicts: streams-contrib/streams-provider-facebook/src/main/java/org/apache/streams/facebook/api/FacebookPostActivityConverter.java commit ce2a794dc9c79a28d3650377c58942fb95d82eb2 Author: Steve Blackmon Date: 2014-11-16T00:56:05Z improved configuration scheme commit 3b4135574735359ad819a1486fdd750387662f15 Author: Steve Blackmon Date: 2014-11-16T01:11:41Z fixed mergeMerge branch 'master' into STREAMS-68,218 Conflicts: streams-contrib/streams-provider-facebook/src/main/java/org/apache/streams/facebook/api/FacebookPostActivityConverter.java commit b5cb7eb693be2fd3621c1b62c08f5548dee30dee Author: Steve Blackmon Date: 2014-11-16T21:04:49Z Added code comments ---- > Platform-level type conversion > ------------------------------ > > Key: STREAMS-147 > URL: https://issues.apache.org/jira/browse/STREAMS-147 > Project: Streams > Issue Type: Improvement > Reporter: Steve Blackmon > Assignee: Steve Blackmon > > Investigate / implements mechanism for a module to declare supported inbound/outbound objects. StreamsBuilder should be able to check that types at each hand-off point are the same, and insert a generic processor to convert between jackson-compatible classes where they differ. -- This message was sent by Atlassian JIRA (v6.3.4#6332)