streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (STREAMS-147) Platform-level type conversion
Date Sun, 16 Nov 2014 23:00:34 GMT


ASF GitHub Bot commented on STREAMS-147:

GitHub user steveblackmon opened a pull request:

    Streams 68,218

    This is a big PR, because STREAMS-147/STREAMS-218 is tackling a big problem: the proliferation
of classes and code snippets that perform simple or advanced conversion between typical document
types String, ObjectNode, Activity, and pojos generated from schemas matching provider data
    The goal is to have platform-level utilities and processors that perform these functions
with a 'plug-in' architecture where individual modules can aid those processors in conversion.
 Thus the responsibility of the module with regard to conversion would be only to define schemas,
classify documents that align with those schemas, and provide classes the generic processors
can use to convert specific classified documents.
    A major benefit of this approach are that the code footprint of each module can be smaller
- because generic type converters are available and easy to use, components can set preconditions
about what type of documents they will accept.  Stream pipelines will be able to use a less
diverse set of processors to accomplish the same work, so we can focus on making these utility
components solid and all modules and streams will benefit.
    Additionally use cases involving in-bound queues of many types of documents from multiple
providers are now simple to implement, whereas without generic converters those use cases
would have been hard and ugly.
    Look forward to your comments and seeing this approach or some variation become the standard.
 I'm open to changes to the new interfaces or to the converter resolution/instantiation lifecycle.
 Also please note that these generic classes currently do not support a parallelism hint >
1, due to object serialization errors.  Still looking into this and will add tests to ensure
they can be parallelized.

You can merge this pull request into a Git repository by running:

    $ git pull STREAMS-68,218

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #131
commit aad0e88742b9156e6e11d1bf06d23133ec6ec961
Author: sblackmon <>
Date:   2014-11-13T23:08:21Z

    omni-bus update
    primarily addresses STREAMS-218 (Generic Activity Serializer Processor)
    addresses STREAMS-68 (ActivitySerializer should be ActivityConverter) - was touching related
classes anyway
    some additional refactoring of class and method names is advisable but this is a big chunk
    resolves STREAMS-219 (.gitignore files generated by maven plugins)
    needed to knock this out to avoid committing these files along with other changes

commit 7a067ce133288fb79e41aed2fcc7ccd389ff1284
Author: sblackmon <>
Date:   2014-11-13T23:35:41Z

    fixed pig runtime test

commit 8737bdadfe595f06dd47b6652d538eec3e3d7f36
Author: sblackmon <>
Date:   2014-11-14T00:15:44Z

    assessed additional provider modules
    either refactored for compatibility with streams-converters
    and/or added comments/todos regarding deprecation etc...

commit 93026a7c2110df4ff441d2732f68085ebf933d53
Author: Steve Blackmon <>
Date:   2014-11-15T18:51:11Z

    improved error handling
    serialize twitter type support classes

commit a24b3be723dbc7ccca7f34cc0f58ae8bb8838470
Author: Steve Blackmon <>
Date:   2014-11-16T00:34:32Z

    improved configuration scheme

commit df648a80d17c36220cba95a77331c1bbdcfd3532
Author: Steve Blackmon <>
Date:   2014-11-16T00:55:54Z

    Merge branch 'master' into STREAMS-68,218

commit ce2a794dc9c79a28d3650377c58942fb95d82eb2
Author: Steve Blackmon <>
Date:   2014-11-16T00:56:05Z

    improved configuration scheme

commit 3b4135574735359ad819a1486fdd750387662f15
Author: Steve Blackmon <>
Date:   2014-11-16T01:11:41Z

    fixed mergeMerge branch 'master' into STREAMS-68,218

commit b5cb7eb693be2fd3621c1b62c08f5548dee30dee
Author: Steve Blackmon <>
Date:   2014-11-16T21:04:49Z

    Added code comments


> Platform-level type conversion
> ------------------------------
>                 Key: STREAMS-147
>                 URL:
>             Project: Streams
>          Issue Type: Improvement
>            Reporter: Steve Blackmon
>            Assignee: Steve Blackmon
> Investigate / implements mechanism for a module to declare supported inbound/outbound
objects.  StreamsBuilder should be able to check that types at each hand-off point are the
same, and insert a generic processor to convert between jackson-compatible classes where they

This message was sent by Atlassian JIRA

View raw message