nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jvwing <>
Subject [GitHub] nifi pull request: Nifi 1540 - AWS Kinesis Get and Put Processors
Date Fri, 27 May 2016 04:29:22 GMT
Github user jvwing commented on the pull request:
    Your suggestion about the plain old SDK Kinesis API is interesting, but I guess I don't
recommend it.  Using the plain SDK would let NiFi handle the multithreading, retries, batching,
etc.  It would be a win for the NiFi model.  But then we would also have to pick up at least
some of the KCL's other features, like balancing traffic across shards, or leave that as an
excercise for the user.  It sounds complicated and hard, and I'm lazy.  Also, the KCL is recommended
by Amazon, as you pointed out earlier, and not using it might be a point of confusion as to
why we didn't do it the "right" way, especially if a new version of the KCL is released with
features we hadn't thought of.  So even though the KCL is an awkward fit in NiFi, as long
as it is an opt-in feature, it seems like a good addition to NiFi's AWS interoperability story.
    Part of why I would prefer to not change the base AWS processor classes is to keep it
opt-in.  I'm not sure how the other AWS Get* processors would benefit from the Kinesis-like
threading model, they do not have applications or threads to drive activity outside of NiFi's
scheduling.  But I can see lobbying for a change to `AbstractProcessor` to remove `onTrigger`'s
`final` modifier for cases like these without divorcing the class hierarchies.
    A recently merged [change in the 0.x branch](
for NIFI-786 has some shared credential property descriptors and validation logic that you
may be able to use, that will hopefully make it easier to implement separate base classes.
 Let me know if I can help with that.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

View raw message