nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mr TheSegfault (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (MINIFICPP-954) Consider moving the C2 line protocol to Google Protobuf
Date Fri, 26 Jul 2019 17:05:00 GMT

    [ https://issues.apache.org/jira/browse/MINIFICPP-954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893988#comment-16893988
] 

Mr TheSegfault edited comment on MINIFICPP-954 at 7/26/19 5:04 PM:
-------------------------------------------------------------------

[~bakaid]  I like some of the concepts you present but prior CVEs were why we moved away
from protobuf initially in community discussions.  I think offloading the testing to protobuf
maintainers is a reason to have Protobuf but I think both will need to exist.  We also considered
flatbuffers and capn proto – would love to see that added to this discussion.

 

I think this ultimately would be either adding protobuf/flatbuffers/capn as a selectable line
format. We have a very simple line format that isn't going to change ( this is because our
line formats are designed to use the RESTFul API for the complicated stuff ). The line format
is simply for sending a very tiny subset of information. We also want to use the same code
for nanofi, where I think our consumers are more likely to appreciate not including protobuf.
As a result we can add this as an extension ( in fact, I love the idea ) – especially since
we can mitigate concerns of others by having it be configurable in the agent ( and compile
time for nanofi ECUs ).

If we went down the route of cap'n proto we can rely on the embedded RPC and use that as a
separate C2 protocol. I have used Cap'n Proto, for example, and we could easily define those
RPC calls to lessen the load,  but I would likely change this ticket to exploring alternative
protocols/wire formats and then making a decision as an addition to what we have now as opposed
to a replacement.

We will be forced to maintain it but we have no choice either as we have users who specifically
don't want protobuf and we have released it so we are forced to maintain it and improve its
testing.

Keep in mind that whomever tackles this will need to balance maintaining what we already have
( not replacing ) so we then increase what we must test. We can rely on capn proto or protobuf's
tests, but then we still need to perform our test in the venn diagram of testing. Would that
effort be better spent improving tests for our work since we already need to do that and must
maintain it?

I'll leave that question up to the implementor – but it's certainly not an affront against
doing this work, more a word of caution. I'm glad we decided against protobuf at the time,
though – would have made that decision again. It was a calculated risk with many thoughts
in mind – but would love to see something else like protobuf/flatbuffers/capn .


Aside from this comment I would love the community to weigh in and make a choice. I've made
my points known and have no intention of implementing this myself due to other technical debt
I feel is more important so thoughts on what to do are up to the implementation seeker.


was (Author: phrocker):
[~bakaid]  I like some of the concepts you present but prior CVEs were why we moved away
from protobuf initially in community discussions.  I think offloading the testing to protobuf
maintainers is a reason to have Protobuf but I think both will need to exist.  We also considered
flatbuffers and capn proto.

 

I think this ultimately would be either adding protobuf/flatbuffers/capn as a selectable line
format. We have a very simple line format that isn't going to change ( this is because our
line formats are designed to use the RESTFul API for the complicated stuff ). The line format
is simply for sending a very tiny subset of information. We also want to use the same code
for nanofi, where I think our consumers are more likely to appreciate not including protobuf.
As a result we can add this as an extension ( in fact, I love the idea ) – especially since
we can mitigate concerns of others by having it be configurable in the agent ( and compile
time for nanofi ECUs ). 

If we went down the route of cap'n proto we can rely on the embedded RPC and use that as a
separate C2 protocol. I have used Cap'n Proto, for example, and we could easily define those
RPC calls to lessen the load,  but I would likely change this ticket to exploring alternative
protocols/wire formats and then making a decision as an addition to what we have now as opposed
to a replacement.

We will be forced to maintain it but we have no choice either as we have users who specifically
don't want protobuf and we have released it so we are forced to maintain it and improve its
testing. 

Keep in mind that whomever tackles this will need to balance maintaining what we already have
( not replacing ) so we then increase what we must test. We can rely on capn proto or protobuf's
tests, but then we still need to perform our test in the venn diagram of testing. Would that
effort be better spent improving tests for our work since we already need to do that and must
maintain it?

I'll leave that question up to the implementor – but it's certainly not an affront against
doing this work, more a word of caution. I'm glad we decided against protobuf at the time,
though – would have made that decision again. It was a calculated risk with many thoughts
in mind – but would love to see something else like protobuf/flatbuffers/capn .

 

 

> Consider moving the C2 line protocol to Google Protobuf
> -------------------------------------------------------
>
>                 Key: MINIFICPP-954
>                 URL: https://issues.apache.org/jira/browse/MINIFICPP-954
>             Project: Apache NiFi MiNiFi C++
>          Issue Type: Improvement
>            Reporter: Daniel Bakai
>            Priority: Major
>
> * Very compact serialized format
>  * Easy protocol description
>  * Support for generating parsers for many languages from the proto file
>  * Has good versioning support
>  * Thoroughly tested, fuzzed
>  * Less chance of introducing security vulnerabilities with hand-written parsers



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Mime
View raw message