nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Psaltis <psaltis.and...@gmail.com>
Subject Re: Processors on the fly for many sensor devices
Date Tue, 01 Nov 2016 10:57:29 GMT
Good morning Davy,
I have all (barring tests that fail) code in place to fulfill NIFI-2615
[1]. I plan on wrapping up the testing of it today and submitting a P/R.
One thing to call out is that at present it is it's own module. However, I
plan on adding it to the NiFi Standard Processor bundle before submitting
the P/R. Let me know if you want me to add in the processing for the
demarcator splitting.

Thanks,
Andrew

On Mon, Oct 31, 2016 at 4:37 PM, Andrew Psaltis <psaltis.andrew@gmail.com>
wrote:

> Hi Davy,
> Thanks for the P/R - I merged that in just a little bit ago.
>
> I think the feature you are describing makes total sense, I'm sure this
> would be useful in many different use cases. I am in the process of
> wrapping up the changes to fulfill JIRA NIFI-2615 [1]. Do you want to add
> the "delimiter based spliter" feature to that JIRA? Are you planning on
> implementing it or are you asking if I would?
>
> I should have the updated bits up in GH this evening/very early am.
>
> Thanks your kinda words regarding our loss.
>
> Thanks,
> Andrew
>
> On Sun, Oct 30, 2016 at 7:26 PM, Davy De Waele <ddewaele@gmail.com> wrote:
>
>> Hi Andrew,
>>
>> I've looked at your code and submitted a pull request as I needed to do
>> some code fixes to get it working
>> Also posted an issue / question on how you see this GetTCP processor
>> evolve.
>>
>> We're using Nifi to capture sensor data over TCP so a good GetTCP is
>> definitely something we need in our Nifi tool-belt
>>
>> We would like to use it in a slightly different manner as per the current
>> implementation. We would like to be able to specify a delimiter (byte
>> sequence) in order to generate flow files out of this processor.
>> Our sensor devices use a delimiter (end of transmission byte) to mark the
>> end of a message in the tcp stream.
>>
>> That way, the GetTCP processor would already "split" the tcp stream into
>> separate FlowFiles where each FlowFile would correspond to a response
>> received from the tcp stream. I've seen something similar in the ListenTCP
>> processor.
>>
>> Sorry to hear about your loss. My condolences.
>>
>> Regards,
>> Davy
>>
>>
>>
>> On Tue, Oct 18, 2016 at 10:16 AM, Andrew Psaltis <
>> psaltis.andrew@gmail.com> wrote:
>>
>>> Davey,
>>> Sorry for the delay in getting this done, I was away from the keyboard
>>> for almost 1.5 weeks after a death in the family. I have moved the GetTCP
>>> processor code to it's own bundle and separate repo [1] with NiFi 1.0.0 as
>>> the target. I'm not sure if that is the long term home for it. However, it
>>> does feel a little cleaner then mucking with the NiFi Standard Bundle or me
>>> making the assumption it would be adopted as part of the standard bundle.
>>>
>>> I know of at least one bug in the processor that has been reported -- it
>>> fails to reconnect if the connection is severed. I will be working on this,
>>> however, I will not be able to get to it till Wednesday night or most
>>> likely Thursday morning.
>>>
>>> Thanks again for your patience and interest.
>>>
>>> Andrew
>>>
>>> [1] https://github.com/apsaltis/nifi-gettcp-bundle
>>>
>>> On Mon, Oct 3, 2016 at 10:56 AM, Davy De Waele <ddewaele@gmail.com>
>>> wrote:
>>>
>>>> Hi Andrew,
>>>>
>>>> We'd be happy to test and provide feedback.
>>>> We have a use-case now with Nifi that we're in the process of
>>>> implementing.
>>>>
>>>> If you have sources or binaries feel free to let me know.....
>>>>
>>>> Regards,
>>>> Davy
>>>>
>>>> On Fri, Sep 30, 2016 at 9:24 PM, Andrew Psaltis <
>>>> psaltis.andrew@gmail.com> wrote:
>>>>
>>>>> Hi Davy,
>>>>> Sorry for the slow response, I am traveling today. However, I should
>>>>> be able to get a build of it for 1.0.0 out later today or tomorrow.
>>>>>
>>>>> Thanks,
>>>>> Andrew
>>>>>
>>>>> On Fri, Sep 30, 2016 at 6:54 PM, Davy De Waele <ddewaele@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Andrew,
>>>>>>
>>>>>> We tested your 0.0.6 binary and it seemed to do what we want.
>>>>>> Unfortunately it is not compatible with the current Nifi version.
>>>>>> Do you have a binary available that would work with nifi 1.0.0 ?
>>>>>>
>>>>>> Thx.
>>>>>>
>>>>>> On Wed, Sep 28, 2016 at 11:39 PM, Andrew Psaltis <
>>>>>> psaltis.andrew@gmail.com> wrote:
>>>>>>
>>>>>>> Davy,
>>>>>>> The processor I have been working on may meet your needs. You
are
>>>>>>> correct, at this time I have not pushed the source for it, still
working
>>>>>>> through some hurdles. The one thing to work out is how you would
>>>>>>> dynamically add the processors -- suppose you may be able to
use the REST
>>>>>>> API for NiFi. I would imagine there are quite a number of these
devices
>>>>>>> that you would need to have processors for. In the use case I
have been
>>>>>>> working on, there may be 600 or so endpoints that I need to connect
to and
>>>>>>> I'm trying to figure out does it make sense to do it this way.
>>>>>>>
>>>>>>> I'll hopefully be in a place soon that I can push the code I
have
>>>>>>> for the GetTCP processor.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Sep 29, 2016 at 5:22 AM, Bryan Bende <bbende@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Just wanted to clarify something about ListenTCP... it does
support
>>>>>>>> multiple incoming connections, however if you using the batch
output
>>>>>>>> capability, one flow file will contain data across all the
connections.
>>>>>>>>
>>>>>>>> I do agree with Andrew that based on the description it sounds
like
>>>>>>>> NiFi is expected to be the client that initiates a connection
and keeps
>>>>>>>> reading for some amount of time/threshold, like a GetTCP
processor.
>>>>>>>>
>>>>>>>> Currently we have ListenTCP which is waiting for incoming
>>>>>>>> connections, and PutTCP which makes a connection, but only
writes data.
>>>>>>>>
>>>>>>>> On Wed, Sep 28, 2016 at 5:02 PM, Andrew Psaltis <
>>>>>>>> psaltis.andrew@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Davy,
>>>>>>>>> It sounds like you need a GetTCP type of processor that
connects
>>>>>>>>> from NiFi to the TCP endpoint on the sensor, is that
correct?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Andrew
>>>>>>>>>
>>>>>>>>> On Thu, Sep 29, 2016 at 4:46 AM, Davy De Waele <ddewaele@gmail.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Thanks for the response ... it's an existing network
of sensors.
>>>>>>>>>> The sensors spit out data over a serial interface
that is exposed over a
>>>>>>>>>> tcp connection. (rs232 -> ethernet converter in
the sensor).
>>>>>>>>>> The current sensor architecture involves clients
making direct
>>>>>>>>>> connections to the individual sensors. (establishing
a tcp connection to
>>>>>>>>>> the specific ip of the sensor).
>>>>>>>>>>
>>>>>>>>>> If I understand correctly, ListenTCP would not work
in this case
>>>>>>>>>> for multiple sensors
>>>>>>>>>>
>>>>>>>>>> Are you talking about a setup where the sensors would
be in a
>>>>>>>>>> "client" mode where each sensor would each establish
a tcp connections to a
>>>>>>>>>> single ListTCP processor  ?
>>>>>>>>>>
>>>>>>>>>> Thx
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Sep 28, 2016 at 10:03 PM, Joe Witt <joe.witt@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello
>>>>>>>>>>>
>>>>>>>>>>> Can you talk a bit about why you'd want ListenTCP
processors
>>>>>>>>>>> tied to a
>>>>>>>>>>> given sensor?  You should be able to have many
sensors to a
>>>>>>>>>>> single
>>>>>>>>>>> ListenTCP.  Each stream will be between a source/sensor
and nifi
>>>>>>>>>>> so
>>>>>>>>>>> data won't be getting intermingled there.  If
we're not providing
>>>>>>>>>>> enough session/stream metadata on the flow files
to make demux
>>>>>>>>>>> of the
>>>>>>>>>>> streams easy using something like RouteOnAttribute
or whatnot we
>>>>>>>>>>> definitely should.
>>>>>>>>>>>
>>>>>>>>>>> Now, that said, you could certainly programmatically
deploy (via
>>>>>>>>>>> the
>>>>>>>>>>> REST API) instances of these processors along
the lines of what
>>>>>>>>>>> your
>>>>>>>>>>> endpoint registry tells you.  It just seems on
the surface like
>>>>>>>>>>> doing
>>>>>>>>>>> so would be avoidable at least for the listening
of data.
>>>>>>>>>>> Typically
>>>>>>>>>>> such a registry would be useful to do additional
>>>>>>>>>>> tagging/enrichment of
>>>>>>>>>>> the data and would occur once it is in the flow.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Joe
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Sep 28, 2016 at 3:39 PM, Davy De Waele
<
>>>>>>>>>>> ddewaele@gmail.com> wrote:
>>>>>>>>>>> > We have a large number of sensors that send
out data via TCP.
>>>>>>>>>>> The idea is to
>>>>>>>>>>> > use a ListenTCP processor in Nifi to capture
the data, do some
>>>>>>>>>>> filtering /
>>>>>>>>>>> > basic transformation before sending it upstream
into our stack.
>>>>>>>>>>> >
>>>>>>>>>>> > We can configure individual ListenTCP processors
for each
>>>>>>>>>>> sensor, and that
>>>>>>>>>>> > works fine when the number of sensors is
small, but once you
>>>>>>>>>>> hit a larger
>>>>>>>>>>> > number if becomes cumbersome and difficult
to manage.
>>>>>>>>>>> >
>>>>>>>>>>> > We have an inventory of those sensors (exposed
via a REST
>>>>>>>>>>> service endpoint),
>>>>>>>>>>> > containing  the sensor tcp information like
ip and port)
>>>>>>>>>>> >
>>>>>>>>>>> > Is there an easy way to create these ListenTCP
processors on
>>>>>>>>>>> the fly based
>>>>>>>>>>> > on a REST endpoint or some other external
configuration ? How
>>>>>>>>>>> would that
>>>>>>>>>>> > work ?
>>>>>>>>>>> >
>>>>>>>>>>> > Thx.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Thanks,
>>>>>>>>> Andrew
>>>>>>>>>
>>>>>>>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
>>>>>>>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
>>>>>>>>> twiiter: @itmdata
>>>>>>>>> <http://twitter.com/intent/user?screen_name=itmdata>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks,
>>>>>>> Andrew
>>>>>>>
>>>>>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
>>>>>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
>>>>>>> twiiter: @itmdata
>>>>>>> <http://twitter.com/intent/user?screen_name=itmdata>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Andrew
>>>>>
>>>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
>>>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
>>>>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Andrew
>>>
>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
>>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>>>
>>
>>
>
>
> --
> Thanks,
> Andrew
>
> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>



-- 
Thanks,
Andrew

Subscribe to my book: Streaming Data <http://manning.com/psaltis>
<https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>

Mime
View raw message