nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Davy De Waele <ddewa...@gmail.com>
Subject Re: Processors on the fly for many sensor devices
Date Sun, 30 Oct 2016 23:26:59 GMT
Hi Andrew,

I've looked at your code and submitted a pull request as I needed to do
some code fixes to get it working
Also posted an issue / question on how you see this GetTCP processor evolve.

We're using Nifi to capture sensor data over TCP so a good GetTCP is
definitely something we need in our Nifi tool-belt

We would like to use it in a slightly different manner as per the current
implementation. We would like to be able to specify a delimiter (byte
sequence) in order to generate flow files out of this processor.
Our sensor devices use a delimiter (end of transmission byte) to mark the
end of a message in the tcp stream.

That way, the GetTCP processor would already "split" the tcp stream into
separate FlowFiles where each FlowFile would correspond to a response
received from the tcp stream. I've seen something similar in the ListenTCP
processor.

Sorry to hear about your loss. My condolences.

Regards,
Davy



On Tue, Oct 18, 2016 at 10:16 AM, Andrew Psaltis <psaltis.andrew@gmail.com>
wrote:

> Davey,
> Sorry for the delay in getting this done, I was away from the keyboard for
> almost 1.5 weeks after a death in the family. I have moved the GetTCP
> processor code to it's own bundle and separate repo [1] with NiFi 1.0.0 as
> the target. I'm not sure if that is the long term home for it. However, it
> does feel a little cleaner then mucking with the NiFi Standard Bundle or me
> making the assumption it would be adopted as part of the standard bundle.
>
> I know of at least one bug in the processor that has been reported -- it
> fails to reconnect if the connection is severed. I will be working on this,
> however, I will not be able to get to it till Wednesday night or most
> likely Thursday morning.
>
> Thanks again for your patience and interest.
>
> Andrew
>
> [1] https://github.com/apsaltis/nifi-gettcp-bundle
>
> On Mon, Oct 3, 2016 at 10:56 AM, Davy De Waele <ddewaele@gmail.com> wrote:
>
>> Hi Andrew,
>>
>> We'd be happy to test and provide feedback.
>> We have a use-case now with Nifi that we're in the process of
>> implementing.
>>
>> If you have sources or binaries feel free to let me know.....
>>
>> Regards,
>> Davy
>>
>> On Fri, Sep 30, 2016 at 9:24 PM, Andrew Psaltis <psaltis.andrew@gmail.com
>> > wrote:
>>
>>> Hi Davy,
>>> Sorry for the slow response, I am traveling today. However, I should be
>>> able to get a build of it for 1.0.0 out later today or tomorrow.
>>>
>>> Thanks,
>>> Andrew
>>>
>>> On Fri, Sep 30, 2016 at 6:54 PM, Davy De Waele <ddewaele@gmail.com>
>>> wrote:
>>>
>>>> Hi Andrew,
>>>>
>>>> We tested your 0.0.6 binary and it seemed to do what we want.
>>>> Unfortunately it is not compatible with the current Nifi version.
>>>> Do you have a binary available that would work with nifi 1.0.0 ?
>>>>
>>>> Thx.
>>>>
>>>> On Wed, Sep 28, 2016 at 11:39 PM, Andrew Psaltis <
>>>> psaltis.andrew@gmail.com> wrote:
>>>>
>>>>> Davy,
>>>>> The processor I have been working on may meet your needs. You are
>>>>> correct, at this time I have not pushed the source for it, still working
>>>>> through some hurdles. The one thing to work out is how you would
>>>>> dynamically add the processors -- suppose you may be able to use the
REST
>>>>> API for NiFi. I would imagine there are quite a number of these devices
>>>>> that you would need to have processors for. In the use case I have been
>>>>> working on, there may be 600 or so endpoints that I need to connect to
and
>>>>> I'm trying to figure out does it make sense to do it this way.
>>>>>
>>>>> I'll hopefully be in a place soon that I can push the code I have for
>>>>> the GetTCP processor.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Sep 29, 2016 at 5:22 AM, Bryan Bende <bbende@gmail.com>
wrote:
>>>>>
>>>>>> Just wanted to clarify something about ListenTCP... it does support
>>>>>> multiple incoming connections, however if you using the batch output
>>>>>> capability, one flow file will contain data across all the connections.
>>>>>>
>>>>>> I do agree with Andrew that based on the description it sounds like
>>>>>> NiFi is expected to be the client that initiates a connection and
keeps
>>>>>> reading for some amount of time/threshold, like a GetTCP processor.
>>>>>>
>>>>>> Currently we have ListenTCP which is waiting for incoming
>>>>>> connections, and PutTCP which makes a connection, but only writes
data.
>>>>>>
>>>>>> On Wed, Sep 28, 2016 at 5:02 PM, Andrew Psaltis <
>>>>>> psaltis.andrew@gmail.com> wrote:
>>>>>>
>>>>>>> Davy,
>>>>>>> It sounds like you need a GetTCP type of processor that connects
>>>>>>> from NiFi to the TCP endpoint on the sensor, is that correct?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Andrew
>>>>>>>
>>>>>>> On Thu, Sep 29, 2016 at 4:46 AM, Davy De Waele <ddewaele@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Thanks for the response ... it's an existing network of sensors.
>>>>>>>> The sensors spit out data over a serial interface that is
exposed over a
>>>>>>>> tcp connection. (rs232 -> ethernet converter in the sensor).
>>>>>>>> The current sensor architecture involves clients making direct
>>>>>>>> connections to the individual sensors. (establishing a tcp
connection to
>>>>>>>> the specific ip of the sensor).
>>>>>>>>
>>>>>>>> If I understand correctly, ListenTCP would not work in this
case
>>>>>>>> for multiple sensors
>>>>>>>>
>>>>>>>> Are you talking about a setup where the sensors would be
in a
>>>>>>>> "client" mode where each sensor would each establish a tcp
connections to a
>>>>>>>> single ListTCP processor  ?
>>>>>>>>
>>>>>>>> Thx
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Sep 28, 2016 at 10:03 PM, Joe Witt <joe.witt@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hello
>>>>>>>>>
>>>>>>>>> Can you talk a bit about why you'd want ListenTCP processors
tied
>>>>>>>>> to a
>>>>>>>>> given sensor?  You should be able to have many sensors
to a single
>>>>>>>>> ListenTCP.  Each stream will be between a source/sensor
and nifi so
>>>>>>>>> data won't be getting intermingled there.  If we're not
providing
>>>>>>>>> enough session/stream metadata on the flow files to make
demux of
>>>>>>>>> the
>>>>>>>>> streams easy using something like RouteOnAttribute or
whatnot we
>>>>>>>>> definitely should.
>>>>>>>>>
>>>>>>>>> Now, that said, you could certainly programmatically
deploy (via
>>>>>>>>> the
>>>>>>>>> REST API) instances of these processors along the lines
of what
>>>>>>>>> your
>>>>>>>>> endpoint registry tells you.  It just seems on the surface
like
>>>>>>>>> doing
>>>>>>>>> so would be avoidable at least for the listening of data.
>>>>>>>>> Typically
>>>>>>>>> such a registry would be useful to do additional
>>>>>>>>> tagging/enrichment of
>>>>>>>>> the data and would occur once it is in the flow.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Joe
>>>>>>>>>
>>>>>>>>> On Wed, Sep 28, 2016 at 3:39 PM, Davy De Waele <ddewaele@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> > We have a large number of sensors that send out
data via TCP.
>>>>>>>>> The idea is to
>>>>>>>>> > use a ListenTCP processor in Nifi to capture the
data, do some
>>>>>>>>> filtering /
>>>>>>>>> > basic transformation before sending it upstream
into our stack.
>>>>>>>>> >
>>>>>>>>> > We can configure individual ListenTCP processors
for each
>>>>>>>>> sensor, and that
>>>>>>>>> > works fine when the number of sensors is small,
but once you hit
>>>>>>>>> a larger
>>>>>>>>> > number if becomes cumbersome and difficult to manage.
>>>>>>>>> >
>>>>>>>>> > We have an inventory of those sensors (exposed via
a REST
>>>>>>>>> service endpoint),
>>>>>>>>> > containing  the sensor tcp information like ip and
port)
>>>>>>>>> >
>>>>>>>>> > Is there an easy way to create these ListenTCP processors
on the
>>>>>>>>> fly based
>>>>>>>>> > on a REST endpoint or some other external configuration
? How
>>>>>>>>> would that
>>>>>>>>> > work ?
>>>>>>>>> >
>>>>>>>>> > Thx.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks,
>>>>>>> Andrew
>>>>>>>
>>>>>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
>>>>>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
>>>>>>> twiiter: @itmdata
>>>>>>> <http://twitter.com/intent/user?screen_name=itmdata>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Andrew
>>>>>
>>>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
>>>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
>>>>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Andrew
>>>
>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
>>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>>>
>>
>>
>
>
> --
> Thanks,
> Andrew
>
> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>

Mime
View raw message