nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Carenza <nick.care...@thecontrolgroup.com>
Subject Re: Problems with ListenBeats 1.3.0
Date Thu, 24 Aug 2017 19:08:27 GMT
It seemed to hold up a little bit better but I ended up getting the same
errors.

On Thu, Aug 24, 2017 at 11:35 AM, Nick Carenza <
nick.carenza@thecontrolgroup.com> wrote:

> Okay, thank you Andre!
>
> - Nick
>
> On Wed, Aug 23, 2017 at 4:03 PM, Andre <andre-lists@fucs.org> wrote:
>
>> Nick,
>>
>> This is a bug (NIFI-3753). I am also able to confirm there is a
>> workaround: just change the compression settings to a lower compression
>> level (I believe anything between none and 3 should work) and/or reduce the
>> number of messages batched on a single beats payload.
>>
>> Cheers
>>
>> On 23 Aug 2017 09:11, "Nick Carenza" <nick.carenza@thecontrolgroup.com>
>> wrote:
>>
>>> I am trying to cut Logstash out of my stack because i have moved all of
>>> its functionality into Nifi but I haven't been able to get Filebeat to play
>>> nicely with the ListenBeats processor so I have to keep  Logstash around
>>> just to forward messages to Nifi.
>>>
>>> I am using latest stable 5.5.2 but the ListenBeats processor throws
>>> these errors and gets yielded for 30s in an endless loop.
>>>
>>> 2017-08-22 22:26:53,183 ERROR [pool-812781-thread-29]
>>>> o.a.nifi.processors.beats.ListenBeats ListenBeats[id=9e14179f-101f-1159-4cf1-738e9a425a6e]
>>>> Error reading Beats frames due to Error decoding Beats frame: Error
>>>> decompressing  frame: Unexpected end of ZLIB input stream:
>>>> org.apache.nifi.processors.beats.frame.BeatsFrameException: Error
>>>> decoding Beats frame: Error decompressing  frame: Unexpected end of ZLIB
>>>> input stream
>>>>
>>>> org.apache.nifi.processors.beats.frame.BeatsFrameException: Error
>>>> decoding Beats frame: Error decompressing  frame: Unexpected end of ZLIB
>>>> input stream
>>>>
>>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.process(
>>>> BeatsDecoder.java:123)
>>>>
>>>> at org.apache.nifi.processors.beats.handler.BeatsSocketChannelH
>>>> andler.processBuffer(BeatsSocketChannelHandler.java:71)
>>>>
>>>> Caused by: org.apache.nifi.processors.beats.frame.BeatsFrameException:
>>>> Error decompressing  frame: Unexpected end of ZLIB input stream
>>>>
>>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.processP
>>>> AYLOAD(BeatsDecoder.java:292)
>>>>
>>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.process(
>>>> BeatsDecoder.java:103)
>>>>
>>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.processP
>>>> AYLOAD(BeatsDecoder.java:277)
>>>>
>>> while filebeat is logging
>>>
>>>
>>>
>>> Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> sync.go:85: Failed to publish events caused by: read tcp
>>>> localhost:45416->remotehost:5044: i/o timeout
>>>
>>> Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> single.go:91: Error publishing events (retrying): read tcp
>>>> localhost:45416-> remotehost:5044: i/o timeout
>>>
>>> Aug 22 22:29:26 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> metrics.go:39: Non-zero metrics in the last 30s:
>>>> libbeat.logstash.call_count.PublishEvents=1
>>>> libbeat.logstash.publish.read_bytes=186 libbeat.logstash.publish.read_errors=1
>>>> libbeat.logstash.publish.write_bytes=31399
>>>> libbeat.logstash.published_and_acked_events=98
>>>> libbeat.logstash.published_but_not_acked_events=1950
>>>
>>> Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> sync.go:85: Failed to publish events caused by: read tcp
>>>> localhost:49790-> remotehost:5044: i/o timeout
>>>
>>> Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> single.go:91: Error publishing events (retrying): read tcp
>>>> localhost:49790-> remotehost:5044: i/o timeout
>>>
>>> Aug 22 22:29:56 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> metrics.go:39: Non-zero metrics in the last 30s:
>>>> libbeat.logstash.call_count.PublishEvents=1
>>>> libbeat.logstash.publish.read_bytes=138 libbeat.logstash.publish.read_errors=1
>>>> libbeat.logstash.publish.write_bytes=27456
>>>> libbeat.logstash.published_and_acked_events=31
>>>> libbeat.logstash.published_but_not_acked_events=1919
>>>
>>> Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> sync.go:85: Failed to publish events caused by: read tcp
>>>> localhost:54060-> remotehost:5044: i/o timeout
>>>
>>> Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> single.go:91: Error publishing events (retrying): read tcp
>>>> localhost:54060-> remotehost:5044: i/o timeout
>>>
>>> Aug 22 22:30:26 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> metrics.go:39: Non-zero metrics in the last 30s:
>>>> libbeat.logstash.call_count.PublishEvents=1
>>>> libbeat.logstash.publish.read_bytes=492 libbeat.logstash.publish.read_errors=1
>>>> libbeat.logstash.publish.write_bytes=63110
>>>> libbeat.logstash.published_and_acked_events=23
>>>> libbeat.logstash.published_but_not_acked_events=1896
>>>
>>> Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> sync.go:85: Failed to publish events caused by: read tcp
>>>> localhost:58359-> remotehost:5044: i/o timeout
>>>
>>> Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> single.go:91: Error publishing events (retrying): read tcp
>>>> localhost:58359-> remotehost:5044: i/o timeout
>>>
>>>
>>>  has anyone seen something similar or have any idea what might be going
>>> on here?
>>>
>>> my processor params are:
>>>
>>>
>>> Port
>>> 5044
>>>
>>> Receive Buffer Size
>>> 10 MB
>>>
>>> Max Size of Message Queue
>>> 10000
>>>
>>> Max Size of Socket Buffer
>>> 8 MB
>>>
>>> Character Set
>>> UTF-8
>>>
>>> Max Batch Size
>>> 1
>>>
>>> Batching Message Delimiter
>>> \n
>>>
>>> Max Number of TCP Connections
>>> 100
>>>
>>> SSL Context Service
>>> No value set
>>>
>>> Thanks,
>>> Nick
>>>
>>
>

Mime
View raw message