pulsar-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [pulsar] zhengqz opened a new issue #4243: Message size is bigger than 5232640 bytes
Date Thu, 09 May 2019 03:12:06 GMT
zhengqz opened a new issue #4243: Message size is bigger than 5232640 bytes
URL: https://github.com/apache/pulsar/issues/4243
 
 
   **when I use pulsar-2.3.1 run in standalone,  and sync mysql binlog from canal, when message
size more than 5M  cause IllegalReferenceCountException**
   
   # ENV
   1. pulsar 2.3.1
   2. Centos 7.6.1810
   3. canal-mysql-source-config.yaml 
   ```yaml
   configs:
     zkServers: ""
     batchSize: "5120"
     destination: "item_0"
     username: ""
     password: ""
     cluster: false
     singleHostname: "192.168.1.174"
     singlePort: "11111"
   ```
   4. Command
   ```bash
   /opt/apache-pulsar-2.3.1/bin/pulsar-admin source create --archive /opt/apache-pulsar-2.3.1/connectors/pulsar-io-canal-2.3.1.nar
--classname org.apache.pulsar.io.canal.CanalStringSource --tenant public --namespace default
--name canal --destination-topic-name my-topic --source-config-file /opt/apache-pulsar-2.3.1/conf/canal-mysql-source-config.yaml
--parallelism 1
   ```
   
   **/opt/apache-pulsar-2.3.1/logs/functions/public/default/canal/canal-0.log**
   ```bash filename=/opt/apache-pulsar-2.3.1/logs/functions/public/default/canal/canal-0.log
   00:01:37.844 [pulsar-client-io-1-1] INFO  org.apache.pulsar.io.canal.CanalAbstractSource
- CanalRecord ack id is 159877
   00:01:37.986 [pulsar-client-io-1-1] INFO  org.apache.pulsar.io.canal.CanalAbstractSource
- CanalRecord ack id is 159878
   00:01:38.058 [pulsar-timer-4-1] ERROR org.apache.pulsar.functions.sink.PulsarSink - Failed
to publish to topic [my-topic] with error [org.apache.pulsar.client.api.PulsarClientException$InvalidMessageException:
Message size is bigger than 5232640 bytes] with sr
   c sequence id [159879]
   00:01:38.079 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.091 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.102 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.113 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.124 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.135 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.146 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.157 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.168 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.169 [public/default/canal-0] WARN  org.apache.pulsar.client.impl.ProducerImpl
- [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.169 [public/default/canal-0] ERROR org.apache.pulsar.functions.sink.PulsarSink
- Failed to publish to topic [my-topic] with error [org.apache.pulsar.client.api.PulsarClientException:
org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalRefe
   renceCountException: refCnt: 0] with src sequence id [159880]
   00:01:38.179 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.189 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.200 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.211 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.252 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.263 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.270 [public/default/canal-0] WARN  org.apache.pulsar.client.impl.ProducerImpl
- [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.271 [public/default/canal-0] ERROR org.apache.pulsar.functions.sink.PulsarSink
- Failed to publish to topic [my-topic] with error [org.apache.pulsar.client.api.PulsarClientException:
org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalRefe
   renceCountException: refCnt: 0] with src sequence id [159881]
   00:01:38.273 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.284 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.295 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.296 [public/default/canal-0] WARN  org.apache.pulsar.client.impl.ProducerImpl
- [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.297 [public/default/canal-0] ERROR org.apache.pulsar.functions.sink.PulsarSink
- Failed to publish to topic [my-topic] with error [org.apache.pulsar.client.api.PulsarClientException:
org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalRefe
   renceCountException: refCnt: 0] with src sequence id [159882]
   00:01:38.307 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.319 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.331 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.343 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.355 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.367 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.379 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.390 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   00:01:38.401 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic]
[standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException:
refCnt: 0
   
   ```
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message