samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Erickson <ja...@stormpath.com>
Subject Re: Samza task not receiving on one partition
Date Fri, 29 Apr 2016 21:45:54 GMT
The issue was that there was a message that exceeded the default
max.message.bytes for a kafka consumer. But rather than give any sort of
error message, the thread responsible for consuming that partition would
simply hang on that message. The other partition threads would continue
along happily.

I have to say that the lack of any sort of timeout or error message was
really a time sink for us. It would have been very helpful to have samza
report that it could not read from that partition because of the size of
the message.

Once we configured systems.kafka.consumer.max.message.bytes to a large
enough value to consume each of the messages on the partition and restarted
the job, it picked up where it had left off and everything started working
as expected.
​

On Mon, Apr 25, 2016 at 6:41 PM Jason Erickson <jason@stormpath.com> wrote:

> My earlier answer to this was replying to the wrong message.  I am
> launching this with ThreadJobFactory.
>
>
> On Mon, Apr 25, 2016 at 4:26 PM Jagadish Venkatraman <
> jagadish1989@gmail.com> wrote:
>
>> Are you running in Yarn or as a local process?
>>
>> If you're running in Yarn, you can use the AppMaster UI to see what the
>> jobmodel is.
>>
>> On Mon, Apr 25, 2016 at 2:33 PM, Jason Erickson <jason@stormpath.com>
>> wrote:
>>
>> > I have a puzzling issue with one of my samza tasks. It works correctly
>> > except for messages on one partition. I have 9 partitions on the topic.
>> If
>> > I send 1000 messages, I only receive about 890 of them. I have checked
>> with
>> > kafka-console-consumer with partition keys that I know don’t get
>> processed
>> > by my samza job and the console consumer DOES see the message, so I know
>> > it’s getting written to the topic and that at least a vanilla consumer
>> can
>> > see it just fine.
>> >
>> >
>> > I’m happy to share whatever interesting configuration information would
>> > help narrow this down, but right now, I’m a bit mystified about what I
>> > would even share.
>> >
>> >
>> >  I’m running
>> >
>> > * samza-kafka_2.10 version 0.9.1
>> >
>> > * kafka_2.10 version 0.8.2.1 on the client
>> >
>> > * kafka broker 0.9.0.0
>> > ​
>> >
>>
>>
>>
>> --
>> Jagadish V,
>> Graduate Student,
>> Department of Computer Science,
>> Stanford University
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message