kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Measuring remaining data
Date Thu, 02 Jan 2014 22:42:46 GMT
Since 0.8 we have migrated from physical offset to logical offset, and that
is why we can no longer trace bytes in size. KAFKA-1197 is filed for part
of this requirement, does it satisfy what you need?

As for KAFKA-656, it is not abandoned but we are still trying to figure out
the correct way of doing so.

Guozhang


On Thu, Jan 2, 2014 at 2:26 PM, Jonathan Davey <jond3k@gmail.com> wrote:

> That's certainly a step in the right direction. The thing is, I'd
> ideally like to use bytes rather than just messages and that data no
> longer seems to be available in Kafka 0.8. KAFKA-656 might give me
> what I need but that seems abandoned.
>
> On 2 January 2014 21:26, Guozhang Wang <wangguoz@gmail.com> wrote:
> > Hi Jonathan,
> >
> > Have you checked out the ConsumerOffsetChecker tool?
> >
> > Guozhang
> >
> >
> > On Thu, Jan 2, 2014 at 1:15 PM, Jonathan Davey <jond3k@gmail.com> wrote:
> >
> >> I need to provide a report that shows the total amount of unread data
> >> available to a consumer group for a particular set of topics.
> >>
> >> Right now I do this by adding custom offset tracking logic to the
> >> producers and consumers. This works but I think I can do better with
> >> one of the following ideas:
> >>
> >> * Running something alongside each broker that grabs the data from the
> >> log dir and zookeeper
> >> * Extending Kafka to do the same as above but expose it as a request
> >> through the protocol
> >>
> >> If I wanted to do these things, where would be a good place to start
> >> looking?
> >>
> >
> >
> >
> > --
> > -- Guozhang
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message