beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eugene Kirpichov <kirpic...@google.com>
Subject Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available
Date Thu, 08 Mar 2018 23:58:14 GMT
Hi Thomas!

In case of tailing a Kafka partition, the restriction would be
[start_offset, infinity), and it would keep being split by checkpointing
into [start_offset, end_offset) and [end_offset, infinity)

On Thu, Mar 8, 2018 at 3:52 PM Thomas Weise <thw@apache.org> wrote:

> Eugene,
>
> I actually had one question regarding the application of SDF for the Kafka
> consumer. Reading through a topic partition can be parallel by splitting a
> partition into multiple restrictions (for use cases where order does not
> matter). But how would the tail read be managed? I assume there would not
> be a new restriction whenever new records arrive (added latency)? The
> examples on slide 40 show an end offset for Kafka, but for a continuous
> read there wouldn't be an end offset?
>
> Thanks,
> Thomas
>
>
> On Thu, Mar 8, 2018 at 2:59 PM, Thomas Weise <thw@apache.org> wrote:
>
>> Great, thanks for sharing!
>>
>>
>> On Thu, Mar 8, 2018 at 12:16 PM, Eugene Kirpichov <kirpichov@google.com>
>> wrote:
>>
>>> Oops that's just the template I used. Thanks for noticing, will
>>> regenerate the PDF and reupload when I get to it.
>>>
>>>
>>> On Thu, Mar 8, 2018, 11:59 AM Dan Halperin <dhalperi@apache.org> wrote:
>>>
>>>> Looks like it was a good talk! Why is it Google Confidential &
>>>> Proprietary, though?
>>>>
>>>> Dan
>>>>
>>>> On Thu, Mar 8, 2018 at 11:49 AM, Eugene Kirpichov <kirpichov@google.com
>>>> > wrote:
>>>>
>>>>> Hey all,
>>>>>
>>>>> The slides for my yesterday's talk at Strata San Jose
>>>>> https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/63696
have
>>>>> been posted on the talk page. They may be of interest both to users and
IO
>>>>> authors.
>>>>>
>>>>> Thanks.
>>>>>
>>>>
>>>>
>>
>

Mime
View raw message