samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Zuljevic <r.zulje...@levi9.com>
Subject RE: [SAMZA-690] Changelog topic creation should not be in the container code
Date Fri, 26 Jun 2015 07:36:03 GMT
Hi Yi,

Thank you for your quick response! Your suggestions make a lot of sense, and I will begin
implementing them right away : )

Met vriendelijke groet / Kind regards,
Robert Žuljević
Software Developer


Address: Trifkovicev trg 6, 21000 Novi Sad, Serbia
Tel.: +31 20 6701 947 | +381 21 2155 500
Mobile: +381 64 428 28 46
Skype: robert.zuljevic
Internet: www.levi9.com

Chamber of commerce Levi9 Holding: 34221951
Chamber of commerce Levi9 IT Services BV: 34224746

This e-mail may contain confidential or privileged information. If you are not (one of) the
intended recipient(s), please notify the sender immediately by reply e-mail and delete this
message and any attachments permanently without retaining a copy. Any review, disclosure,
copying, distribution or taking any action in reliance on the contents of this e-mail by persons
or entities other than the intended recipient(s) is strictly prohibited and may be unlawful.
The services of Levi9 are exclusively subject to its general terms and conditions. These general
terms and conditions can be found on www.levi9.com and a copy will be promptly submitted to
you on your request and free of charge.

-----Original Message-----
From: Yi Pan [mailto:nickpan47@gmail.com] 
Sent: Thursday, June 25, 2015 6:38 PM
To: dev@samza.apache.org
Subject: Re: [SAMZA-690] Changelog topic creation should not be in the container code

Hi, Robert,

Thanks for digging into this. I am embedding my answers below:


On Thu, Jun 25, 2015 at 7:40 AM, Robert Zuljevic <r.zuljevic@levi9.com>
wrote:

>  1.       Is checkpoint topic referred to in the description coordinator
> stream/topic?
>

In the master branch, checkpoint topic is deprecated (except for migration purpose). The checkpoints
will be sent to coordinator stream. Hence, there is no need to change the creation of checkpoint
topic any more.


>  2.       Changelog topic creation is handled by TaskStorageManager class
> which is required by SamzaContainer. Would it be preferable to:
>
> a.       Create TaskStoreManager instance(s) in JobRunner and pass them
> to SamzaContainer
>
> b.      Create changelog stream in JobRunner/JobCoordinator and skip it
> in TaskStoreManager
>
I prefer option b in your above proposal, w/ a slight modification:
1. JobCoordinator now has a changelogManager which reads the changelog partition to task mapping
from the coordinator stream 2. JobCoordinator also has access to the job config and will be
able to figure out what are the changelog topics needed in the job 3. JobCoordinator start
should have an additional step to create all the changelog topics needed in the job. If exists,
validate the partition numbers 4. In TaskStoreManager, we should just get the changelog topic
metadata and validate the partitions are correct.

Does that sound reasonable?

-Yi


>
>
>
>
> Met vriendelijke groet / Kind regards,
>
> Robert Žuljević
>
> Software Developer
>
> [image: Title: Levi9 IT Services]
>  ------------------------------
>
> Address: Trifkovicev trg 6, 21000 Novi Sad, Serbia
>
> Tel.: +31 20 6701 947 | +381 21 2155 500
>
> Mobile: +381 64 428 28 46
>
> Skype: robert.zuljevic
>
> Internet: www.levi9.com
>
>
>
> Chamber of commerce Levi9 Holding: 34221951
>
> Chamber of commerce Levi9 IT Services BV: 34224746
>  ------------------------------
>
> This e-mail may contain confidential or privileged information. If you 
> are not (one of) the intended recipient(s), please notify the sender 
> immediately by reply e-mail and delete this message and any 
> attachments permanently without retaining a copy. Any review, 
> disclosure, copying, distribution or taking any action in reliance on 
> the contents of this e-mail by persons or entities other than the 
> intended recipient(s) is strictly prohibited and may be unlawful.
>
> The services of Levi9 are exclusively subject to its general terms and 
> conditions. These general terms and conditions can be found on 
> www.levi9.com and a copy will be promptly submitted to you on your 
> request and free of charge.
>
>
>
Mime
View raw message