flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Congxian Qiu <qcx978132...@gmail.com>
Subject Re: Re:Apache Flink - Operator name and uuid best practices
Date Thu, 21 Nov 2019 12:26:37 GMT
Hi
1. I think this issue[1] can help to understand the directory layout
2. chk-6 directory or the metafilePath, for more information, you can ref
to[2][3]
3. every checkpoint contains a meta-file record such, maybe you can ref
to[4], and debug it for more information
4. currently, you need to go through  the restore logic to see if the
files' contents

[1] https://issues.apache.org/jira/browse/FLINK-8531
[2]
https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L1153
[3]
https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/state/checkpoints.html#resuming-from-a-retained-checkpoint
[4]
https://github.com/apache/flink/blob/master/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CheckpointMetadataLoadingTest.java
Best,
Congxian


M Singh <mans2singh@yahoo.com> 于2019年11月21日周四 下午7:44写道:

> Hi Congxian:
>
> For my application i see many uuids under the chk-6 directory ( I posted
> one in the sample above).   I am trying to understand that if I restart the
> application with this checkpoint (which I believe I can just like a
> savepoint - I am using chk-6 as an example below)
> 1. I believe each chk-<d> - is a complete state of checkpoint. Is that
> correct ?
> 2. How to I point it to the checkpoint (chk-6) when I submit the job - Do
> I point it to the jobid or the chk-6 directory ?  I am presuming the latter
> (ie, pointing to the chk06 directory but just want to confirm.
> 3. Secondly, how does the application map the the files under the chk-6 to
> restore the state of each of the stateful operators ?
> 4. Is there any API by which I can examine the contents of the files under
> the chk-6 directory ?
>
> Thanks for your help.
>
> Mans
>
> On Wednesday, November 20, 2019, 09:13:39 PM EST, Congxian Qiu <
> qcx978132955@gmail.com> wrote:
>
>
> Hi
>
> Currently, the last part (uuid a4d87cda-2afd-47d4-8d3f-b0658466fb2d) is
> generated by UUID.randomUUID(), so there is not a easy way to map this to
> the assigned in the application.
> In another word, the last part (uuid a4d87cda-2afd-47d4-8d3f-b0658466fb2d) belongs
> to one checkpoint, and the assigned in the application belongs to one
> operator, they are different.
>
> Best,
> Congxian
>
>
> M Singh <mans2singh@yahoo.com> 于2019年11月21日周四 上午6:18写道:
>
> Hi Arvid:
>
> Thanks for your clarification.
>
> I am giving supplying uid for the stateful operators and find the
> following directory structure on in the chkpoint directory:
>
> f4e78cb47f9dc12859558be7d15f39d0/*chk*
> -6/a4d87cda-2afd-47d4-8d3f-b0658466fb2d
> The first part f4e78cb47f9dc12859558be7d15f39d0 is the job_id
> Is there a way to map the last part (uuid
> a4d87cda-2afd-47d4-8d3f-b0658466fb2d) -  to the uid assigned in the
> application ?
>
> Thanks
>
> On Wednesday, November 20, 2019, 07:52:49 AM EST, Arvid Heise <
> arvid@ververica.com> wrote:
>
>
> Hi Mans,
>
> just to follow up. There are no limitations for name or uuid.
>
> The uuid will be in fact hashed internally while the StreamGraph is being
> generated, so all characters are allowed.
> The name is only for debugging purposes and web ui. If you use very
> special characters, you may see oddities in logs/web ui, but nothing should
> break.
> Spaces or parentheses should work in any case.
>
> Best,
>
> Arvid
>
> On Sat, Nov 16, 2019 at 6:40 PM M Singh <mans2singh@yahoo.com> wrote:
>
> Thanks Jiayi for your response. I am thinking on the same lines.
>
> Regarding using the same name and uuid, I believe the checkpoint state for
> an operator will be easy to identify if the uuid is the same as name.  But
> I am not sure if having a very long name and uuid or a character like
> parenthesis, etc might cause any issues, so just wanted to check.
>
> Mans
>
> On Saturday, November 16, 2019, 11:19:08 AM EST, Jiayi Liao <
> bupt_ljy@163.com> wrote:
>
>
> Hi Mans!
>
>
> Firstly let’s see how operator’s name and uid is used. AFAIK, operator’s
> name is used in WebUI and metrics reporting, and uid is used to mark the
> uniqueness of operator which is useful when you’re using state[1].
>
>
> > Are there any restrictions on the length of the name and uuid attributes?
>
> It’s pretty much the same as you define a string value, so there is no
> special restrictions on this.
>
>
> > Are there any restrictions on the characters used for name and uuid
> (blank spaces, etc) ?
>
> I’m not a hundred percent sure about this but I run a testing program and
> it works fine.
>
>
> > Can the name and uuid be the same ?
>
> Yes. But uuids accross operators cannot be same.
>
>
> For me I usually set name and uuid for almost every operator, which gives
> me better experience in monitoring and scaling.
>
>
> Hope this helps.
>
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#matching-operator-state
>
>
>
> Best,
>
> Jiayi Liao
>
>
> At 2019-11-16 18:35:38, "M Singh" <mans2singh@yahoo.com> wrote:
>
> Hi:
>
> I am working on a project and wanted to find out what are the best
> practices for setting name and uuid for operators:
>
> 1. Are there any restrictions on the length of the name and uuid
> attributes ?
> 2. Are there any restrictions on the characters used for name and uuid
> (blank spaces, etc) ?
> 3. Can the name and uuid be the same ?
>
> Please let me know if there is any other advice.
>
> Thanks
>
> Mans
>
>
>
>
>
>

Mime
View raw message