flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vishal Santoshi <vishal.santo...@gmail.com>
Subject Re: K8s job cluster and cancel and resume from a save point ?
Date Tue, 12 Mar 2019 15:55:02 GMT
Awesome, thanks!

On Tue, Mar 12, 2019 at 11:53 AM Gary Yao <gary@ververica.com> wrote:

> The RC artifacts are only deployed to the Maven Central Repository when
> the RC
> is promoted to a release. As written in the 1.8.0 RC1 voting email [1], you
> can find the maven artifacts, and the Flink binaries here:
>
>     -
> https://repository.apache.org/content/repositories/orgapacheflink-1210/
>     - https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc1/
>
> Alternatively, you can apply the patch yourself, and build Flink 1.7 from
> sources [2]. On my machine this takes around 10 minutes if tests are
> skipped.
>
> Best,
> Gary
>
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-1-8-0-release-candidate-1-td27637.html
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-1.7/flinkDev/building.html#build-flink
>
> On Tue, Mar 12, 2019 at 4:01 PM Vishal Santoshi <vishal.santoshi@gmail.com>
> wrote:
>
>> Do you have a mvn repository ( at mvn central )  set up for 1,8 release
>> candidate. We could test it for you.
>>
>> Without 1.8and this exit code we are essentially held up.
>>
>> On Tue, Mar 12, 2019 at 10:56 AM Gary Yao <gary@ververica.com> wrote:
>>
>>> Nobody can tell with 100% certainty. We want to give the RC some exposure
>>> first, and there is also a release process that is prescribed by the ASF
>>> [1].
>>> You can look at past releases to get a feeling for how long the release
>>> process lasts [2].
>>>
>>> [1] http://www.apache.org/legal/release-policy.html#release-approval
>>> [2]
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/template/NamlServlet.jtp?macro=search_page&node=1&query=%5BVOTE%5D+Release&days=0
>>>
>>>
>>> On Tue, Mar 12, 2019 at 3:38 PM Vishal Santoshi <
>>> vishal.santoshi@gmail.com> wrote:
>>>
>>>> And when is the 1.8.0 release expected ?
>>>>
>>>> On Tue, Mar 12, 2019 at 10:32 AM Vishal Santoshi <
>>>> vishal.santoshi@gmail.com> wrote:
>>>>
>>>>> :) That makes so much more sense. Is  k8s native flink a part of this
>>>>> release ?
>>>>>
>>>>> On Tue, Mar 12, 2019 at 10:27 AM Gary Yao <gary@ververica.com>
wrote:
>>>>>
>>>>>> Hi Vishal,
>>>>>>
>>>>>> This issue was fixed recently [1], and the patch will be released
>>>>>> with 1.8. If
>>>>>> the Flink job gets cancelled, the JVM should exit with code 0. There
>>>>>> is a
>>>>>> release candidate [2], which you can test.
>>>>>>
>>>>>> Best,
>>>>>> Gary
>>>>>>
>>>>>> [1] https://issues.apache.org/jira/browse/FLINK-10743
>>>>>> [2]
>>>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-1-8-0-release-candidate-1-td27637.html
>>>>>>
>>>>>> On Tue, Mar 12, 2019 at 3:21 PM Vishal Santoshi <
>>>>>> vishal.santoshi@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks Vijay,
>>>>>>>
>>>>>>> This is the larger issue.  The cancellation routine is itself
broken.
>>>>>>>
>>>>>>> On cancellation flink does remove the checkpoint counter
>>>>>>>
>>>>>>> *2019-03-12 14:12:13,143
>>>>>>> INFO  org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter
 -
>>>>>>> Removing /checkpoint-counter/00000000000000000000000000000000
from
>>>>>>> ZooKeeper *
>>>>>>>
>>>>>>> but exist with a non zero code
>>>>>>>
>>>>>>> *2019-03-12 14:12:13,477
>>>>>>> INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint 
       -
>>>>>>> Terminating cluster entrypoint process StandaloneJobClusterEntryPoint
with
>>>>>>> exit code 1444.*
>>>>>>>
>>>>>>>
>>>>>>> That I think is an issue. A cancelled job is a complete job and
thus
>>>>>>> the exit code should be 0 for k8s to mark it complete.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Mar 12, 2019 at 10:18 AM Vijay Bhaskar <
>>>>>>> bhaskar.ebay77@gmail.com> wrote:
>>>>>>>
>>>>>>>> Yes Vishal. Thats correct.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Bhaskar
>>>>>>>>
>>>>>>>> On Tue, Mar 12, 2019 at 7:14 PM Vishal Santoshi <
>>>>>>>> vishal.santoshi@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> This really not cool but here you go. This seems to work.
Agreed
>>>>>>>>> that this cannot be this painful. The cancel does not
exit with an exit
>>>>>>>>> code pf 0 and thus the job has to manually delete. Vijay
does this align
>>>>>>>>> with what you have had to do ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>    - Take a save point . This returns a request id
>>>>>>>>>
>>>>>>>>>    curl  --header "Content-Type: application/json" --request
POST --data '{"target-directory":"hdfs://nn-crunchy:8020/tmp/xyz14","cancel-job":false}' 
  https://*************/jobs/00000000000000000000000000000000/savepoints
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>    - Make sure the save point succeeded
>>>>>>>>>
>>>>>>>>>    curl  --request GET   https://****************/jobs/00000000000000000000000000000000/savepoints/2c053ce3bea31276aa25e63784629687
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>    - cancel the job
>>>>>>>>>
>>>>>>>>>    curl  --request PATCH https://***************/jobs/00000000000000000000000000000000?mode=cancel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>    - Delete the job and deployment
>>>>>>>>>
>>>>>>>>>    kubectl delete -f manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>>>>>>>>>
>>>>>>>>>    kubectl delete -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>    - Edit the job-cluster-job-deployment.yaml. Add/Edit
>>>>>>>>>
>>>>>>>>>    args: ["job-cluster",
>>>>>>>>>
>>>>>>>>>                   "--fromSavepoint",
>>>>>>>>>
>>>>>>>>>                   "hdfs://************/tmp/xyz14/savepoint-000000-1d4f71345e22",
>>>>>>>>>                   "--job-classname", .........
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>    - Restart
>>>>>>>>>
>>>>>>>>>    kubectl create -f manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>>>>>>>>>
>>>>>>>>>    kubectl create -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>    - Make sure from the UI, that it restored from the
specific
>>>>>>>>>    save point.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Mar 12, 2019 at 7:26 AM Vijay Bhaskar <
>>>>>>>>> bhaskar.ebay77@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Yes Its supposed to work.  But unfortunately it was
not working.
>>>>>>>>>> Flink community needs to respond to this behavior.
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> Bhaskar
>>>>>>>>>>
>>>>>>>>>> On Tue, Mar 12, 2019 at 3:45 PM Vishal Santoshi <
>>>>>>>>>> vishal.santoshi@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Aah.
>>>>>>>>>>> Let me try this out and will get back to you.
>>>>>>>>>>> Though I would assume that save point with cancel
is a single
>>>>>>>>>>> atomic step, rather then a save point *followed*
 by a
>>>>>>>>>>> cancellation ( else why would that be an option
).
>>>>>>>>>>> Thanks again.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Mar 12, 2019 at 4:50 AM Vijay Bhaskar
<
>>>>>>>>>>> bhaskar.ebay77@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Vishal,
>>>>>>>>>>>>
>>>>>>>>>>>> yarn-cancel doesn't mean to be for yarn cluster.
It works for
>>>>>>>>>>>> all clusters. Its recommended command.
>>>>>>>>>>>>
>>>>>>>>>>>> Use the following command to issue save point.
>>>>>>>>>>>>  curl  --header "Content-Type: application/json"
--request POST
>>>>>>>>>>>> --data '{"target-directory":"hdfs://*********:8020/tmp/xyz1",
>>>>>>>>>>>> "cancel-job":false}'  \ https://
>>>>>>>>>>>> ************.ingress.*******/jobs/00000000000000000000000000000000/savepoints
>>>>>>>>>>>>
>>>>>>>>>>>> Then issue yarn-cancel.
>>>>>>>>>>>> After that  follow the process to restore
save point
>>>>>>>>>>>>
>>>>>>>>>>>> Regards
>>>>>>>>>>>> Bhaskar
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Mar 12, 2019 at 2:11 PM Vishal Santoshi
<
>>>>>>>>>>>> vishal.santoshi@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello Vijay,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                Thank you for the reply.
This though is k8s
>>>>>>>>>>>>> deployment ( rather then yarn ) but may
be they follow the same lifecycle.
>>>>>>>>>>>>> I issue a* save point with cancel*  as
documented here
>>>>>>>>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/rest_api.html#jobs-jobid-savepoints,
>>>>>>>>>>>>> a straight up
>>>>>>>>>>>>>  curl  --header "Content-Type: application/json"
--request
>>>>>>>>>>>>> POST --data
>>>>>>>>>>>>> '{"target-directory":"hdfs://*********:8020/tmp/xyz1","cancel-job":true}'
>>>>>>>>>>>>> \ https://
>>>>>>>>>>>>> ************.ingress.*******/jobs/00000000000000000000000000000000/savepoints
>>>>>>>>>>>>>
>>>>>>>>>>>>> I would assume that after taking the
save point, the jvm
>>>>>>>>>>>>> should exit, after all the k8s deployment
is of kind: job and if it is a
>>>>>>>>>>>>> job cluster then a cancellation should
exit the jvm and hence the pod. It
>>>>>>>>>>>>> does seem to do some things right. It
stops a bunch of stuff ( the
>>>>>>>>>>>>> JobMaster, the slotPol, zookeeper coordinator
etc ) . It also remove the
>>>>>>>>>>>>> checkpoint counter but does not exit
 the job. And after a little bit the
>>>>>>>>>>>>> job is restarted which does not make
sense and absolutely not the right
>>>>>>>>>>>>> thing to do  ( to me at least ).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Further if I delete the deployment and
the job from k8s and
>>>>>>>>>>>>> restart the job and deployment fromSavePoint,
it refuses to honor the
>>>>>>>>>>>>> fromSavePoint. I have to delete the zk
chroot for it to consider the save
>>>>>>>>>>>>> point.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thus the process of cancelling and resuming
from a SP on a k8s
>>>>>>>>>>>>> job cluster deployment  seems to be
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - cancel with save point as defined
hre
>>>>>>>>>>>>>    https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/rest_api.html#jobs-jobid-savepoints
>>>>>>>>>>>>>    - delete the job manger job  and task
manager deployments
>>>>>>>>>>>>>    from k8s almost immediately.
>>>>>>>>>>>>>    - clear the ZK chroot for the 0000000......
job  and may
>>>>>>>>>>>>>    be the checkpoints directory.
>>>>>>>>>>>>>    - resumeFromCheckPoint
>>>>>>>>>>>>>
>>>>>>>>>>>>> If some body can say that this indeed
is the process ?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>  Logs are attached.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:43,871 INFO
>>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobMaster
                 -
>>>>>>>>>>>>> Savepoint stored in
>>>>>>>>>>>>> hdfs://*********:8020/tmp/xyz3/savepoint-000000-6d5bdc9b53ae.
Now
>>>>>>>>>>>>> cancelling 00000000000000000000000000000000.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:43,871 INFO
>>>>>>>>>>>>> org.apache.flink.runtime.executiongraph.ExecutionGraph
       -
>>>>>>>>>>>>> Job anomaly_echo (00000000000000000000000000000000)
switched from state
>>>>>>>>>>>>> RUNNING to CANCELLING.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,227 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator
>>>>>>>>>>>>>     - Completed checkpoint 10 for job
>>>>>>>>>>>>> 00000000000000000000000000000000 (7238
bytes in 311 ms).
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,232 INFO
>>>>>>>>>>>>> org.apache.flink.runtime.executiongraph.ExecutionGraph
       -
>>>>>>>>>>>>> Source: Barnacle Anomalies Kafka topic
-> Map -> Sink: Logging Sink (1/1)
>>>>>>>>>>>>> (e2d02ca40a9a6c96a0c1882f5a2e4dd6) switched
from RUNNING to CANCELING.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,274 INFO
>>>>>>>>>>>>> org.apache.flink.runtime.executiongraph.ExecutionGraph
       -
>>>>>>>>>>>>> Source: Barnacle Anomalies Kafka topic
-> Map -> Sink: Logging Sink (1/1)
>>>>>>>>>>>>> (e2d02ca40a9a6c96a0c1882f5a2e4dd6) switched
from CANCELING to CANCELED.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,276 INFO
>>>>>>>>>>>>> org.apache.flink.runtime.executiongraph.ExecutionGraph
       -
>>>>>>>>>>>>> Job anomaly_echo (00000000000000000000000000000000)
switched from state
>>>>>>>>>>>>> CANCELLING to CANCELED.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,276 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator
>>>>>>>>>>>>>     - Stopping checkpoint coordinator
for job
>>>>>>>>>>>>> 00000000000000000000000000000000.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,277 INFO
>>>>>>>>>>>>> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore
>>>>>>>>>>>>> - Shutting down
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,323 INFO  org.apache.flink.runtime.checkpoint.CompletedCheckpoint
>>>>>>>>>>>>>       - Checkpoint with ID 8 at
>>>>>>>>>>>>> 'hdfs://nn-crunchy:8020/tmp/xyz2/savepoint-000000-859e626cbb00'
not
>>>>>>>>>>>>> discarded.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,437 INFO
>>>>>>>>>>>>> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore
>>>>>>>>>>>>> - Removing
>>>>>>>>>>>>> /k8s_anomalyecho/k8s_anomalyecho/checkpoints/00000000000000000000000000000000
>>>>>>>>>>>>> from ZooKeeper
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,437 INFO  org.apache.flink.runtime.checkpoint.CompletedCheckpoint
>>>>>>>>>>>>>       - Checkpoint with ID 10 at
>>>>>>>>>>>>> 'hdfs://*************:8020/tmp/xyz3/savepoint-000000-6d5bdc9b53ae'
not
>>>>>>>>>>>>> discarded.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,447 INFO
>>>>>>>>>>>>> org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter
>>>>>>>>>>>>> - Shutting down.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,447 INFO
>>>>>>>>>>>>> org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter
>>>>>>>>>>>>> - Removing
>>>>>>>>>>>>> /checkpoint-counter/00000000000000000000000000000000
from ZooKeeper
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,463 INFO
>>>>>>>>>>>>> org.apache.flink.runtime.dispatcher.MiniDispatcher
           -
>>>>>>>>>>>>> Job 00000000000000000000000000000000
reached globally terminal state
>>>>>>>>>>>>> CANCELED.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,467 INFO
>>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobMaster
                 -
>>>>>>>>>>>>> Stopping the JobMaster for job
>>>>>>>>>>>>> anomaly_echo(00000000000000000000000000000000).
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,468 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint
>>>>>>>>>>>>>         - Shutting StandaloneJobClusterEntryPoint
down with
>>>>>>>>>>>>> application status CANCELED. Diagnostics
null.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,468 INFO
>>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint
>>>>>>>>>>>>> - Shutting down rest endpoint.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,473 INFO
>>>>>>>>>>>>> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService
>>>>>>>>>>>>> - Stopping ZooKeeperLeaderRetrievalService
>>>>>>>>>>>>> /leader/resource_manager_lock.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,475 INFO
>>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobMaster
                 -
>>>>>>>>>>>>> Close ResourceManager connection d38c6e599d16415a69c65c8b2a72d9a2:
>>>>>>>>>>>>> JobManager is shutting down..
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,475 INFO
>>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.slotpool.SlotPool
         -
>>>>>>>>>>>>> Suspending SlotPool.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,476 INFO
>>>>>>>>>>>>> org.apache.flink.runtime.jobmaster.slotpool.SlotPool
         -
>>>>>>>>>>>>> Stopping SlotPool.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,476 INFO
>>>>>>>>>>>>> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager
>>>>>>>>>>>>> - Disconnect job manager a0dcf8aaa3fadcfd6fef49666d7344ca
>>>>>>>>>>>>> @akka.tcp://flink@anomalyecho:6123/user/jobmanager_0
for job
>>>>>>>>>>>>> 00000000000000000000000000000000 from
the resource manager.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019-03-12 08:10:44,477 INFO
>>>>>>>>>>>>> org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService
>>>>>>>>>>>>> - Stopping ZooKeeperLeaderElectionService
>>>>>>>>>>>>> ZooKeeperLeaderElectionService{leaderPath='/leader/00000000000000000000000000000000/job_manager_lock'}.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> After a little bit
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Starting the job-cluster
>>>>>>>>>>>>>
>>>>>>>>>>>>> used deprecated key `jobmanager.heap.mb`,
please replace with
>>>>>>>>>>>>> key `jobmanager.heap.size`
>>>>>>>>>>>>>
>>>>>>>>>>>>> Starting standalonejob as a console application
on host
>>>>>>>>>>>>> anomalyecho-mmg6t.
>>>>>>>>>>>>>
>>>>>>>>>>>>> ..
>>>>>>>>>>>>>
>>>>>>>>>>>>> ..
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Mar 12, 2019 at 3:25 AM Vijay
Bhaskar <
>>>>>>>>>>>>> bhaskar.ebay77@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Vishal
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Save point with cancellation internally
use  /cancel  REST
>>>>>>>>>>>>>> API. Which is not stable API.  It
always exits with 404. Best  way to issue
>>>>>>>>>>>>>> is:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> a) First issue save point REST API
>>>>>>>>>>>>>> b) Then issue  /yarn-cancel  rest
API( As described in
>>>>>>>>>>>>>> http://mail-archives.apache.org/mod_mbox/flink-user/201804.mbox/%3C0ffa63f4-e6ed-42d8-1928-37a7adaaac0e@apache.org%3E
>>>>>>>>>>>>>> )
>>>>>>>>>>>>>> c) Then After resuming your job,
provide save point Path as
>>>>>>>>>>>>>> argument for the run jar REST API,
which is returned by the (a)
>>>>>>>>>>>>>> Above is the smoother way
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>> Bhaskar
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Mar 12, 2019 at 2:46 AM Vishal
Santoshi <
>>>>>>>>>>>>>> vishal.santoshi@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There are some issues I see and
would want to get some
>>>>>>>>>>>>>>> feedback
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1. On Cancellation With SavePoint
with a Target Directory ,
>>>>>>>>>>>>>>> the k8s  job  does not exit (
it is not a deployment ) . I would assume
>>>>>>>>>>>>>>> that on cancellation the jvm
should exit, after cleanup etc, and thus the
>>>>>>>>>>>>>>> pod should too. That does not
happen and thus the job pod remains live. Is
>>>>>>>>>>>>>>> that expected ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2. To resume fro a save point
it seems that I have to delete
>>>>>>>>>>>>>>> the job id ( 0000000000.... )
 from ZooKeeper ( this is HA ), else it
>>>>>>>>>>>>>>> defaults to the latest checkpoint
no matter what
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am kind of curious as to what
in 1.7.2 is the tested
>>>>>>>>>>>>>>> process of cancelling with a
save point and resuming  and what is the
>>>>>>>>>>>>>>> cogent story around job id (
defaults to 000000000000.. ). Note that
>>>>>>>>>>>>>>> --job-id does not work with 1.7.2
so even though that does not make sense,
>>>>>>>>>>>>>>> I still can not provide a new
job id.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Vishal.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>

Mime
View raw message