flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clay Teeter <clay.tee...@maalka.com>
Subject Re: Flink job manager doesn't remove stale checkmarks
Date Thu, 26 Sep 2019 18:09:16 GMT
I see, I'll try turning off incremental checkpoints to see if that helps.

re: Diskspace, i could see a scenario with my application where i could get
10,000+ checkpoints, if the checkpoints are additive.  I'll let you know
what i see.

Thanks!
Clay


On Wed, Sep 25, 2019 at 5:40 PM Fabian Hueske <fhueske@gmail.com> wrote:

> Hi,
>
> You enabled incremental checkpoints.
> This means that parts of older checkpoints that did not change since the
> last checkpoint are not removed because they are still referenced by the
> incremental checkpoints.
> Flink will automatically remove them once they are not needed anymore.
>
> Are you sure that the size of your application's state is not growing too
> large?
>
> Best, Fabian
>
> Am Di., 24. Sept. 2019 um 10:47 Uhr schrieb Clay Teeter <
> clay.teeter@maalka.com>:
>
>> Oh geez,  checkmarks  = checkpoints... sorry.
>>
>> What i mean by stale "checkpoints" are checkpoints that should be reaped
>> by: "state.checkpoints.num-retained: 3".
>>
>> What is happening is that directories:
>>   - state.checkpoints.dir: file:///opt/ha/49/checkpoints
>>   - high-availability.storageDir: file:///opt/ha/49/ha
>> are growing with every checkpoint and i'm running out of disk space.
>>
>> On Tue, Sep 24, 2019 at 4:55 AM Biao Liu <mmyy1110@gmail.com> wrote:
>>
>>> Hi Clay,
>>>
>>> Sorry I don't get your point. I'm not sure what the "stale checkmarks"
>>> exactly means. The HA storage and checkpoint directory left after shutting
>>> down cluster?
>>>
>>> Thanks,
>>> Biao /'bɪ.aʊ/
>>>
>>>
>>>
>>> On Tue, 24 Sep 2019 at 03:12, Clay Teeter <clay.teeter@maalka.com>
>>> wrote:
>>>
>>>> I'm trying to get my standalone cluster to remove stale checkmarks.
>>>>
>>>> The cluster is composed of a single job and task manager backed by
>>>> rocksdb with high availability.
>>>>
>>>> The configuration on both the job and task manager are:
>>>>
>>>> state.backend: rocksdb
>>>> state.checkpoints.dir: file:///opt/ha/49/checkpoints
>>>> state.backend.incremental: true
>>>> state.checkpoints.num-retained: 3
>>>> jobmanager.heap.size: 1024m
>>>> taskmanager.heap.size: 2048m
>>>> taskmanager.numberOfTaskSlots: 24
>>>> parallelism.default: 1
>>>> high-availability.jobmanager.port: 6123
>>>> high-availability.zookeeper.path.root: ********_49
>>>> high-availability: zookeeper
>>>> high-availability.storageDir: file:///opt/ha/49/ha
>>>> high-availability.zookeeper.quorum: ******t:2181
>>>>
>>>> Both machines have access to /opt/ha/49 and /opt/ha/49/checkpoints via
>>>> NFS and are owned by the flink user.  Also, there are no errors that i can
>>>> find.
>>>>
>>>> Does anyone have any ideas that i could try?
>>>>
>>>>

Mime
View raw message