flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Checkpointing with RocksDB as statebackend
Date Thu, 16 Mar 2017 13:11:46 GMT
Yes, you can change the GC using the env.java.opts parameter.
We are not setting any GC on YARN.

On Thu, Mar 16, 2017 at 1:50 PM, Stephan Ewen <sewen@apache.org> wrote:

> The only immediate workaround is to use windows with "reduce" or "fold" or
> "aggregate" and not "apply". And to not use an evictor.
>
> The good news is that I think we have a good way of fixing this soon,
> making an adjustment in RocksDB.
>
> For the Yarn / g1gc question: Not 100% sure about that - you can check if
> it used g1gc. If not, you may be able to pass this through the
> "env.java.opts" parameter. (cc robert for confirmation)
>
> Stephan
>
>
>
> On Thu, Mar 16, 2017 at 8:31 AM, vinay patil <vinay18.patil@gmail.com>
> wrote:
>
>> Hi Stephan,
>>
>> What can be the workaround for this ?
>>
>> Also need one confirmation : Is G1 GC used by default when running the
>> pipeline on YARN. (I see a thread of 2015 where G1 is used by default for
>> JAVA8)
>>
>>
>>
>> Regards,
>> Vinay Patil
>>
>> On Wed, Mar 15, 2017 at 10:32 PM, Stephan Ewen [via Apache Flink User
>> Mailing List archive.] <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=12234&i=0>> wrote:
>>
>>> Hi Vinay!
>>>
>>> Savepoints also call the same problematic RocksDB function,
>>> unfortunately.
>>>
>>> We will have a fix next month. We either (1) get a patched RocksDB
>>> version or we (2) implement a different pattern for ListState in Flink.
>>>
>>> (1) would be the better solution, so we are waiting for a response from
>>> the RocksDB folks. (2) is always possible if we cannot get a fix from
>>> RocksDB.
>>>
>>> Stephan
>>>
>>>
>>> On Wed, Mar 15, 2017 at 5:53 PM, vinay patil <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=12225&i=0>> wrote:
>>>
>>>> Hi Stephan,
>>>>
>>>> Thank you for making me aware of this.
>>>>
>>>> Yes I am using a window without reduce function (Apply function). The
>>>> discussion happening on JIRA is exactly what I am observing, consistent
>>>> failure of checkpoints after some time and the stream halts.
>>>>
>>>> We want to go live in next month, not sure how this will affect in
>>>> production as we are going to get above 200 million data.
>>>>
>>>> As a workaround can I take the savepoint while the pipeline is running
>>>> ? Let's say if I take savepoint after every 30minutes, will it work ?
>>>>
>>>>
>>>>
>>>> Regards,
>>>> Vinay Patil
>>>>
>>>> On Tue, Mar 14, 2017 at 10:02 PM, Stephan Ewen [via Apache Flink User
>>>> Mailing List archive.] <[hidden email]
>>>> <http:///user/SendEmail.jtp?type=node&node=12224&i=0>> wrote:
>>>>
>>>>> The issue in Flink is https://issues.apache.org/jira/browse/FLINK-5756
>>>>>
>>>>> On Tue, Mar 14, 2017 at 3:40 PM, Stefan Richter <[hidden email]
>>>>> <http:///user/SendEmail.jtp?type=node&node=12209&i=0>>
wrote:
>>>>>
>>>>>> Hi Vinay,
>>>>>>
>>>>>> I think the issue is tracked here: https://github.com/faceb
>>>>>> ook/rocksdb/issues/1988.
>>>>>>
>>>>>> Best,
>>>>>> Stefan
>>>>>>
>>>>>> Am 14.03.2017 um 15:31 schrieb Vishnu Viswanath <[hidden email]
>>>>>> <http:///user/SendEmail.jtp?type=node&node=12209&i=1>>:
>>>>>>
>>>>>> Hi Stephan,
>>>>>>
>>>>>> Is there a ticket number/link to track this, My job has all the
>>>>>> conditions you mentioned.
>>>>>>
>>>>>> Thanks,
>>>>>> Vishnu
>>>>>>
>>>>>> On Tue, Mar 14, 2017 at 7:13 AM, Stephan Ewen <[hidden email]
>>>>>> <http:///user/SendEmail.jtp?type=node&node=12209&i=2>>
wrote:
>>>>>>
>>>>>>> Hi Vinay!
>>>>>>>
>>>>>>> We just discovered a bug in RocksDB. The bug affects windows
without
>>>>>>> reduce() or fold(), windows with evictors, and ListState.
>>>>>>>
>>>>>>> A certain access pattern in RocksDB starts being so slow after
a
>>>>>>> certain size-per-key that it basically brings down the streaming
program
>>>>>>> and the snapshots.
>>>>>>>
>>>>>>> We are reaching out to the RocksDB folks and looking for workarounds
>>>>>>> in Flink.
>>>>>>>
>>>>>>> Greetings,
>>>>>>> Stephan
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 1, 2017 at 12:10 PM, Stephan Ewen <[hidden email]
>>>>>>> <http:///user/SendEmail.jtp?type=node&node=12209&i=3>>
wrote:
>>>>>>>
>>>>>>>> @vinay  Can you try to not set the buffer timeout at all?
I am
>>>>>>>> actually not sure what would be the effect of setting it
to a negative
>>>>>>>> value, that can be a cause of problems...
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Feb 27, 2017 at 7:44 PM, Seth Wiesman <[hidden
email]
>>>>>>>> <http:///user/SendEmail.jtp?type=node&node=12209&i=4>>
wrote:
>>>>>>>>
>>>>>>>>> Vinay,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The bucketing sink performs rename operations during
the
>>>>>>>>> checkpoint and if it tries to rename a file that is not
yet consistent that
>>>>>>>>> would cause a FileNotFound exception which would fail
the checkpoint.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Stephan,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Currently my aws fork contains some very specific assumptions
>>>>>>>>> about the pipeline that will in general only hold for
my pipeline. This is
>>>>>>>>> because there were still some open questions that  I
had about how to solve
>>>>>>>>> consistency issues in the general case. I will comment
on the Jira issue
>>>>>>>>> with more specific.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Seth Wiesman
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *From: *vinay patil <[hidden email]
>>>>>>>>> <http:///user/SendEmail.jtp?type=node&node=12209&i=5>>
>>>>>>>>> *Reply-To: *"[hidden email]
>>>>>>>>> <http:///user/SendEmail.jtp?type=node&node=12209&i=6>"
<[hidden
>>>>>>>>> email] <http:///user/SendEmail.jtp?type=node&node=12209&i=7>>
>>>>>>>>> *Date: *Monday, February 27, 2017 at 1:05 PM
>>>>>>>>> *To: *"[hidden email]
>>>>>>>>> <http:///user/SendEmail.jtp?type=node&node=12209&i=8>"
<[hidden
>>>>>>>>> email] <http:///user/SendEmail.jtp?type=node&node=12209&i=9>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *Subject: *Re: Checkpointing with RocksDB as statebackend
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Seth,
>>>>>>>>>
>>>>>>>>> Thank you for your suggestion.
>>>>>>>>>
>>>>>>>>> But if the issue is only related to S3, then why does
this happen
>>>>>>>>> when I replace the S3 sink  to HDFS as well (for checkpointing
I am using
>>>>>>>>> HDFS only )
>>>>>>>>>
>>>>>>>>> Stephan,
>>>>>>>>>
>>>>>>>>> Another issue I see is when I set env.setBufferTimeout(-1)
, and
>>>>>>>>> keep the checkpoint interval to 10minutes, I have observed
that nothing
>>>>>>>>> gets written to sink (tried with S3 as well as HDFS),
atleast I was
>>>>>>>>> expecting pending files here.
>>>>>>>>>
>>>>>>>>> This issue gets worst when checkpointing is disabled
 as nothing
>>>>>>>>> is written.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Vinay Patil
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Feb 27, 2017 at 10:55 PM, Stephan Ewen [via Apache
Flink
>>>>>>>>> User Mailing List archive.] <[hidden email]> wrote:
>>>>>>>>>
>>>>>>>>> Hi Seth!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Wow, that is an awesome approach.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> We have actually seen these issues as well and we are
looking to
>>>>>>>>> eventually implement our own S3 file system (and circumvent
Hadoop's S3
>>>>>>>>> connector that Flink currently relies on):
>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-5706
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Do you think your patch would be a good starting point
for that
>>>>>>>>> and would you be willing to share it?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The Amazon AWS SDK for Java is Apache 2 licensed, so
that is
>>>>>>>>> possible to fork officially, if necessary...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Greetings,
>>>>>>>>>
>>>>>>>>> Stephan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Feb 27, 2017 at 5:15 PM, Seth Wiesman <[hidden
email]
>>>>>>>>> <http://user/SendEmail.jtp?type=node&node=11943&i=0>>
wrote:
>>>>>>>>>
>>>>>>>>> Just wanted to throw in my 2cts.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I’ve been running pipelines with similar state size
using rocksdb
>>>>>>>>> which externalize to S3 and bucket to S3. I was getting
stalls like this
>>>>>>>>> and ended up tracing the problem to S3 and the bucketing
sink. The solution
>>>>>>>>> was two fold:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 1)       I forked hadoop-aws and have it treat flink
as a source
>>>>>>>>> of truth. Emr uses a dynamodb table to determine if S3
is inconsistent.
>>>>>>>>> Instead I say that if flink believes that a file exists
on S3 and we don’t
>>>>>>>>> see it then I am going to trust that flink is in a consistent
state and S3
>>>>>>>>> is not. In this case, various operations will perform
a back off and retry
>>>>>>>>> up to a certain number of times.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2)       The bucketing sink performs multiple renames
over the
>>>>>>>>> lifetime of a file, occurring when a checkpoint starts
and then again on
>>>>>>>>> notification after it completes. Due to S3’s consistency
guarantees the
>>>>>>>>> second rename of file can never be assured to work and
will eventually fail
>>>>>>>>> either during or after a checkpoint. Because there is
no upper bound on the
>>>>>>>>> time it will take for a file on S3 to become consistent,
retries cannot
>>>>>>>>> solve this specific problem as it could take upwards
of many minutes to
>>>>>>>>> rename which would stall the entire pipeline. The only
viable solution I
>>>>>>>>> could find was to write a custom sink which understands
S3. Each writer
>>>>>>>>> will write file locally and then copy it to S3 on checkpoint.
By only
>>>>>>>>> interacting with S3 once per file it can circumvent consistency
issues all
>>>>>>>>> together.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hope this helps,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Seth Wiesman
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *From: *vinay patil <[hidden email]
>>>>>>>>> <http://user/SendEmail.jtp?type=node&node=11943&i=1>>
>>>>>>>>> *Reply-To: *"[hidden email]
>>>>>>>>> <http://user/SendEmail.jtp?type=node&node=11943&i=2>"
<[hidden
>>>>>>>>> email] <http://user/SendEmail.jtp?type=node&node=11943&i=3>>
>>>>>>>>> *Date: *Saturday, February 25, 2017 at 10:50 AM
>>>>>>>>> *To: *"[hidden email]
>>>>>>>>> <http://user/SendEmail.jtp?type=node&node=11943&i=4>"
<[hidden
>>>>>>>>> email] <http://user/SendEmail.jtp?type=node&node=11943&i=5>>
>>>>>>>>> *Subject: *Re: Checkpointing with RocksDB as statebackend
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> HI Stephan,
>>>>>>>>>
>>>>>>>>> Just to avoid the confusion here, I am using S3 sink
for writing
>>>>>>>>> the data, and using HDFS for storing checkpoints.
>>>>>>>>>
>>>>>>>>> There are 2 core nodes (HDFS) and two task nodes on EMR
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I replaced s3 sink with HDFS for writing data in my last
test.
>>>>>>>>>
>>>>>>>>> Let's say the checkpoint interval is 5 minutes, now within
>>>>>>>>> 5minutes of run the state size grows to 30GB ,  after
checkpointing the
>>>>>>>>> 30GB state that is maintained in rocksDB has to be copied
to HDFS, right ?
>>>>>>>>> is this causing the pipeline to stall ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Vinay Patil
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Feb 25, 2017 at 12:22 AM, Vinay Patil <[hidden
email]>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi Stephan,
>>>>>>>>>
>>>>>>>>> To verify if S3 is making teh pipeline stall, I have
replaced the
>>>>>>>>> S3 sink with HDFS and kept minimum pause between checkpoints
to 5minutes,
>>>>>>>>> still I see the same issue with checkpoints getting failed.
>>>>>>>>>
>>>>>>>>> If I keep the  pause time to 20 seconds, all checkpoints
are
>>>>>>>>> completed , however there is a hit in overall throughput.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Vinay Patil
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Feb 24, 2017 at 10:09 PM, Stephan Ewen [via Apache
Flink
>>>>>>>>> User Mailing List archive.] <[hidden email]> wrote:
>>>>>>>>>
>>>>>>>>> Flink's state backends currently do a good number of
"make sure
>>>>>>>>> this exists" operations on the file systems. Through
Hadoop's S3
>>>>>>>>> filesystem, that translates to S3 bucket list operations,
where there is a
>>>>>>>>> limit in how many operation may happen per time interval.
After that, S3
>>>>>>>>> blocks.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It seems that operations that are totally cheap on HDFS
are
>>>>>>>>> hellishly expensive (and limited) on S3. It may be that
you are affected by
>>>>>>>>> that.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> We are gradually trying to improve the behavior there
and be more
>>>>>>>>> S3 aware.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Both 1.3-SNAPSHOT and 1.2-SNAPSHOT already contain improvements
>>>>>>>>> there.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>>
>>>>>>>>> Stephan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Feb 24, 2017 at 4:42 PM, vinay patil <[hidden
email]
>>>>>>>>> <http://user/SendEmail.jtp?type=node&node=11891&i=0>>
wrote:
>>>>>>>>>
>>>>>>>>> Hi Stephan,
>>>>>>>>>
>>>>>>>>> So do you mean that S3 is causing the stall , as I have
mentioned
>>>>>>>>> in my previous mail, I could not see any progress for
16minutes as
>>>>>>>>> checkpoints were getting failed continuously.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Feb 24, 2017 8:30 PM, "Stephan Ewen [via Apache Flink
User
>>>>>>>>> Mailing List archive.]" <[hidden email]
>>>>>>>>> <http://user/SendEmail.jtp?type=node&node=11887&i=0>>
wrote:
>>>>>>>>>
>>>>>>>>> Hi Vinay!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> True, the operator state (like Kafka) is currently not
>>>>>>>>> asynchronously checkpointed.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> While it is rather small state, we have seen before that
on S3 it
>>>>>>>>> can cause trouble, because S3 frequently stalls uploads
of even data
>>>>>>>>> amounts as low as kilobytes due to its throttling policies.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> That would be a super important fix to add!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>>
>>>>>>>>> Stephan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Feb 24, 2017 at 2:58 PM, vinay patil <[hidden
email]
>>>>>>>>> <http://user/SendEmail.jtp?type=node&node=11885&i=0>>
wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I have attached a snapshot for reference:
>>>>>>>>> As you can see all the 3 checkpointins failed , for checkpoint
ID
>>>>>>>>> 2 and 3 it
>>>>>>>>> is stuck at the Kafka source after 50%
>>>>>>>>> (The data sent till now by Kafka source 1 is 65GB and
sent by
>>>>>>>>> source 2 is
>>>>>>>>> 15GB )
>>>>>>>>>
>>>>>>>>> Within 10minutes 15M records were processed, and for
the next
>>>>>>>>> 16minutes the
>>>>>>>>> pipeline is stuck , I don't see any progress beyond 15M
because of
>>>>>>>>> checkpoints getting failed consistently.
>>>>>>>>>
>>>>>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.na
>>>>>>>>> bble.com/file/n11882/Checkpointing_Failed.png>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> View this message in context: http://apache-flink-user-maili
>>>>>>>>> ng-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-
>>>>>>>>> RocksDB-as-statebackend-tp11752p11882.html
>>>>>>>>>
>>>>>>>>> Sent from the Apache Flink User Mailing List archive.
mailing list
>>>>>>>>> archive at Nabble.com.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ------------------------------
>>>>>>>>>
>>>>>>>>> *If you reply to this email, your message will be added
to the
>>>>>>>>> discussion below:*
>>>>>>>>>
>>>>>>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>>>>>>>> ble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp1175
>>>>>>>>> 2p11885.html
>>>>>>>>>
>>>>>>>>> To start a new topic under Apache Flink User Mailing
List
>>>>>>>>> archive., email [hidden email]
>>>>>>>>> <http://user/SendEmail.jtp?type=node&node=11887&i=1>
>>>>>>>>> To unsubscribe from Apache Flink User Mailing List archive.,
click
>>>>>>>>> here.
>>>>>>>>> NAML
>>>>>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ------------------------------
>>>>>>>>>
>>>>>>>>> View this message in context: Re: Checkpointing with
RocksDB as
>>>>>>>>> statebackend
>>>>>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p11887.html>
>>>>>>>>>
>>>>>>>>> Sent from the Apache Flink User Mailing List archive.
mailing
>>>>>>>>> list archive
>>>>>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>>>>>>>> at Nabble.com.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ------------------------------
>>>>>>>>>
>>>>>>>>> *If you reply to this email, your message will be added
to the
>>>>>>>>> discussion below:*
>>>>>>>>>
>>>>>>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>>>>>>>> ble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp1175
>>>>>>>>> 2p11891.html
>>>>>>>>>
>>>>>>>>> To start a new topic under Apache Flink User Mailing
List
>>>>>>>>> archive., email [hidden email]
>>>>>>>>> To unsubscribe from Apache Flink User Mailing List archive.,
click
>>>>>>>>> here.
>>>>>>>>> NAML
>>>>>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ------------------------------
>>>>>>>>>
>>>>>>>>> View this message in context: Re: Checkpointing with
RocksDB as
>>>>>>>>> statebackend
>>>>>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p11913.html>
>>>>>>>>> Sent from the Apache Flink User Mailing List archive.
mailing
>>>>>>>>> list archive
>>>>>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>>>>>>>> at Nabble.com.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ------------------------------
>>>>>>>>>
>>>>>>>>> *If you reply to this email, your message will be added
to the
>>>>>>>>> discussion below:*
>>>>>>>>>
>>>>>>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>>>>>>>> ble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp1175
>>>>>>>>> 2p11943.html
>>>>>>>>>
>>>>>>>>> To start a new topic under Apache Flink User Mailing
List
>>>>>>>>> archive., email [hidden email]
>>>>>>>>> To unsubscribe from Apache Flink User Mailing List archive.,
click
>>>>>>>>> here.
>>>>>>>>> NAML
>>>>>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ------------------------------
>>>>>>>>>
>>>>>>>>> View this message in context: Re: Checkpointing with
RocksDB as
>>>>>>>>> statebackend
>>>>>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p11949.html>
>>>>>>>>> Sent from the Apache Flink User Mailing List archive.
mailing
>>>>>>>>> list archive
>>>>>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>>>>>>>> at Nabble.com.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> ------------------------------
>>>>> If you reply to this email, your message will be added to the
>>>>> discussion below:
>>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>>>> ble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp1175
>>>>> 2p12209.html
>>>>> To start a new topic under Apache Flink User Mailing List archive.,
>>>>> email [hidden email]
>>>>> <http:///user/SendEmail.jtp?type=node&node=12224&i=1>
>>>>> To unsubscribe from Apache Flink User Mailing List archive., click
>>>>> here.
>>>>> NAML
>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> View this message in context: Re: Checkpointing with RocksDB as
>>>> statebackend
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p12224.html>
>>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>>> archive
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>>> at Nabble.com.
>>>>
>>>
>>>
>>>
>>> ------------------------------
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>> ble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p12225.html
>>> To start a new topic under Apache Flink User Mailing List archive.,
>>> email [hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=12234&i=1>
>>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>>> NAML
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>
>>
>> ------------------------------
>> View this message in context: Re: Checkpointing with RocksDB as
>> statebackend
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p12234.html>
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>> at Nabble.com.
>>
>
>

Mime
View raw message