flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
Subject RE: is flink' states functionality futile?
Date Tue, 26 Sep 2017 11:39:25 GMT
Hi Ziv,

if I mark it as transient, then its value erased and I get null pointer exception. 
on the other hand, leave it without 'transient', it's a compilation error - implementation
of function is not serializable. 
For unserializable class fields that you need to use for processing, you need to make it transient,
and only initialize it in the open() method.

To override the open() method you will need to extend the Rich variant of the operator you
are implementing. For example, the RichFlatMapFunction is the Rich variant of the FlatMapFunction.

Does that answer your question?

Cheers,
Gordon

On 26 September 2017 at 11:58:43 AM, Meri Ziv (ziv.meri@elbitsystems.com) wrote:

hi Tzu-Li,  

sorry I have to reopen it but I'm still having troubles with using data from previous records.
 
you said:  

“If that’s the case, and you don’t care about losing state on failures and don’t care
about at-least-once / exactly-once, then yes you don’t have to use the managed state APIs
in Flink.  
You can just have ordinary fields to achieve this since Flink’s streaming operators are
basically long-running processes that continuously process events in the stream and manipulates
its state.”  

look that if one of my fields is of unserializable type, then:  
if I mark it as transient, then its value erased and I get null pointer exception.  
on the other hand, leave it without 'transient', it's a compilation error - implementation
of function is not serializable.  

how you explain that?  

Best, Ziv.  

From: Tzu-Li (Gordon) Tai [mailto:tzulitai@apache.org]  
Sent: 28 July 2017 09:12  
To: dev@flink.apache.org; Meri Ziv  
Subject: RE: is flink' states functionality futile?  

Here it is (sorry about that):  

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/internals/stream_checkpointing.html
 


On 26 July 2017 at 4:15:54 PM, ziv (ziv.meri@elbitsystems.com<mailto:ziv.meri@elbitsystems.com>)
wrote:  
Hi, what is the reference you wanted to link in [1]?  

From: Tzu-Li (Gordon) Tai [via Apache Flink Mailing List archive.] [mailto:ml+s1008284n18936h39@n3.nabble.com]<mailto:[mailto:ml+s1008284n18936h39@n3.nabble.com]>
 
Sent: 26 July 2017 05:44  
To: Meri Ziv  
Subject: Re: is flink' states functionality futile?  

<https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/state.html>
 
it's a program that sinks messages only after enough items accumulated in  
the buffe  
Yes, that’s another important aspect of how Flink’s checkpointing is used to achieve at-least-once
/ exactly-once delivery to external systems.  

For more details on how this works, I recommend taking at the docs here [1].  


Now, assume I'm not bothered of recovering failures and only want the  
simplest way to implement a program that remembers data from the last run in  
the stream  

By “the last run in the stream”, you mean the history of the stream so far, correct? 

If that’s the case, and you don’t care about losing state on failures and don’t care
about at-least-once / exactly-once, then yes you don’t have to use the managed state APIs
in Flink.  
You can just have ordinary fields to achieve this since Flink’s streaming operators are
basically long-running processes that continuously process events in the stream and manipulates
its state.  

Cheers,  
Gordon  

On 23 July 2017 at 10:28:37 PM, ziv ([hidden email]</user/SendEmail.jtp?type=node&node=18936&i=0>)
wrote:  

Ok, Let me see if I understand you correctly.  
You actually state that flink' states functionality is introduced only to  
handle recovering from failures.  
Let's take the example given in 1.3 documentary -  
https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/state.html  
<https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/state.html>
 
it's a program that sinks messages only after enough items accumulated in  
the buffer.  
Now, assume I'm not bothered of recovering failures and only want the  
simplest way to implement a program that remembers data from the last run in  
the stream, then, according to you, I may not use none of the elements  
associated with flink' states -  
ListState  
snapshotState  
initializeState  
restoreState  
and the program still functions correctly?  





--  
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/is-flink-states-functionality-futile-tp18867p18879.html
 
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.  

________________________________  
If you reply to this email, your message will be added to the discussion below:  
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/is-flink-states-functionality-futile-tp18867p18936.html
 
To start a new topic under Apache Flink Mailing List archive., email ml+s1008284n1h54@n3.nabble.com<mailto:ml+s1008284n1h54@n3.nabble.com<mailto:ml+s1008284n1h54@n3.nabble.com%3cmailto:ml+s1008284n1h54@n3.nabble.com>>
 
To unsubscribe from is flink' states functionality futile?, click here<http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=18867&code=eml2Lm1lcmlAZWxiaXRzeXN0ZW1zLmNvbXwxODg2N3wxNzE4MzA5NDcz>.
 
NAML<http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
 

The information in this e-mail transmission contains proprietary and business  
sensitive information. Unauthorized interception of this e-mail may constitute  
a violation of law. If you are not the intended recipient, you are hereby  
notified that any review, dissemination, distribution or duplication of this  
communication is strictly prohibited. You are also asked to contact the sender  
by reply email and immediately destroy all copies of the original message.  




--  
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/is-flink-states-functionality-futile-tp18867p18939.html
 
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.  

The information in this e-mail transmission contains proprietary and business  
sensitive information. Unauthorized interception of this e-mail may constitute  
a violation of law. If you are not the intended recipient, you are hereby  
notified that any review, dissemination, distribution or duplication of this  
communication is strictly prohibited. You are also asked to contact the sender  
by reply email and immediately destroy all copies of the original message.  

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message