flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Lublinsky <boris.lublin...@lightbend.com>
Subject Re: Stateful functions Harness
Date Wed, 27 May 2020 20:20:38 GMT
Thanks Seth
Will take a look.


> On May 27, 2020, at 3:15 PM, Seth Wiesman <sjwiesman@gmail.com> wrote:
> 
> Hi Boris, 
> 
> Example usage of flink sources and sink is available in the documentation[1]. 
> 
> [1] https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/io-module/flink-connectors.html
<https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/io-module/flink-connectors.html>
> On Wed, May 27, 2020 at 1:08 PM Boris Lublinsky <boris.lublinsky@lightbend.com <mailto:boris.lublinsky@lightbend.com>>
wrote:
> Thats not exactly the usage question that I am asking
> When I am writing IO module I have to write Ingress and Egress spec.
> You have an example for Kafka, which looks like
> 
> def getIngressSpec: IngressSpec[GreetRequest] =
>   KafkaIngressBuilder.forIdentifier(GREETING_INGRESS_ID)
>     .withKafkaAddress(kafkaAddress)
>     .withTopic("names")
>     .withDeserializer(classOf[GreetKafkaDeserializer])
>     .withProperty(ConsumerConfig.GROUP_ID_CONFIG, "greetings")
>     .build
> 
> def getEgressSpec: EgressSpec[GreetResponse] =
>   KafkaEgressBuilder.forIdentifier(GREETING_EGRESS_ID)
>     .withKafkaAddress(kafkaAddress)
>     .withSerializer(classOf[GreetKafkaSerializer])
>     .build
> How is it going to look if I am using SourceSinkModule?
> Do I just specify stream names? Something else?
> 
> 
> 
> 
> 
>> On May 27, 2020, at 11:29 AM, Tzu-Li (Gordon) Tai <tzulitai@apache.org <mailto:tzulitai@apache.org>>
wrote:
>> 
>> 
>> 
>> On Thu, May 28, 2020, 12:19 AM Boris Lublinsky <boris.lublinsky@lightbend.com
<mailto:boris.lublinsky@lightbend.com>> wrote:
>> I think I figured this out.
>> The project seems to be missing
>> 
>> resources <https://github.com/ververica/flink-statefun-workshop/tree/master/statefun-workshop-functions/src/main/resources>/META-INF
<https://github.com/ververica/flink-statefun-workshop/tree/master/statefun-workshop-functions/src/main/resources/META-INF>/services
directory, which should contain services
>> 
>> Yes, the functions / ingresses / regresses etc. are not discoverable if the service
file isnt present in the classpath.
>> 
>> For the examples, if you are running it straight from the repo, should all have that
service file defined and therefore readily runnable.
>> 
>> If you are creating your own application project, you'll have to add that yourself.
>> 
>> 
>> Another question:
>> I see org.apache.flink.statefun.flink.io.datastream.SourceSinkModule
>> 
>> Class, which I think allows to use existing data streams as ingress/egress.
>> 
>> Are there any examples of its usage
>> 
>> On the Harness class, there is a withFlinkSourceFunction method in which you can
directly add a Flink source function as the ingress.
>> 
>> If you want to use that directly in a normal application (not just execution in IDE
with the Harness), you can define your ingesses/egresses by binding SourceFunctionSpec / SinkFunctionSpec.
>> Please see how they are being used in the Harness class for examples.
>> 
>> Gordon
>> 
>> 
>> 
>>> On May 27, 2020, at 11:10 AM, Tzu-Li (Gordon) Tai <tzulitai@apache.org <mailto:tzulitai@apache.org>>
wrote:
>>> 
>>> Hi,
>>> 
>>> The example is working fine on my side (also using IntelliJ).
>>> This could most likely be a problem with your project setup in the IDE, where
the classpath isn't setup correctly.
>>> 
>>> What do you see when you right click on the statefun-flink-harness-example directory
(in the IDE) --> Open Module Settings, and then under the "Sources" / "Dependencies" tab?
>>> Usually this should all be automatically setup correctly when importing the project.
>>> 
>>> Gordon
>>> 
>>> On Wed, May 27, 2020 at 11:46 PM Boris Lublinsky <boris.lublinsky@lightbend.com
<mailto:boris.lublinsky@lightbend.com>> wrote:
>>> The project https://github.com/apache/flink-statefun/tree/release-2.0/statefun-examples/statefun-flink-harness-example
<https://github.com/apache/flink-statefun/tree/release-2.0/statefun-examples/statefun-flink-harness-example>
>>> Does not work in Intellij.
>>> 
>>> The problem is that when running in Intellij, method public static Modules loadFromClassPath()
{
>>> Does not pick up classes, which are local in Intellij
>>> 
>>> Any work arounds?
>>> 
>>> 
>>> 
>>> 
>>>> On May 22, 2020, at 12:03 AM, Tzu-Li (Gordon) Tai <tzulitai@apache.org
<mailto:tzulitai@apache.org>> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> Sorry, I need to correct my comment on using the Kafka ingress / egress with
the Harness.
>>>> 
>>>> That is actually doable, by adding an extra dependency to `statefun-flink-distribution`
in your Harness program.
>>>> That pulls in all the other required dependencies required by the Kafka ingress
/ egress, such as the source / sink providers and Flink Kafka connectors.
>>>> 
>>>> Cheers,
>>>> Gordon
>>>> 
>>>> On Fri, May 22, 2020 at 12:04 PM Tzu-Li (Gordon) Tai <tzulitai@apache.org
<mailto:tzulitai@apache.org>> wrote:
>>>> Are you getting an exception from running the Harness?
>>>> The Harness should already have the required configurations, such as the
parent first classloading config.
>>>> 
>>>> Otherwise, if you would like to add your own configuration, use the `withConfiguration`
method on the `Harness` class.
>>>> 
>>>> On Fri, May 22, 2020 at 7:19 AM Boris Lublinsky <boris.lublinsky@lightbend.com
<mailto:boris.lublinsky@lightbend.com>> wrote:
>>>> Also, where do I put flint-conf.yaml in Idea to add additional required config
parameter:
>>>> classloader.parent-first-patterns.additional: org.apache.flink.statefun;org.apache.kafka;com.google.protobuf
>>>> 
>>>> 
>>>>> On May 21, 2020, at 12:22 PM, Boris Lublinsky <boris.lublinsky@lightbend.com
<mailto:boris.lublinsky@lightbend.com>> wrote:
>>>>> 
>>>>> Hi, 
>>>>> I am trying to run https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-greeter-example
<https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-greeter-example>
locally
>>>>> using https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-flink-harness-example
<https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-flink-harness-example>

>>>>> And have several questions.
>>>>> 1. It seems fairly straightforward to use it with in memory message generators,
but I can’t figure out how to add Kafka ingress/Egress so that I can use it with Kafk
>>>>> 2. GreetingModule already creates StatefulFunctionUniverse  and so does
Harness. Is there a way to short circuit it and have Harness get StatefulFunctionUniverse
directly
>>>>> 3. Is there an example on how to write Flink main for stageful function?
>>>>> 4. Is there an example anywhere on how to run such examples in the IDE
with Kafka?
>>>>> 5 There is a great stateful functions example https://github.com/ververica/flink-statefun-workshop
<https://github.com/ververica/flink-statefun-workshop>, but its readme does not really
describe implementation and neither does this article, referencing it https://dev.to/morsapaes/flink-stateful-functions-where-to-start-2j39
<https://dev.to/morsapaes/flink-stateful-functions-where-to-start-2j39>. Is there anything
that describes this implementation?
>>>>> 
>>>> 
>>> 
>> 
> 


Mime
View raw message