flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vino yang <yanghua1...@gmail.com>
Subject Re: processWindowFunction
Date Mon, 20 Aug 2018 09:55:50 GMT
Hi antonio,

Oh, if you can't use KeyedProcessFunction, then this would be a pity.
Then you can use MapState, where Key is used to store the key of your
partition.
But I am not sure if this will achieve the effect you want.

Thanks, vino.

antonio saldivar <ansale10@gmail.com> 于2018年8月20日周一 下午4:32写道:

> Hello
>
> Thank you for the information, for some reason this KeyedProcessFunction
> is not found in my Flink version 1.4.2 I can only find ProcessFunction and
> work like this
>
> public class TxnProcessFn extends ProcessFunction<Object,Object> {
>
>  public void open(Configuration parameters) throws Exception {
>
> state1 = getRuntimeContext().getState(new ValueStateDescriptor<>(
> "objState1", Object.class));
>
> state2 = getRuntimeContext().getState(new ValueStateDescriptor<>(
> "objState2", Object.class));
>
> state3 = getRuntimeContext().getState(new ValueStateDescriptor<>(
> "objState3", Object.class));
>
> }
>
> @Override
>
> public void processElement(
>
> Object obj,
>
> Context ctx,
>
> Collector<Transaction> out) throws Exception {
>
> // TODO Auto-generated method stub
>
> Object current = state.value();
>
> if (current == null) {
>
>             current = new Object();
>
>             current.id=obj.id();
>
>
>
>         }
>
> }
>
> El lun., 20 ago. 2018 a las 2:24, vino yang (<yanghua1127@gmail.com>)
> escribió:
>
>> Hi antonio,
>>
>> First, I suggest you use KeyedProcessFunction if you have an operation
>> similar to keyBy.
>> The implementation is similar to the Fixed window.
>> You can create three state collections to determine whether the time of
>> each element belongs to a state collection.
>> At the time of the trigger, the elements in the collection are evaluated.
>>
>> Thanks, vino.
>>
>> antonio saldivar <ansale10@gmail.com> 于2018年8月20日周一 上午11:54写道:
>>
>>> Thank you fro the references
>>>
>>> I have now my processFunction and getting the state but now how can i do
>>> for the threshold times to group the elements and also as this is a global
>>> window, how to purge because if going to keep increasing
>>>
>>> El dom., 19 ago. 2018 a las 8:57, vino yang (<yanghua1127@gmail.com>)
>>> escribió:
>>>
>>>> Hi antonio,
>>>>
>>>> Regarding your scenario, I think maybe you can consider using the
>>>> ProcessFunction (or keyed ProcessFunction) function directly on the Stream.
>>>> [1]
>>>> It can handle each of your elements with a Timer, and you can combine
>>>> Flink's state API[2] to store your data.
>>>>
>>>> [1]:
>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#process-function-low-level-operations
>>>> [2]:
>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/state/state.html#working-with-state
>>>>
>>>> Thanks, vino.
>>>>
>>>> antonio saldivar <ansale10@gmail.com> 于2018年8月19日周日 上午10:18写道:
>>>>
>>>>> hi Vino
>>>>>
>>>>> it is possible to use global window, then set the trigger onElement
>>>>> comparing the element that has arrived with for example 10 mins, 20 mins
>>>>> and 60 mins of data?
>>>>>
>>>>> I have rules evaluating sum of amount for 10,20 or 60 mins for the
>>>>> same keyed element if the same id sum like $200 total within those
>>>>> thresholds and count more or equals to 3 I need to be able to set some
>>>>> values to the object if the object does not reach those thresholds i
do not
>>>>> set the values and keep sending the output with or without those value.
>>>>>
>>>>> just processing the object on the fly and send output
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> El vie., 17 ago. 2018 a las 22:14, vino yang (<yanghua1127@gmail.com>)
>>>>> escribió:
>>>>>
>>>>>> Hi antonio,
>>>>>>
>>>>>> Yes, ProcessWindowFunction is a very low level window function.
>>>>>> It allows you to access the data in the window and allows you to
>>>>>> customize the output of the window.
>>>>>> So if you use it, while giving you flexibility, you need to think
>>>>>> about other things, which may require you to write more processing
logic.
>>>>>>
>>>>>> Generally speaking, sliding windows usually have some data that is
>>>>>> repeated, but a common mode is to apply a reduce function on it to
get your
>>>>>> calculation results.
>>>>>> If you only send data, there will definitely be some duplication.
>>>>>>
>>>>>> Thanks, vino.
>>>>>>
>>>>>> antonio saldivar <ansale10@gmail.com> 于2018年8月17日周五
下午12:01写道:
>>>>>>
>>>>>>> Hi Vino
>>>>>>> thank you for the information, actually I am using a trigger
alert
>>>>>>> and processWindowFunction to send my results, but when my window
slides or
>>>>>>> ends it sends again the objects and I an getting duplicated data
>>>>>>>
>>>>>>> El jue., 16 ago. 2018 a las 22:05, vino yang (<yanghua1127@gmail.com>)
>>>>>>> escribió:
>>>>>>>
>>>>>>>> Hi Antonio,
>>>>>>>>
>>>>>>>> What results do not you want to get when creating each window?
>>>>>>>> Examples of the use of ProcessWindowFunction are included
in many
>>>>>>>> test files in Flink's project, such as SideOutputITCase.scala
or
>>>>>>>> WindowTranslationTest.scala.
>>>>>>>>
>>>>>>>> For more information on ProcessWindowFunction, you can refer
to the
>>>>>>>> official website.[1]
>>>>>>>>
>>>>>>>> [1]:
>>>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/windows.html#processwindowfunction
>>>>>>>>
>>>>>>>> Thanks, vino.
>>>>>>>>
>>>>>>>> antonio saldivar <ansale10@gmail.com> 于2018年8月17日周五
上午6:24写道:
>>>>>>>>
>>>>>>>>> Hello
>>>>>>>>>
>>>>>>>>> I am implementing a data stream where I use sliding windows
but I
>>>>>>>>> am stuck because I need to set values to my object based
on some if
>>>>>>>>> statements in my process function  and send the object
to the next step but
>>>>>>>>> I don't want results every time a window is creating
>>>>>>>>>
>>>>>>>>> if anyone has a good example on this that can help me
>>>>>>>>>
>>>>>>>>

Mime
View raw message