flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-9642) Reduce the count to deal with state during a CEP process
Date Fri, 10 Aug 2018 13:17:02 GMT

    [ https://issues.apache.org/jira/browse/FLINK-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576245#comment-16576245
] 

ASF GitHub Bot commented on FLINK-9642:
---------------------------------------

dawidwys commented on a change in pull request #6205: [FLINK-9642]Reduce the count to deal
with state during a CEP process
URL: https://github.com/apache/flink/pull/6205#discussion_r209232519
 
 

 ##########
 File path: flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/sharedbuffer/SharedBufferAccessor.java
 ##########
 @@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOVICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  Vhe ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cep.nfa.sharedbuffer;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.cep.nfa.DeweyNumber;
+import org.apache.flink.util.WrappingRuntimeException;
+
+import org.apache.commons.lang3.StringUtils;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Stack;
+
+import static org.apache.flink.cep.nfa.compiler.NFAStateNameHandler.getOriginalNameFromInternal;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A shared buffer implementation which stores values under according state. Additionally,
the values can be
+ * versioned such that it is possible to retrieve their predecessor element in the buffer.
+ *
+ * <p>The idea of the implementation is to have a buffer for incoming events with unique
ids assigned to them. This way
+ * we do not need to deserialize events during processing and we store only one copy of the
event.
+ *
+ * <p>The entries in {@link SharedBufferAccessor} are {@link SharedBufferNode}. The
shared buffer node allows to store
+ * relations between different entries. A dewey versioning scheme allows to discriminate
between
+ * different relations (e.g. preceding element).
+ *
+ * <p>The implementation is strongly based on the paper "Efficient Pattern Matching
over Event Streams".
+ *
+ * @param <V> Type of the values
+ * @see <a href="https://people.cs.umass.edu/~yanlei/publications/sase-sigmod08.pdf">
+ * https://people.cs.umass.edu/~yanlei/publications/sase-sigmod08.pdf</a>
+ */
+public class SharedBufferAccessor<V> implements AutoCloseable{
+
+	/** The cache of sharedBuffer.*/
+	private SharedBuffer<V> sharedBuffer;
+
+	public SharedBufferAccessor(SharedBuffer<V> sharedBuffer) {
+		this.sharedBuffer = sharedBuffer;
+	}
+
+	public void setSharedBuffer(SharedBuffer<V> sharedBuffer) {
 
 Review comment:
   This method is unnecessary.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Reduce the count to deal with state during a CEP process 
> ---------------------------------------------------------
>
>                 Key: FLINK-9642
>                 URL: https://issues.apache.org/jira/browse/FLINK-9642
>             Project: Flink
>          Issue Type: Improvement
>          Components: CEP
>    Affects Versions: 1.6.0
>            Reporter: aitozi
>            Assignee: aitozi
>            Priority: Major
>              Labels: pull-request-available
>
> With the rework of sharedBuffer Flink-9418, the lock & release operation is deal
with rocksdb state which is different from the previous version which will read the state
of sharedBuffer all to memory, i think we can add a cache or variable in sharedbuffer to cache
the LockAble Object to mark the ref change in once process in NFA, this will reduce the count
when the events point to the same NodeId.. And flush the result to MapState at the end of
process. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message