flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stavros Kontopoulos (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2147) Approximate calculation of frequencies in data streams
Date Mon, 16 May 2016 13:26:12 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15284476#comment-15284476

Stavros Kontopoulos commented on FLINK-2147:

Ok i agree then we calculate statistics per window in isolated manner like sum, mean etc without
the aggregation in buffer. Ok so lets see how we avoid that correct?

> Approximate calculation of frequencies in data streams
> ------------------------------------------------------
>                 Key: FLINK-2147
>                 URL: https://issues.apache.org/jira/browse/FLINK-2147
>             Project: Flink
>          Issue Type: New Feature
>          Components: Streaming
>            Reporter: Gabor Gevay
>              Labels: approximate, statistics
> Count-Min sketch is a hashing-based algorithm for approximately keeping track of the
frequencies of elements in a data stream. It is described by Cormode et al. in the following
> http://dimacs.rutgers.edu/~graham/pubs/papers/cmsoft.pdf
> Note that this algorithm can be conveniently implemented in a distributed way, as described
in section 3.2 of the paper.
> The paper
> http://www.vldb.org/conf/2002/S10P03.pdf
> also describes algorithms for approximately keeping track of frequencies, but here the
user can specify a threshold below which she is not interested in the frequency of an element.
The error-bounds are also different than the Count-min sketch algorithm.

This message was sent by Atlassian JIRA

View raw message