Mailing-List: contact issues-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
From: sachingoel0101 <git@git.apache.org>
To: issues@flink.incubator.apache.org
Reply-To: issues@flink.incubator.apache.org
References: <git-pr-861-flink@git.apache.org>
In-Reply-To: <git-pr-861-flink@git.apache.org>
Subject: [GitHub] flink pull request: [FLINK-2030][FLINK-2274][ml]Online
 Histograms ...
Content-Type: text/plain
Message-Id: <20150820173726.E5AC4E7DD9@git1-us-west.apache.org>
Date: Thu, 20 Aug 2015 17:37:26 +0000 (UTC)

Github user sachingoel0101 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/861#discussion_r37557797
  
    --- Diff: flink-staging/flink-ml/src/main/scala/org/apache/flink/ml/statistics/ContinuousHistogram.scala ---
    @@ -0,0 +1,315 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.ml.statistics
    +
    +import scala.Double.MaxValue
    +import scala.collection.mutable
    +
    +/** Implementation of a continuous valued online histogram
    +  * Adapted from Ben-Haim and Yom-Tov's Streaming Decision Tree Algorithm
    +  * Refer http://www.jmlr.org/papers/volume11/ben-haim10a/ben-haim10a.pdf
    +  *
    +  * =Parameters=
    +  * -[[capacity]]:
    +  * Number of bins to be used in the histogram
    +  *
    +  * -[[min]]:
    +  * Lower limit on the elements
    +  *
    +  * -[[max]]:
    +  * Upper limit on the elements
    +  */
    +class ContinuousHistogram(capacity: Int, min: Double, max: Double) extends OnlineHistogram {
    +
    +  private val lower = min
    +  private val upper = max
    +
    +  require(capacity > 0, "Capacity should be a positive integer")
    +  require(lower < upper, "Lower must be less than upper")
    +
    +  val data = new mutable.ArrayBuffer[(Double, Int)]()
    +
    +  /** Adds a new item to the histogram
    +    *
    +    * @param p value to be added
    +    */
    +  override def add(p: Double): Unit = {
    +    require(p > lower && p < upper, p + " not in (" + lower + "," + upper + ")")
    +    // search for the index where the value is just higher than p
    +    val search = find(p)
    +    // add the new value there, shifting everything to the right
    +    data.insert(search, (p, 1))
    +    // If we're over capacity or any two elements are within 1e-9 of each other, merge.
    +    // This will take care of the case if p was actually equal to some value in the histogram and
    +    // just increment the value there
    +    mergeElements()
    +  }
    +
    +  /** Merges the histogram with h and returns a histogram with capacity B
    +    *
    +    * @param h histogram to be merged
    +    * @param B capacity of the resultant histogram
    +    * @return Merged histogram with capacity B
    +    */
    +  override def merge(h: OnlineHistogram, B: Int): ContinuousHistogram = {
    +    h match {
    +      case temp: ContinuousHistogram => {
    +        val m: Int = bins
    +        val n: Int = temp.bins
    +        var i, j: Int = 0
    +        val mergeList = new mutable.ArrayBuffer[(Double, Int)]()
    +        while (i < m || j < n) {
    +          if (i >= m) {
    +            mergeList += ((temp.getValue(j), temp.getCounter(j)))
    +            j = j + 1
    +          } else if (j >= n || getValue(i) <= temp.getValue(j)) {
    +            mergeList += data.apply(i)
    +            i = i + 1
    +          } else {
    +            mergeList += ((temp.getValue(j), temp.getCounter(j)))
    +            j = j + 1
    +          }
    +        }
    +        // the size will be brought to capacity while constructing the new histogram itself
    +        val finalLower = Math.min(lower, temp.lower)
    +        val finalUpper = Math.max(upper, temp.upper)
    +        val ret = new ContinuousHistogram(B, finalLower, finalUpper)
    +        ret.loadData(mergeList.toArray)
    --- End diff --
    
    Yes. I'm working on this. The best way to optimize this would be to change the signature and have `merge` actually modify the current histogram. This would be minimally expensive in terms of copying the data. But comes at the cost of having to duplicate the quantities `min`, `max` and `capacity`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---