spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jkbradley <...@git.apache.org>
Subject [GitHub] spark pull request: [SPARK-1545] [mllib] Add Random Forests
Date Thu, 25 Sep 2014 19:15:07 GMT
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/2435#discussion_r18053331
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/tree/impl/BaggedPoint.scala ---
    @@ -0,0 +1,80 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.mllib.tree.impl
    +
    +import cern.jet.random.Poisson
    +import cern.jet.random.engine.DRand
    +import org.apache.spark.rdd.RDD
    +import org.apache.spark.util.Utils
    +
    +/**
    + * Internal representation of a datapoint which belongs to several subsamples of the
same dataset,
    + * particularly for bagging (e.g., for random forests).
    + *
    + * This holds one instance, as well as an array of weights which represent the (weighted)
    + * number of times which this instance appears in each subsample.
    + * E.g., (datum, [1, 0, 4]) indicates that there are 3 subsamples of the dataset and
that
    + * this datum has 1 copy, 0 copies, and 4 copies in the 3 subsamples, respectively.
    + *
    + * @param datum  Data instance
    + * @param subsampleWeights  Weight of this instance in each subsampled dataset.
    + *
    + * TODO: This does not currently support (Double) weighted instances.  Once MLlib has
weighted
    + *       dataset support, update.  (We store subsampleWeights as Double for this future
extension.)
    + */
    +private[tree] class BaggedPoint[Datum](val datum: Datum, val subsampleWeights: Array[Double])
    +  extends Serializable {
    +}
    +
    +private[tree] object BaggedPoint {
    +
    +  /**
    +   * Convert an input dataset into its BaggedPoint representation,
    +   * choosing subsample counts for each instance.
    +   * Each subsample has the same number of instances as the original dataset,
    +   * and is created by subsampling with replacement.
    +   * @param input     Input dataset.
    +   * @param numSubsamples  Number of subsamples of this RDD to take.
    +   * @param seed   Random seed.
    +   * @return  BaggedPoint dataset representation
    +   */
    +  def convertToBaggedRDD[Datum](
    +      input: RDD[Datum],
    +      numSubsamples: Int,
    +      seed: Int = Utils.random.nextInt()): RDD[BaggedPoint[Datum]] = {
    --- End diff --
    
    OK, but that seems a little deceptive in terms of API.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message