spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yanboliang <...@git.apache.org>
Subject [GitHub] spark pull request: [SPARK-11940][PYSPARK] Python API for ml.clust...
Date Mon, 29 Feb 2016 10:51:54 GMT
Github user yanboliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10242#discussion_r54394622
  
    --- Diff: python/pyspark/ml/clustering.py ---
    @@ -167,6 +167,200 @@ def getInitSteps(self):
             return self.getOrDefault(self.initSteps)
     
     
    +class LDAModel(JavaModel):
    +    """ A clustering model derived from the LDA method.
    +
    +    Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
    +    Terminology
    +    - "word" = "term": an element of the vocabulary
    +    - "token": instance of a term appearing in a document
    +    - "topic": multinomial distribution over words representing some concept
    +    References:
    +    - Original LDA paper (journal version):
    +    Blei, Ng, and Jordan.  "Latent Dirichlet Allocation."  JMLR, 2003.
    +    """
    +
    +    @since("1.7.0")
    +    def isDistributed(self):
    +        """Indicates whether this instance is of type DistributedLDAModel"""
    +        return self._call_java("isDistributed")
    +
    +    @since("1.7.0")
    +    def vocabSize(self):
    +        """Vocabulary size (number of terms or terms in the vocabulary)"""
    +        return self._call_java("vocabSize")
    +
    +    @since("1.7.0")
    +    def topicsMatrix(self):
    +        """Inferred topics, where each topic is represented by a distribution over terms."""
    +        return self._call_java("topicsMatrix")
    +
    +    @since("1.7.0")
    +    def logLikelihood(self, dataset):
    +        """Calculates a lower bound on the log likelihood of the entire corpus."""
    +        return self._call_java("logLikelihood", dataset)
    +
    +    @since("1.7.0")
    +    def logPerplexity(self, dataset):
    +        """Calculate an upper bound bound on perplexity.  (Lower is better.)"""
    +        return self._call_java("logPerplexity", dataset)
    +
    +    @since("1.7.0")
    +    def describeTopics(self, maxTermsPerTopic=10):
    +        """Return the topics described by weighted terms.
    +
    +        WARNING: If vocabSize and k are large, this can return a large object!
    +
    +        :param maxTermsPerTopic: Maximum number of terms to collect for each topic.
    +            (default: vocabulary size)
    +        :return: Array over topics. Each topic is represented as a pair of matching arrays:
    +            (term indices, term weights in topic).
    +            Each topic's terms are sorted in order of decreasing weight.
    +        """
    +        return self._call_java("describeTopics", maxTermsPerTopic)
    +
    +
    +class DistributedLDAModel(LDAModel):
    +    """
    +    Model fitted by LDA.
    +
    +    .. versionadded:: 1.7.0
    +    """
    +    def toLocal(self):
    +        return self._call_java("toLocal")
    +
    +
    +class LocalLDAModel(LDAModel):
    +    """
    +    Model fitted by LDA.
    +
    +    .. versionadded:: 1.7.0
    +    """
    +    pass
    +
    +
    +class LDA(JavaEstimator, HasFeaturesCol, HasMaxIter, HasSeed, HasCheckpointInterval):
    +    """ A clustering model derived from the LDA method.
    +
    +    Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
    +    Terminology
    +    - "word" = "term": an element of the vocabulary
    +    - "token": instance of a term appearing in a document
    +    - "topic": multinomial distribution over words representing some concept
    +    References:
    +    - Original LDA paper (journal version):
    +    Blei, Ng, and Jordan.  "Latent Dirichlet Allocation."  JMLR, 2003.
    +
    +    >>> from pyspark.mllib.linalg import Vectors, SparseVector
    +    >>> from pyspark.ml.clustering import LDA
    +    >>> df = sqlContext.createDataFrame([[1, Vectors.dense([0.0, 1.0])], \
    --- End diff --
    
    Here we usually make the next line start with ```...```, you can refer [here](https://github.com/apache/spark/blob/master/python/pyspark/ml/clustering.py#L58).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message