spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sro...@apache.org
Subject spark git commit: Fix example of tf_idf with minDocFreq
Date Mon, 17 Oct 2016 09:56:43 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-2.0 d1a021178 -> a0d9015b3


Fix example of tf_idf with minDocFreq

## What changes were proposed in this pull request?

The python example for tf_idf with the parameter "minDocFreq" is not properly set up because
the same variable is used to transform the document for both with and without the "minDocFreq"
parameter.
The IDF(minDocFreq=2) is stored in the variable "idfIgnore" but then it is the original variable
"idf" used to transform the "tf" instead of the "idfIgnore".

## How was this patch tested?

Before the results for "tfidf" and "tfidfIgnore" were the same:
tfidf:
(1048576,[1046921],[3.75828890549])
(1048576,[1046920],[3.75828890549])
(1048576,[1046923],[3.75828890549])
(1048576,[892732],[3.75828890549])
(1048576,[892733],[3.75828890549])
(1048576,[892734],[3.75828890549])
tfidfIgnore:
(1048576,[1046921],[3.75828890549])
(1048576,[1046920],[3.75828890549])
(1048576,[1046923],[3.75828890549])
(1048576,[892732],[3.75828890549])
(1048576,[892733],[3.75828890549])
(1048576,[892734],[3.75828890549])

After the fix those are how they should be:
tfidf:
(1048576,[1046921],[3.75828890549])
(1048576,[1046920],[3.75828890549])
(1048576,[1046923],[3.75828890549])
(1048576,[892732],[3.75828890549])
(1048576,[892733],[3.75828890549])
(1048576,[892734],[3.75828890549])
tfidfIgnore:
(1048576,[1046921],[0.0])
(1048576,[1046920],[0.0])
(1048576,[1046923],[0.0])
(1048576,[892732],[0.0])
(1048576,[892733],[0.0])
(1048576,[892734],[0.0])

Author: Maxime Rihouey <maxime.rihouey@gmail.com>

Closes #15503 from maximerihouey/patch-1.

(cherry picked from commit e3bf37fa3ada43624b2e77bef90ad3d3dbcd8ce1)
Signed-off-by: Sean Owen <sowen@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a0d9015b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a0d9015b
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a0d9015b

Branch: refs/heads/branch-2.0
Commit: a0d9015b3f34582c5d43bd31bbf35a0e92b1da29
Parents: d1a0211
Author: Maxime Rihouey <maxime.rihouey@gmail.com>
Authored: Mon Oct 17 10:56:22 2016 +0100
Committer: Sean Owen <sowen@cloudera.com>
Committed: Mon Oct 17 10:56:37 2016 +0100

----------------------------------------------------------------------
 examples/src/main/python/mllib/tf_idf_example.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/a0d9015b/examples/src/main/python/mllib/tf_idf_example.py
----------------------------------------------------------------------
diff --git a/examples/src/main/python/mllib/tf_idf_example.py b/examples/src/main/python/mllib/tf_idf_example.py
index c4d5333..b66412b 100644
--- a/examples/src/main/python/mllib/tf_idf_example.py
+++ b/examples/src/main/python/mllib/tf_idf_example.py
@@ -43,7 +43,7 @@ if __name__ == "__main__":
     # In such cases, the IDF for these terms is set to 0.
     # This feature can be used by passing the minDocFreq value to the IDF constructor.
     idfIgnore = IDF(minDocFreq=2).fit(tf)
-    tfidfIgnore = idf.transform(tf)
+    tfidfIgnore = idfIgnore.transform(tf)
     # $example off$
 
     print("tfidf:")


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message