spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Qi Dai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-13289) Word2Vec generate infinite distances when numIterations>5
Date Mon, 22 Feb 2016 18:11:18 GMT

    [ https://issues.apache.org/jira/browse/SPARK-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157401#comment-15157401
] 

Qi Dai commented on SPARK-13289:
--------------------------------

Do you have a runable distribution of the latest master somewhere? I tried to build it but
didn't get through. It failed at building spark-catalyst_2.11 with the following error: Failed
to execute goal org.apache.maven.plugins:maven-shade-plugin:2.4.3:shade (default) on project
spark-catalyst_2.11: Error creating shaded jar: Method code too large! -> [Help 1] 

I was building with "build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -Phive -Phive-thriftserver
-DskipTests clean package"

I tried "export MAVEN_OPTS="-Xmx4g -XX:MaxPermSize=4g -XX:ReservedCodeCacheSize=2g"" and "export
MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"" 

I'm using java 1.7.80 on Ubuntu 14.04.3

> Word2Vec generate infinite distances when numIterations>5
> ---------------------------------------------------------
>
>                 Key: SPARK-13289
>                 URL: https://issues.apache.org/jira/browse/SPARK-13289
>             Project: Spark
>          Issue Type: Bug
>          Components: MLlib
>    Affects Versions: 1.6.0
>         Environment: Linux, Scala
>            Reporter: Qi Dai
>              Labels: features
>
> I recently ran some word2vec experiments on a cluster with 50 executors on some large
text dataset but find out that when number of iterations is larger than 5 the distance between
words will be all infinite. My code looks like this:
> val text = sc.textFile("/project/NLP/1_biliion_words/train").map(_.split(" ").toSeq)
> import org.apache.spark.mllib.feature.{Word2Vec, Word2VecModel}
> val word2vec = new Word2Vec().setMinCount(25).setVectorSize(96).setNumPartitions(99).setNumIterations(10).setWindowSize(5)
> val model = word2vec.fit(text)
> val synonyms = model.findSynonyms("who", 40)
> for((synonym, cosineSimilarity) <- synonyms) {
>   println(s"$synonym $cosineSimilarity")
> }
> The results are: 
> to Infinity
> and Infinity
> that Infinity
> with Infinity
> said Infinity
> it Infinity
> by Infinity
> be Infinity
> have Infinity
> he Infinity
> has Infinity
> his Infinity
> an Infinity
> ) Infinity
> not Infinity
> who Infinity
> I Infinity
> had Infinity
> their Infinity
> were Infinity
> they Infinity
> but Infinity
> been Infinity
> I tried many different datasets and different words for finding synonyms.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message