Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 22 Feb 2016 18:11:18 +0000 (UTC)
From: "Qi Dai (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.12938623.1455225071000.111839.1456164678237@Atlassian.JIRA>
In-Reply-To: <JIRA.12938623.1455225071000@Atlassian.JIRA>
References: <JIRA.12938623.1455225071000@Atlassian.JIRA>
 <JIRA.12938623.1455225071974@arcas>
Subject: [jira] [Commented] (SPARK-13289) Word2Vec generate infinite
 distances when numIterations>5
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/SPARK-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157401#comment-15157401 ] 

Qi Dai commented on SPARK-13289:
--------------------------------

Do you have a runable distribution of the latest master somewhere? I tried to build it but didn't get through. It failed at building spark-catalyst_2.11 with the following error: Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:2.4.3:shade (default) on project spark-catalyst_2.11: Error creating shaded jar: Method code too large! -> [Help 1] 

I was building with "build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -Phive -Phive-thriftserver -DskipTests clean package"

I tried "export MAVEN_OPTS="-Xmx4g -XX:MaxPermSize=4g -XX:ReservedCodeCacheSize=2g"" and "export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"" 

I'm using java 1.7.80 on Ubuntu 14.04.3

> Word2Vec generate infinite distances when numIterations>5
> ---------------------------------------------------------
>
>                 Key: SPARK-13289
>                 URL: https://issues.apache.org/jira/browse/SPARK-13289
>             Project: Spark
>          Issue Type: Bug
>          Components: MLlib
>    Affects Versions: 1.6.0
>         Environment: Linux, Scala
>            Reporter: Qi Dai
>              Labels: features
>
> I recently ran some word2vec experiments on a cluster with 50 executors on some large text dataset but find out that when number of iterations is larger than 5 the distance between words will be all infinite. My code looks like this:
> val text = sc.textFile("/project/NLP/1_biliion_words/train").map(_.split(" ").toSeq)
> import org.apache.spark.mllib.feature.{Word2Vec, Word2VecModel}
> val word2vec = new Word2Vec().setMinCount(25).setVectorSize(96).setNumPartitions(99).setNumIterations(10).setWindowSize(5)
> val model = word2vec.fit(text)
> val synonyms = model.findSynonyms("who", 40)
> for((synonym, cosineSimilarity) <- synonyms) {
>   println(s"$synonym $cosineSimilarity")
> }
> The results are: 
> to Infinity
> and Infinity
> that Infinity
> with Infinity
> said Infinity
> it Infinity
> by Infinity
> be Infinity
> have Infinity
> he Infinity
> has Infinity
> his Infinity
> an Infinity
> ) Infinity
> not Infinity
> who Infinity
> I Infinity
> had Infinity
> their Infinity
> were Infinity
> they Infinity
> but Infinity
> been Infinity
> I tried many different datasets and different words for finding synonyms.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org