mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lastarsenal <lastarse...@163.com>
Subject Re:Re: mahout spark FilteredRDD problem
Date Fri, 05 Jun 2015 03:44:30 GMT
Hi, pat,
    Thanks, I updated mahout master and ran: /bin/mahout spark-itemsimilarity -i ${input}
-o ${output} --master $MyMaster --sparkExecutorMem 2g
    The problem dismissed.


    Howevever, another new problem comes:


15/06/05 11:37:46 WARN AkkaUtils: Error sending message in 1 attempts
java.util.concurrent.TimeoutException: Futures timed out after [30 seconds]
	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
	at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
	at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
	at scala.concurrent.Await$.result(package.scala:107)
	at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
	at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:399)

    Could somebody give some advice? Very appreciated!






在 2015-05-26 02:00:08,"Pat Ferrel" <pat@occamsmachete.com> 写道:
>Yes, the master was updated last week. You can check the “commits” tab on github to
see the change history. This puts the master a couple of commits ahead of the release. The
master branch reflects the next release’s work in progress snapshot and currently produces
jars as …0.10.1-SNAPSHOT. Remember not to build for Spark 1.3, leave the POMs as-is. This
will build for 1.2.2 but should run, as far as my testing has shown, on 1.3
>
>In the next week or so there will be a branch that will build and run on Spark 1.3 but
will lack the mahout Scala shell for awhile.
>
>
>On May 22, 2015, at 8:37 PM, lastarsenal <lastarsenal@163.com> wrote:
>
>this problem is from git clone mahout master, not from mahout 0.10. maybe mahout master
is updated recently? i will update and try again, thank you!
>
>发自我的 iPhone
>
>> 在 2015年5月22日,下午9:20,Pat Ferrel <pat@occamsmachete.com> 写道:
>> 
>> Mahout 0.10.0 runs on Spark 1.1.1 or below _only_
>> 
>> If you are only using spark-itemsimilarity you can try the unreleased master, which
is being moved to Spark 1.2.2, and is binary compatible with Spark 1.3. Get the latest master
branch from https://github.com/apache/mahout and build from source. Leave the version at Spark
1.2.2. Changing to Spark 1.3 in the pom will cause compile errors like the ones below.
>> 
>> I’ve run a few tests on Spark 1.3 but if you’d like to try also please report
back what you find.
>> 
>> BTW the use of guava has been removed so any source examples are still being updated.
If you are using the command line you should be fine,
>> 
>> 
>> On May 21, 2015, at 11:50 PM, lastarsenal <lastarsenal@163.com> wrote:
>> 
>> Hi, 
>> 
>> 
>> Recently I tried mahout spark, for example: 
>> 
>> 
>> ./bin/mahout spark-itemsimilarity -i ${input} -o ${output} --master $MyMaster --sparkExecutorMem
2g
>> 
>> 
>> then I met a error like "Caused by: java.lang.ClassCastException: org.apache.mahout.sparkbindings.io.MahoutKryoRegistrator
cannot be cast to org.apache.spark.serializer.KryoRegistrator", It seems that our spark version
is NOT compatible with mahout. 
>> 
>> 
>> For the spark system is deployed by ops, so what I can do just follow them. Then
I do something as below:
>> 1.  Modify  <spark.version>1.1.1</spark.version>  to  <spark.version>1.3.0</spark.version>
, which is our spark version,   in pom.xml in mahout.
>> 2. Run mvn -DskipTests clean install
>> 3. Get a error when build: 
>> 
>> 
>> [ERROR] spark/src/main/scala/org/apache/mahout/sparkbindings/drm/CheckpointedDrmSpark.scala:168:
error: value saveAsSequenceFile is not a member of org.apache.mahout.sparkbindings.DrmRdd[K]
>> [ERROR]     rdd.saveAsSequenceFile(path)
>> [ERROR]         ^
>> [ERROR]spark/src/main/scala/org/apache/mahout/sparkbindings/drm/package.scala:26:
error: object FilteredRDD is not a member of package org.apache.spark.rdd
>> [ERROR] import org.apache.spark.rdd.{FilteredRDD, RDD}
>> 
>> 
>> 4. Check 1.3.0 spark, FilteredRDD is dismissed
>> 5. Check 1.1.1 spark, FilteredRDD is available.
>> 
>> 
>> So, my question is how can I solve it?
>> 
>> 
>> The error details is as below when I run ./bin/mahout spark-itemsimilarity -i ${input}
-o ${output} --master $MyMaster --sparkExecutorMem 2g
>> 15/05/22 12:22:27 WARN TaskSetManager: Lost task 8.0 in stage 0.0 (TID 8, 182.118.21.30):
java.io.IOException: org.apache.spark.SparkException: Failed to register classes with Kryo
>>      at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1008)
>>      at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:164)
>>      at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
>>      at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
>>      at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:87)
>>      at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
>>      at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:61)
>>      at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
>>      at org.apache.spark.scheduler.Task.run(Task.scala:56)
>>      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:195)
>>      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>      at java.lang.Thread.run(Thread.java:722)
>> Caused by: org.apache.spark.SparkException: Failed to register classes with Kryo
>>      at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:105)
>>      at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:157)
>>      at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:119)
>>      at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:214)
>>      at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:177)
>>      at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1005)
>>      ... 12 more
>> Caused by: java.lang.ClassCastException: org.apache.mahout.sparkbindings.io.MahoutKryoRegistrator
cannot be cast to org.apache.spark.serializer.KryoRegistrator
>>      at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$3.apply(KryoSerializer.scala:101)
>>      at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$3.apply(KryoSerializer.scala:101)
>>      at scala.Option.map(Option.scala:145)
>>      at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:101)
>>      ... 17 more
>> 
>> 
>
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message