mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (MAHOUT-1603) Tweaks for Spark 1.0.x
Date Thu, 07 Aug 2014 18:52:12 GMT


ASF GitHub Bot commented on MAHOUT-1603:

Github user pferrel commented on the pull request:
    made changes to use the test context in the driver and tests seem to complete correctly
up to the point they try to read the output file, which does contain the correct results.
        val indicatorLines = mahoutCtx.textFile(OutPath + "/indicator-matrix/part-00000")
    The part file is created in the driver using ```rdd.saveAsTextFile(dest)```. It seems
like something was getting done before by shutting down the context, maybe I need to close
the output file(s) (not sure how to do that since it's created inside the saveAsTextFile call)?
    	at org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1215)
    	at org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1222)
    	at org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:456)
    	at org.apache.mahout.drivers.ItemSimilarityDriverSuite$$anonfun$4.apply$mcV$sp(ItemSimilarityDriverSuite.scala:303)

> Tweaks for Spark 1.0.x 
> -----------------------
>                 Key: MAHOUT-1603
>                 URL:
>             Project: Mahout
>          Issue Type: Task
>    Affects Versions: 0.9
>            Reporter: Dmitriy Lyubimov
>            Assignee: Dmitriy Lyubimov
>             Fix For: 1.0
> Tweaks necessary current codebase on top of spark 1.0.x

This message was sent by Atlassian JIRA

View raw message