mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAHOUT-1603) Tweaks for Spark 1.0.x
Date Thu, 07 Aug 2014 18:52:12 GMT

    [ https://issues.apache.org/jira/browse/MAHOUT-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14089599#comment-14089599
] 

ASF GitHub Bot commented on MAHOUT-1603:
----------------------------------------

Github user pferrel commented on the pull request:

    https://github.com/apache/mahout/pull/40#issuecomment-51514939
  
    made changes to use the test context in the driver and tests seem to complete correctly
up to the point they try to read the output file, which does contain the correct results.
    
    ```
        val indicatorLines = mahoutCtx.textFile(OutPath + "/indicator-matrix/part-00000")
    ```
    
    The part file is created in the driver using ```rdd.saveAsTextFile(dest)```. It seems
like something was getting done before by shutting down the context, maybe I need to close
the output file(s) (not sure how to do that since it's created inside the saveAsTextFile call)?
    
    ```
    java.lang.NullPointerException
    	at org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1215)
    	at org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1222)
    	at org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:456)
    	at org.apache.mahout.drivers.ItemSimilarityDriverSuite$$anonfun$4.apply$mcV$sp(ItemSimilarityDriverSuite.scala:303)
    ```


> Tweaks for Spark 1.0.x 
> -----------------------
>
>                 Key: MAHOUT-1603
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-1603
>             Project: Mahout
>          Issue Type: Task
>    Affects Versions: 0.9
>            Reporter: Dmitriy Lyubimov
>            Assignee: Dmitriy Lyubimov
>             Fix For: 1.0
>
>
> Tweaks necessary current codebase on top of spark 1.0.x



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message