spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Halit Olali (JIRA)" <j...@apache.org>
Subject [jira] [Closed] (SPARK-5146) saveAsHadoopFiles does not work with checkpointing
Date Thu, 08 Jan 2015 16:11:34 GMT

     [ https://issues.apache.org/jira/browse/SPARK-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Halit Olali closed SPARK-5146.
------------------------------
       Resolution: Not a Problem
    Fix Version/s: 1.2.0

sorry about the trouble,  this has already been fixed in the last release

> saveAsHadoopFiles does not work with checkpointing
> --------------------------------------------------
>
>                 Key: SPARK-5146
>                 URL: https://issues.apache.org/jira/browse/SPARK-5146
>             Project: Spark
>          Issue Type: Bug
>    Affects Versions: 1.2.0
>            Reporter: Halit Olali
>            Priority: Minor
>             Fix For: 1.2.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Hello folks,
> Consider the following simple app for word counting via network socket:
> {code:title=WordCount.scala|borderStyle=solid}
>     val conf = new SparkConf().setAppName("Sample Application")
>     val sc = new SparkContext(conf)
>     val ssc = new StreamingContext(sc, Seconds(5))
>     ssc.checkpoint("target/checkpointDir")
>     val lines = ssc.socketTextStream("localhost", 9999)
>     val words = lines.flatMap(_.split(" "))
>     val pairs = words.map(word => (word, 1))
>     val wordCounts = pairs.reduceByKey(_ + _)
>     wordCounts.saveAsHadoopFiles("target/prefix","suffix")
>     //nc -lk 9999
>     ssc.start()
>     ssc.awaitTermination(60)
> {code}
> When this is packaged and executed on spark, following exception is thrown:
> java.io.NotSerializableException: org.apache.hadoop.mapred.JobConf 
> JobConf usage inside saveAsHadoopFiles methods seems to be the cause.
> Thanks



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message