spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Littlestar (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-5795) api.java.JavaPairDStream.saveAsNewAPIHadoopFiles may not friendly to java
Date Fri, 13 Feb 2015 09:07:12 GMT
Littlestar created SPARK-5795:
---------------------------------

             Summary: api.java.JavaPairDStream.saveAsNewAPIHadoopFiles may not friendly to
java
                 Key: SPARK-5795
                 URL: https://issues.apache.org/jira/browse/SPARK-5795
             Project: Spark
          Issue Type: Bug
          Components: Streaming
    Affects Versions: 1.2.1
            Reporter: Littlestar


import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

the following code can't compile on java.
JavaPairDStream<Integer, Integer> rs =....
rs.saveAsNewAPIHadoopFiles("prefix", "txt", Integer.class, Integer.class, TextOutputFormat.class,
jobConf);

but similar code in JavaPairRDD works ok.
JavaPairRDD<String, String> counts =.......
counts.saveAsNewAPIHadoopFile("out", Text.class, Text.class, TextOutputFormat.class, jobConf);
====================
mybe the 
  def saveAsNewAPIHadoopFiles(
      prefix: String,
      suffix: String,
      keyClass: Class[_],
      valueClass: Class[_],
      outputFormatClass: Class[_ <: NewOutputFormat[_, _]],
      conf: Configuration = new Configuration) {
    dstream.saveAsNewAPIHadoopFiles(prefix, suffix, keyClass, valueClass, outputFormatClass,
conf)
  }
=====>
def saveAsNewAPIHadoopFiles[F <: NewOutputFormat[_, _]](
      prefix: String,
      suffix: String,
      keyClass: Class[_],
      valueClass: Class[_],
      outputFormatClass: Class[F],
      conf: Configuration = new Configuration) {
    dstream.saveAsNewAPIHadoopFiles(prefix, suffix, keyClass, valueClass, outputFormatClass,
conf)
  }








--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message