flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Norman Spangenberg <wir12...@studserv.uni-leipzig.de>
Subject parse json-file with scala-api and json4s
Date Wed, 06 Aug 2014 15:15:14 GMT
hello,
i hope this is the right place for this question.
i'm currently experimenting and comparing flink/stratosphere and apache 
spark.
my goal is to analyse large json-files of twitter-data and now i'm 
looking for a way to parse the json-tuples in a map-function and put in 
a dataset.
for this i'm using the flink scala api and json4s.
but in flink the problem is to parse the json-file.
val words = cleaned.map { ( line =>  parse(line) }
Error Message is:
     Error analyzing UDT org.json4s.JValue: Subtype 
org.json4s.JsonAST.JInt - Field num: BigInt - Unsupported type BigInt 
Subtype
      org.json4s.JsonAST.JArray - Field arr: 
List[org.json4s.JsonAST.JValue] - Subtype org.json4s.JsonAST.JInt - 
Field num: BigInt - Unsupported type BigInt Subtype
      org.json4s.JsonAST.JArray - Field arr: 
List[org.json4s.JsonAST.JValue] - Subtype org.json4s.JsonAST.JDecimal - 
Field num: BigDecimal - Unsupported type
      BigDecimal Subtype org.json4s.JsonAST.JDecimal - Field num: 
BigDecimal - Unsupported type BigDecimal Subtype 
org.json4s.JsonAST.JObject - Field obj:
      List[(String, org.json4s.JsonAST.JValue)] - Field _2: 
org.json4s.JsonAST.JValue - Subtype org.json4s.JsonAST.JInt - Field num: 
BigInt - Unsupported type BigInt
      Subtype org.json4s.JsonAST.JObject - Field obj: List[(String, 
org.json4s.JsonAST.JValue)] - Field _2: org.json4s.JsonAST.JValue - Subtype
      org.json4s.JsonAST.JDecimal - Field num: BigDecimal - Unsupported 
type BigDecimal


in spark i found a way based on 
https://gist.github.com/cotdp/b471cfff183b59d65ae1

val user_interest = lines.map(line => {parse(line)})
                          .map(json => {implicit lazy val formats = 
org.json4s.DefaultFormats
                                     val name = (json \ 
"name").extract[String]
                                     val location_x = (json \ "location" 
\ "x").extract[Double]
                                     val location_y = (json \ "location" 
\ "y").extract[Double]
                                     val likes = (json \ 
"likes").extract[Seq[String]].map(_.toLowerCase()).mkString(";")
                                     ( UserInterest(name, location_x, 
location_y, likes) )
                                    })

this works fine in spark, but is it possible to do the same with flink?

kind regards,
norman

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message