spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "" <>
Subject returning type of function that needs to be passed to method 'mapWithState'
Date Mon, 11 Mar 2019 08:42:24 GMT
Hi all:
    In the `mapWithState`    method in spark streaming, you need to pass in an anonymous function.
This function maintains a state and should return a  result. It can be said that the final
stateful result can be obtained from the state object.
        So, what is the significance of returning result? 
        I looked up the official API, and it  did not specifically say that the result is
used for,    just give a simple explanation, as follows:


    // A mapping function that maintains an integer state and return a String
    def mappingFunction(key: String, value: Option[Int], state: State[Int]): Option[String]
= {
      // Use state.exists(), state.get(), state.update() and state.remove()
      // to manage state, and return the necessary string
    val spec = StateSpec.function(mappingFunction).numPartitions(10)
    val mapWithStateDStream = keyValueDStream.mapWithState[StateType, MappedType](spec)
   Can anyone help me with this problem?Thanks!
View raw message