spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wyphao.2007" <wyphao.2...@163.com>
Subject How to get removed RDD from windows?
Date Tue, 31 Mar 2015 02:47:46 GMT
I want to get removed RDD from windows as follow, The old RDDs will removed from current window,

//  _____________________________
// |  previous window   _________|___________________
// |___________________|       current window        |  --------------> Time
//                     |_____________________________|
//
// |________ _________|          |________ _________|
//          |                             |
//          V                             V
//       old RDDs                     new RDDs
//
I find  the slice function in DStream class can return the DStream between fromTime to  toTime.
But when I use the function as follow:


    val now = System.currentTimeMillis()
    result.slice(new Time(now - 30 * 1000), new Time(now - 30 * 1000 + result.slideDuration.milliseconds)).foreach(item
=> println("xxx" + item))
    ssc.start()


30 is the window's duration,Then I got zeroTime has not been initialized exception. 


Is anyone can help me? thx!










Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message