spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Hamstra <m...@clearstorydata.com>
Subject Option folding idiom
Date Thu, 26 Dec 2013 20:33:02 GMT
In code added to Spark over the past several months, I'm glad to see more
use of `foreach`, `for`, `map` and `flatMap` over `Option` instead of
pattern matching boilerplate.  There are opportunities to push `Option`
idioms even further now that we are using Scala 2.10 in master, but I want
to discuss the issue here a little bit before committing code whose form
may be a little unfamiliar to some Spark developers.

In particular, I really like the use of `fold` with `Option` to cleanly an
concisely express the "do something if the Option is None; do something
else with the thing contained in the Option if it is Some" code fragment.

An example:

Instead of...

val driver = drivers.find(_.id == driverId)
driver match {
  case Some(d) =>
    if (waitingDrivers.contains(d)) { waitingDrivers -= d }
    else {
      d.worker.foreach { w =>
        w.actor ! KillDriver(driverId)
      }
    }
    val msg = s"Kill request for $driverId submitted"
    logInfo(msg)
    sender ! KillDriverResponse(true, msg)
  case None =>
    val msg = s"Could not find running driver $driverId"
    logWarning(msg)
    sender ! KillDriverResponse(false, msg)
}

...using fold we end up with...

driver.fold
  {
    val msg = s"Could not find running driver $driverId"
    logWarning(msg)
    sender ! KillDriverResponse(false, msg)
  }
  { d =>
    if (waitingDrivers.contains(d)) { waitingDrivers -= d }
    else {
      d.worker.foreach { w =>
        w.actor ! KillDriver(driverId)
      }
    }
    val msg = s"Kill request for $driverId submitted"
    logInfo(msg)
    sender ! KillDriverResponse(true, msg)
  }


So the basic pattern (and my proposed formatting standard) for folding over
an `Option[A]` from which you need to produce a B (which may be Unit if
you're only interested in side effects) is:

anOption.fold
  {
    // something that evaluates to a B if anOption = None
  }
  { a =>
    // something that transforms `a` into a B if anOption = Some(a)
  }


Any thoughts?  Does anyone really, really hate this style of coding and
oppose its use in Spark?

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message