flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Balaji Rajagopalan <balaji.rajagopa...@olacabs.com>
Subject Re: Confusion about multiple use of one ValueState
Date Fri, 13 May 2016 09:31:32 GMT
I wrote a simple helper class, the redis connection are initialized in
the constructor and there are

set and get methods to store and retreive values from your map
functions. If you find any better way

to do this please share :). I am using redis scala client.

object class RedisHelper

{

val redisHost = GlobalConfiguration.getString("redis.host", "localhost")
val redisPort = GlobalConfiguration.getInteger("redis.port", 6379)
LOG.info(s"Using host: [$redisHost] and port : [$redisPort] to connect
to redis")

implicit val executionContext = ExecutionContext.global
implicit val akkaSystem = akka.actor.ActorSystem("redis-flink-actorsystem")
val redisClient = RedisClient(host = redisHost, port = redisPort)

def set(k: String, v: String, exTime: Option[Long]): Unit = {
  redisClient.set(k, v, exTime)
}

def set(k: String, v: String): Unit = {
  redisClient.set(k, v)
}

def get(k: String): Option[String] = {
  try
    {
      type K = Option[String]
      val f = redisClient.get[String](k)
      Await.result(f, 2.seconds)
    }
  catch {
    case e: Exception => {
      LOG.error("Exception while getting data from redis " +
e.fillInStackTrace())
      None
    }
  }

}


On Fri, May 13, 2016 at 2:09 PM, Nirmalya Sengupta <
sengupta.nirmalya@gmail.com> wrote:

>
> Hello Balaji <balaji.rajagopalan@olacabs.com>,
>
> Thanks for your reply. This confirms my earlier assumption that one of
> usual ways to do it is to hold and nurture the application-state in an
> external body; in your case: Redis.
>
> So, I am trying to understand how does one share the handle to this
> external body amongst partitions: do I create a Connector to a Redis
> instance (referring to your case as an example) at the beginning of Flink
> application and share that amongst partitions using the _Broadcast_
> mechanism? Obviously, the assumption is that the external body (Redis, in
> this case) will have to deal with concurrent access of elements of State,
> and updation etc. Operators simply call an API on Redis to store and
> retrieve elements from the application-state.
>
> Is my understanding correct?
>
> Yes, I am aware of the fresh creation of a Mapper for every tuple that
> comes in. In fact, this was the source of my original doubt before I posted
> the question. Thanks again for underscoring that.
>
> -- Nirmalya
>
>
> --
> Software Technologist
> http://www.linkedin.com/in/nirmalyasengupta
> "If you have built castles in the air, your work need not be lost. That is
> where they should be.
> Now put the foundation under them."
>

Mime
View raw message