flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pauline Yeung (yeungp)" <yeu...@cisco.com>
Subject extract fields from nested map
Date Sat, 23 Jul 2016 06:14:09 GMT
I have a file, which each line is one json record

I run the following
    val env = ExecutionEnvironment.getExecutionEnvironment
    val data = env.readTextFile("file:///somefile"<file:///somefile%22>)
               .map(line => JSON.parseFull(line))

and get the following for one json record.  For simplicity, the keys and values are modified,
and not all keys and values are shown.

Some(Map(a->1, b->2, c->3, d->Map(e->5, f->6, g->7))

How do I create a new data set from selected keys values from this data set?   E.g. Map(a
-> 1, b -> 2, e -> 5, f -> 6)

To do the equivalent of the following.

scala> val m = Map('a'->1, 'b'->2, 'c'->3, 'd'->Map('e'->5, 'f'->6, 'g'->7))
m: scala.collection.immutable.Map[Char,Any] = Map(a -> 1, b -> 2, c -> 3, d ->
Map(e -> 5, f -> 6, g -> 7))
scala> val s1 = Set('a', 'b')
s1: scala.collection.immutable.Set[Char] = Set(a, b)
scala> val x1 = m.filterKeys { s1.contains(_) }
x1: scala.collection.immutable.Map[Char,Any] = Map(a -> 1, b -> 2)

scala> val n = m('d').asInstanceOf[Map[Char,Any]]
p: Map[Char,Any] = Map(e -> 5, f -> 6, g -> 7)
scala> val s2 = Set('e', 'f')
s2: scala.collection.immutable.Set[Char] = Set(e, f)
scala> val x2 = n.filterKeys { s2.contains(_) }
x2: scala.collection.immutable.Map[Char,Any] = Map(e -> 5, f -> 6)

scala> x1 ++ x2
res28: scala.collection.immutable.Map[Char,Any] = Map(a -> 1, b -> 2, e -> 5, f ->
6)

Thanks
Pauline

Mime
View raw message