flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Schwarzenberg <schwarzenb...@campus.tu-berlin.de>
Subject hash preservation
Date Thu, 28 Jul 2016 13:21:34 GMT
Dear Flink developers,

I have a question concerning the preservation of hash values.

I have a hashmap keyed by Scala objects that directly inherit the 
hashCode() and equals() methods from Any. (These objects are only used 
to address values in the hashmap mentioned above; they aren't used as 
keys in any Flink operation, such as groupby, for instance.)

I use a collection of such object keys in a MapReduce program. 
Unfortunately, during the Flink map and reduce phases the objects change 
their hash codes and become inconsistent with the keys of the original 
hashmap. However, I need them to conserve their hash values, s.t. they 
can still be used as keys.

The objects/classes discussed above are part of a 3^rd party library. 
Visibility issues prevent me from simply extending them to override the 
hashCode() and equals() methods. Currently, I work with a cloned version 
of that library in which I extend the corresponding class in the 
following manner:

class Key {

     final val hash = System.identityHashCode(this) // cache the hash value

     override def hashCode(): Int =  {



     ...// original code


First experiments suggest that the hash values are indeed preserved 
during MapReduce. However, hacking the library is a very clumsy approach.

My question now is: does Flink provide a more elegant solution?

Thanks for your help!


Robert Schwarzenberg

View raw message