flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "pat.chormai" <pat.chor...@gmail.com>
Subject Re: FLINK-5734 : Code Generation for NormalizedKeySorter
Date Thu, 09 Feb 2017 20:50:50 GMT
Hi Greg,

Thanks for your feedback. I would like to answer your questions here.

Q: Do I understand correctly that the generated code is only dependent on
the 
length of the sort key?

A: Yes, you're right. The generated code is mainly dependent on the length
of the sort key. However, I'm not sure whether we can reuse the generated
code for 2 different types of key even they have the same length. 

Q: How do you propose computing indexEntriesPerSegment? Is the improved
performance not dependent on this value being a power-of-2? 
 
A: This improvement doesn't change the way indexEntriesPerSegment is
computed. What we did here is replacing indexEntriesPerSegment at
expressions that indexEntriesPerSegment is a divisor with its value [1]. As
a result, it does not require value of indexEntriesPerSegment to be an
exponent of 2. More information related to this technique can be found at
[2].

Q: Are you able to compare with FLINK-3722?

A: We're planning to do that as well, however, it might take some time due
to exams.

Also, could you please elaborate more why DividedByConstant and
UsingBitwiseOperators use only half of sort memory when
indexEntriesPerSegment is power of 2?


[1]
https://github.com/heytitle/flink-sorter-performance-evaluation/blob/master/src/main/java/org/evaluation/sorter/individual/optimization/DividedByConstant.java#L353
[2]
https://blogs.msdn.microsoft.com/devdev/2005/12/12/integer-division-by-constants/



--
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/FLINK-5734-Code-Generation-for-NormalizedKeySorter-tp15804p15860.html
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.

Mime
View raw message