Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
To: user <user@flink.apache.org>
From: Urs Schoenenberger <urs.schoenenberger@tngtech.com>
Subject: DataSet: CombineHint heuristics
Message-ID: <4607fb76-8825-0be0-45cb-1aba547c9e31@tngtech.com>
Date: Thu, 31 Aug 2017 13:41:52 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.2.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 31 Aug 2017 11:42:13 -0000

Hi all,

I was wondering about the heuristics for CombineHint:

Flink uses SORT by default, but the doc for HASH says that we should
expect it to be faster if the number of keys is less than 1/10th of the
number of records.

HASH should be faster if it is able to combine a lot of records, which
happens if multiple events for the same key are present in a data chunk
*that fits into a combine-hashtable* (cf handling in
ReduceCombineDriver.java).

Now, if I have 10 billion events and 100 million keys, but only about 1
million records fit into a hashtable, the number of matches may be
extremely low, so very few events are getting combined (of course, this
is similar for SORT as the sorter's memory is bounded, too).

Am I correct in assuming that the actual tradeoff is not only based on
the ratio of #total records/#keys, but also on #total records/#records
that fit into a single Sorter/Hashtable?

Thanks,
Urs

--=20
Urs Sch=C3=B6nenberger - urs.schoenenberger@tngtech.com

TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterf=C3=B6hring
Gesch=C3=A4ftsf=C3=BChrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard M=C3=
=BCller
Sitz: Unterf=C3=B6hring * Amtsgericht M=C3=BCnchen * HRB 135082