beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Work logged] (BEAM-4858) Clean up _BatchSizeEstimator in element-batching transform.
Date Wed, 26 Sep 2018 07:29:00 GMT


ASF GitHub Bot logged work on BEAM-4858:

                Author: ASF GitHub Bot
            Created on: 26/Sep/18 07:28
            Start Date: 26/Sep/18 07:28
    Worklog Time Spent: 10m 
      Work Description: tvalentyn commented on a change in pull request #6375: [BEAM-4858]
Clean up division in batch size estimator.

 File path: sdks/python/apache_beam/transforms/
 @@ -125,6 +126,75 @@ def test_target_overhead(self):
     self.assertEqual(expected_sizes, actual_sizes)
+  def test_variance(self):
+    clock = FakeClock()
 Review comment:
   Actually, it shouldn't matter.

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

Issue Time Tracking

    Worklog Id:     (was: 148010)
    Time Spent: 2h  (was: 1h 50m)

> Clean up _BatchSizeEstimator in element-batching transform.
> -----------------------------------------------------------
>                 Key: BEAM-4858
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py-core
>            Reporter: Valentyn Tymofieiev
>            Assignee: Robert Bradshaw
>            Priority: Minor
>          Time Spent: 2h
>  Remaining Estimate: 0h
> Beam Python 3 conversion [exposed|] non-trivial
performance-sensitive logic in element-batching transform. Let's take a look at [|].

> Due to Python 2 language semantics, the result of {{x2 / x1}} will depend on the type
of the keys - whether they are integers or floats. 
> The keys of key-value pairs contained in {{self._data}} are added as integers [here|],
however, when we 'thin' the collected entries [here|],
the keys will become floats. Surprisingly, using either integer or float division consistently
[in the comparator|]
 negatively affects the performance of a custom pipeline I was using to benchmark these changes.
The performance impact likely comes from changes in the logic that depends on  how division
is evaluated, not from the performance of division operation itself.
> In terms of Python 3 conversion the best course of action that avoids regression seems
to be to preserve the existing Python 2 behavior using {{old_div}} from {{past.utils.division}},
in the medium term we should clean up the logic. We may want to add a targeted microbenchmark
to evaluate performance of this code, and maybe cythonize the code, since it seems to be performance-sensitive.

This message was sent by Atlassian JIRA

View raw message