beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chamikara Jayalath (JIRA)" <>
Subject [jira] [Commented] (BEAM-2265) Python word count with DirectRunner gets stuck during application termination on Windows
Date Fri, 12 May 2017 21:28:04 GMT


Chamikara Jayalath commented on BEAM-2265:

I tried running this with extra logging that prints stack traces of all running threads. I
have not been able to reproduce this with that. Also even without extra logging, only some
of the runs seem to be getting stuck. I suspect some thread scheduling/completion issue when
running on Windows.

For stuck runs, program completes gracefully (producing valid outout) when Ctrl+C is pressed.

> Python word count with DirectRunner gets stuck during application termination on Windows
> ----------------------------------------------------------------------------------------
>                 Key: BEAM-2265
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py
>    Affects Versions: 2.0.0
>            Reporter: Luke Cwik
>            Assignee: Ahmet Altay
>            Priority: Minor
> Using virtualenv 15 + python 2.7.13 + pip 9.0.1 on Windows 2016
> Example logs from DirectRunner:
> {code}
> (beamRC2)PS C:\Users\lcwik\.virtualenvs\beamRC2> python -m apache_beam.examples.wordcount
--input ".\input\*" --output l
> ocal_counts
> No handlers could be found for logger "oauth2client.contrib.multistore_file"
> INFO:root:Missing pipeline option (runner). Executing pipeline using the default runner:
> INFO:root:Running pipeline with DirectRunner.
> {code}
> Application gets stuck here, pressing ctrl-z gets it unstuck and the remainder below
is logged
> {code}
> INFO:root:Starting finalize_write threads with num_shards: 1, batches: 1, num_threads:
> INFO:root:Renamed 1 shards in 0.14 seconds.
> INFO:root:number of empty lines: 47851
> INFO:root:average word length: 4
> {code}
> Output is correct, so it seems as though the bug is somewhere in shutdown.
> Happens when using a local or gs path with the DirectRunner. Enabling DEBUG logging did
not add any additional details.

This message was sent by Atlassian JIRA

View raw message