beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chamikara Jayalath (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-2265) Python word count gets stuck during application termination on Windows
Date Fri, 12 May 2017 00:53:04 GMT

    [ https://issues.apache.org/jira/browse/BEAM-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16007442#comment-16007442
] 

Chamikara Jayalath commented on BEAM-2265:
------------------------------------------

I tried running word count on Windows as well. Word count on Windows for DirectRunner passes
for small inputs but gets stuck for input gs://dataflow-samples/shakespeare/. Jobs did not
get stuck when using DataflowRunner.

> Python word count gets stuck during application termination on Windows
> ----------------------------------------------------------------------
>
>                 Key: BEAM-2265
>                 URL: https://issues.apache.org/jira/browse/BEAM-2265
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py
>            Reporter: Luke Cwik
>            Assignee: Ahmet Altay
>
> Using virtualenv 15 + python 2.7.13 + pip 9.0.1 on Windows 2016
> Example logs from DirectRunner:
> {code}
> (beamRC2)PS C:\Users\lcwik\.virtualenvs\beamRC2> python -m apache_beam.examples.wordcount
--input ".\input\*" --output l
> ocal_counts
> No handlers could be found for logger "oauth2client.contrib.multistore_file"
> INFO:root:Missing pipeline option (runner). Executing pipeline using the default runner:
DirectRunner.
> INFO:root:Running pipeline with DirectRunner.
> {code}
> Application gets stuck here, pressing ctrl-z gets it unstuck and the remainder below
is logged
> {code}
> INFO:root:Starting finalize_write threads with num_shards: 1, batches: 1, num_threads:
1
> INFO:root:Renamed 1 shards in 0.14 seconds.
> INFO:root:number of empty lines: 47851
> INFO:root:average word length: 4
> {code}
> Output is correct, so it seems as though the bug is somewhere in shutdown.
> Happens when using a local or gs path with the DirectRunner or using DataflowRunner.
Enabling DEBUG logging did not add any additional details.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message