beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Liu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (BEAM-1986) Job ALREADY_EXISTS in post commit
Date Tue, 02 May 2017 21:53:04 GMT

    [ https://issues.apache.org/jira/browse/BEAM-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15993850#comment-15993850
] 

Mark Liu edited comment on BEAM-1986 at 5/2/17 9:52 PM:
--------------------------------------------------------

After investigation, I have some ideas to solve it and want to discuss it here:

1. Append some random number in default job_name which is pretty straight forward. 

2. I want to improve the descriptive of job_name, which can solve this problem and potentially
benefit for other test runner in the further. Currently, the default pattern is {code}"beamapp-${USER_NAME}-${DATETIME}"{code}
which is hard to tell which test is running. In Java SDK, default job_name format is more
helpful: {code}"ApplicationName-UserName-Date-RandomInteger"{code} (https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java#L261).
However, Java provides ApplicationNameOptions.class to set this value and is used by constructing
default job_name, but Python doesn't provide the similar options. 

I prefer the second solution but it involves with pipeline options design. Just want to rise
ideas here and hear more thoughts. [~altay] [~pabloem]


was (Author: markflyhigh):
After investigation, I have some ideas to solve it and want to discuss it here:

1. Append some random number in default job_name which is pretty straight forward. 

2. I want to improve the descriptive of job_name, which can solve this problem and potentially
benefit for other test runner in the further. Currently, the default pattern is {code}"beamapp-${USER_NAME}-${DATETIME}"{code}
which is hard to tell which test is running. In Java SDK, default job_name format is more
helpful: {code}"ApplicationName-UserName-Date-RandomInteger"{code} (https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java#L261).
However, Java provides ApplicationNameOptions.class to set this value and is used by constructing
default job_name, but Python doesn't provide the similar options. 

I prefer the second solution but it involves with pipeline options design, so I want to rise
ideas here and here more thoughts. [~altay] [~pabloem]

> Job ALREADY_EXISTS in post commit
> ---------------------------------
>
>                 Key: BEAM-1986
>                 URL: https://issues.apache.org/jira/browse/BEAM-1986
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py
>            Reporter: Ahmet Altay
>            Assignee: Mark Liu
>            Priority: Minor
>
> I noticed a job failed with ALREDY_EXISTS error, a sign of same {{job_name}} auto generated
twice. Could we add a 1 second delay to prevent things like this?
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/1877/consoleFull
> cc: [~pabloem] Another perspective, would it make sense to add a small random component
(e.g. 1-2 digits) to job name to reduce this issue? Or perhaps include ms resolution. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message