beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pei He (JIRA)" <>
Subject [jira] [Created] (BEAM-430) Introducing gcpTempLocation that default to tempLocation
Date Thu, 07 Jul 2016 22:19:10 GMT
Pei He created BEAM-430:

             Summary: Introducing gcpTempLocation that default to tempLocation
                 Key: BEAM-430
             Project: Beam
          Issue Type: Improvement
            Reporter: Pei He
            Assignee: Pei He
            Priority: Minor

Currently, DataflowPipelineOptions.stagingLocation default to tempLocation. And, it requires
tempLocation to be a gcs path.
Another case is BigQueryIO uses tempLocation and also requires it to be on gcs.
So, users cannot set tempLocation to a non-gcs path with DataflowRunner or BigQueryIO.

However, tempLocation could be on any file system. For example, WordCount defaults to output
to tempLocation.

The proposal is to add gcpTempLocation. And, it defaults to tempLocation if tempLocation is
a gcs path.
StagingLocation and BigQueryIO will use gcpTempLocation by default.

This message was sent by Atlassian JIRA

View raw message