beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eugene Kirpichov <kirpic...@google.com>
Subject Re: How to create a runtime ValueProvider
Date Fri, 04 May 2018 01:01:38 GMT
There is no way to achieve this using ValueProvider. Its value is either
fixed at construction time (StaticValueProvider), or completely dynamic
(evaluated every time you call .get()).
You'll need to implement this using a side input. E.g. take a look at
implementation of BigQueryIO, how it generates unique job id tokens -
https://github.com/apache/beam/blob/bf94e36f67a8bc5d24c795e40697ad2504c8594c/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L756


On Thu, May 3, 2018 at 5:42 PM Frank Yellin <fy@fyellin.com> wrote:

> [Sorry, I accidentally hit send before I had finished typing . .]
>
> Is there any way to achieve what I'm looking for?  Or is this just beyond
> the scope of ValueProvider and templates?
>
>
>
> On Thu, May 3, 2018 at 5:36 PM, Frank Yellin <fy@fyellin.com> wrote:
>
>> I'm attempting to create a dataflow template, and within the template
>> have a variable
>>     ValueProvider<DateTime> now
>> such that now is the time the dataflow is started, note the time that the
>> template was created.
>>
>> My first attempt was
>>     ValueProvider<DateTime> now =
>> StaticValueProvider.of(DateTime.now(DateTimeZone.UTC));
>>
>> My second attempt was
>>
>>   public interface MyOptions extends PipelineOptions,
>> DataflowPipelineOptions {
>>     @Description("Now")
>>     @Default.InstanceFactory(GetNow.class)
>>     ValueProvider<DateTime> getNow();
>>     void setNow(ValueProvider<DateTime> value);
>>   }
>>
>>   static class GetNow implements DefaultValueFactory<DateTime> {
>>     @Override
>>     public DateTime create(PipelineOptions options) {
>>       return DateTime.now(DateTimeZone.UTC);
>>     }
>>   }
>>
>>   ValueProvider<DateTime> now = options.getNow()
>>
>> My final attempt was:
>>
>>    ValueProvider<SerializableFunction<Void, DateTime>> nowFn =
>>         StaticValueProvider.of(x -> DateTime.now(DateTimeZone.UTC));
>>
>>     ValueProvider<DateTime> now = NestedValueProvider.of(nowFn, x ->
>> x.apply(null));
>>
>>
>>
>> In every case, it was clear that "now" was being set to template-creation
>> time rather than actual runtime.
>>
>> I note that the documentation talks about a RuntimeValueProvider, but
>> there is no user-visible constructor for this.
>>
>>
>>
>>
>

Mime
View raw message