beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Morand, Sebastien" <sebastien.mor...@veolia.com>
Subject Re: Action in the pipeline after Write
Date Sun, 11 Jun 2017 08:50:01 GMT
Yes this use case can be treated by using parallel operation.

I have a 2nd one, I would like to send a report at the end of the pipeline
when the last line has been written in bigquery: number of lines treated,
number of lines ignored (from another part of the pipeline using graph as
you described), number of files at the begining, and so on.

This report could be:

   1. Write a pub/sub
   2. Send an email
   3. Call an url with parameters

Is this possible?

Regards,



*S├ębastien MORAND*
Team Lead Solution Architect
Technology & Operations / Digital Factory
Veolia - Group Information Systems & Technology (IS&T)
Cell.: +33 7 52 66 20 81 / Direct: +33 1 85 57 71 08
Bureau 0144C (Ouest)
30, rue Madeleine-Vionnet - 93300 Aubervilliers, France
*www.veolia.com <http://www.veolia.com>*
<http://www.veolia.com>
<https://www.facebook.com/veoliaenvironment/>
<https://www.youtube.com/user/veoliaenvironnement>
<https://www.linkedin.com/company/veolia-environnement>
<https://twitter.com/veolia>

On 11 June 2017 at 04:14, Eugene Kirpichov <kirpichov@google.com> wrote:

> Hi!
> It sounds like you want to write data to BigQuery and then load the same
> data back from BigQuery? Why? I'm particularly confused by your comment
> "nothing left in the PCollection" - writing a collection to BigQuery
> doesn't remove data from the collection, a PCollection is just a logical
> description of a dataset, not a mutable container. Transforms are like
> mathematical functions - they don't change their inputs, they only compute
> their outputs.
>
> Perhaps that you're assuming that Beam pipelines can only be a strict
> linear sequence of transforms? That is not the case - pipelines are an
> arbitrary graph, you can use a collection multiple times, i.e. apply
> multiple transforms to it. E.g. you can both write the collection to
> bigquery (step 3) and apply some other transform to the same collection
> (step 5).
>
> Assuming you use Java:
> PCollection<Foo> foos = p.apply(TextIO.read().from(...)).apply(...some
> transform...);
> foos.apply(BigQueryIO.write().to(...));
> PCollection<Bar> bars = foos.apply(...some other transform...);
> bars.apply(BigQueryIO.write().to(...));
>
> Let me know if this helps.
>
> On Sat, Jun 10, 2017 at 3:42 PM Morand, Sebastien <
> sebastien.morand@veolia.com> wrote:
>
>> Hi,
>>
>> Is there any way to add some step after a Write, because Write return un
>> PDone, so I can't do anything, but I would like actually do something.
>>
>> Example :
>>
>>    1. Load data from gcs
>>    2. Some transform
>>    3. Write data into bigquery
>>    => Nothing left in the pcollection, but when 3 is over =>
>>    4. Load data from bigquery
>>    5. Some other transform
>>    6. Write data into bigquery
>>
>> Any way to do that?
>>
>> Thanks,
>>
>> *S├ębastien MORAND*
>> Team Lead Solution Architect
>> Technology & Operations / Digital Factory
>> Veolia - Group Information Systems & Technology (IS&T)
>> Cell.: +33 7 52 66 20 81 / Direct: +33 1 85 57 71 08
>> <+33%201%2085%2057%2071%2008>
>> Bureau 0144C (Ouest)
>> 30, rue Madeleine-Vionnet - 93300 Aubervilliers, France
>> *www.veolia.com <http://www.veolia.com>*
>> <http://www.veolia.com>
>> <https://www.facebook.com/veoliaenvironment/>
>> <https://www.youtube.com/user/veoliaenvironnement>
>> <https://www.linkedin.com/company/veolia-environnement>
>> <https://twitter.com/veolia>
>>
>>
>> ------------------------------------------------------------
>> --------------------------------
>> This e-mail transmission (message and any attached files) may contain
>> information that is proprietary, privileged and/or confidential to Veolia
>> Environnement and/or its affiliates and is intended exclusively for the
>> person(s) to whom it is addressed. If you are not the intended recipient,
>> please notify the sender by return e-mail and delete all copies of this
>> e-mail, including all attachments. Unless expressly authorized, any use,
>> disclosure, publication, retransmission or dissemination of this e-mail
>> and/or of its attachments is strictly prohibited.
>>
>> Ce message electronique et ses fichiers attaches sont strictement
>> confidentiels et peuvent contenir des elements dont Veolia Environnement
>> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc
>> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
>> message par erreur, merci de le retourner a son emetteur et de le detruire
>> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la
>> publication, la distribution, ou la reproduction non expressement
>> autorisees de ce message et de ses pieces attachees sont interdites.
>> ------------------------------------------------------------
>> --------------------------------
>>
>

-- 

--------------------------------------------------------------------------------------------
This e-mail transmission (message and any attached files) may contain 
information that is proprietary, privileged and/or confidential to Veolia 
Environnement and/or its affiliates and is intended exclusively for the 
person(s) to whom it is addressed. If you are not the intended recipient, 
please notify the sender by return e-mail and delete all copies of this 
e-mail, including all attachments. Unless expressly authorized, any use, 
disclosure, publication, retransmission or dissemination of this e-mail 
and/or of its attachments is strictly prohibited. 

Ce message electronique et ses fichiers attaches sont strictement 
confidentiels et peuvent contenir des elements dont Veolia Environnement 
et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc 
destines a l'usage de leurs seuls destinataires. Si vous avez recu ce 
message par erreur, merci de le retourner a son emetteur et de le detruire 
ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la 
publication, la distribution, ou la reproduction non expressement 
autorisees de ce message et de ses pieces attachees sont interdites.
--------------------------------------------------------------------------------------------

Mime
View raw message