airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerard Toonstra <gtoons...@gmail.com>
Subject Re: BigQuery Sensor Operator
Date Wed, 05 Jul 2017 11:01:07 GMT
There is an api where you can get table details in python. There are multiple apis all using
the underlying rest one. The one i talk about is where you can call exists and get rownum
and create and modified details. Saves some money and time perhaps.

Sent from my iPhone

> On 5 Jul 2017, at 12:06, Andrew Maguire <andrewm4894@gmail.com> wrote:
> 
> Hi,
> 
> I'm just wondering if anyone knows if there is a such thing as a sensor
> operator for bigquery?
> 
> I'm trying to just trigger the dag when our daily [ga_sessions_yyyymmdd]
> table we get from google analytics grows.
> 
> Not sure what the best way to do this would be.
> 
> Other options i think might be to just use the bq command line tool and a
> bash based approach. Or i could have a little python script that does the
> sensing or checking if table size has changed.
> 
> Any suggestions as to what a good approach might be would be greatly
> appreciated.
> 
> At the moment i just run the job every few hours regardless which obviously
> is not very efficient.
> 
> Cheers,
> Andy

Mime
View raw message