airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [airflow] eladkal commented on pull request #13152: Convert columns in get_pandas_df to lowercase SnowflakeHook
Date Wed, 03 Feb 2021 10:17:39 GMT

eladkal commented on pull request #13152:
URL: https://github.com/apache/airflow/pull/13152#issuecomment-772396621


   I tend to agree with @dstandish 
   I also don't think this issue is localized only to Snowflake.
   I have ETLs where i do something similar with other databases. Nothing prevents you from
doing:
   
   ```
   def snflk_func(**kwargs):
       snflk = SnowflakeHook('snowflake_default')
       df = snflk.get_pandas_df(sql=sql)
       if kwargs['lowercase_columns']:
           df.columns = df.columns.str.lower()
       # load your df to anywhere you need.
   
   op = PyhtonOperator(task_id='snflk_task',
                       python_callable=snflk_func,
                       op_kwargs={'lowercase_columns': True},
                       dag=dag
                       )
   ```
   
   When you use hooks either you wrap it with python function or you inherit from and can
overwrite the `get_pandas_df`. In both cases it's very easy to handle the problem you presented.
   
   If this PR proceed forward my only comment is that at least don't call it `lowercase_columns`.
It's preferred to call it something like `column_case` that accept lowercase as option. I
for example in my code convert to [snakecase](https://en.wikipedia.org/wiki/Snake_case) so
if it's accepted better to keep the option open for extentions.
   
   Like @dstandish I'm - 0 for this PR but that is just me.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



Mime
View raw message