airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Carey <sean.ca...@aurishealth.com>
Subject Re: Solved: suppress PendingDeprecationWarning messages in airflow logs
Date Fri, 28 Sep 2018 23:09:30 GMT
Thanks, Ash.  I understand what you're saying.  The warnings are coming from the Qubole operator.
 We get a lot of this:

PendingDeprecationWarning: Invalid arguments were passed to QuboleOperator. Support for passing
such arguments will be dropped in Airflow 2.0. Invalid arguments were:
*args: ()
**kwargs: {...}
  category=PendingDeprecationWarning

We've spoken to Qubole about this and they plan to address it.  In the meantime, it generates
a ton of noise in our logs which we'd like to suppress -- we are aware of the issue, and don't
need to be told about it every 5 seconds.

As for it suddenly breaking, being that this is pending Airflow 2.0 I feel the risk is low
and when we do upgrade it will be thoroughly tested.

Thanks!

Sean
 

On 9/28/18, 5:01 PM, "Ash Berlin-Taylor" <ash@apache.org> wrote:

    What deprecation warnings are you getting? Are they from Airflow itself (i.e. things Airflow
is calling like flask_wtf, etc) or of your use of Airflow?
    
    If it is the form could you check and see if someone has already reported a Jira issue
so we can fix them? https://issues.apache.org/jira/issues/?jql=project%3DAIRFLOW
    
    If it is the latter PLEASE DO NOT IGNORE THESE.
    
    Deprecation warnings are how we, the Airflow community tell users that you need to make
a change to your DAG/code/config to upgrade things. If you silence these warnings you will
have a much harder time upgrading to new versions of Airflow (read: you might suddenly find
that things stop working because you turned of the warnings.)
    
    -ash
    
    > On 28 Sep 2018, at 22:52, Sean Carey <sean.carey@aurishealth.com> wrote:
    > 
    > Hello,
    > 
    > I’ve been looking for a way to suppress the PendingDeprecationWarning messages
cluttering our airflow logs and I have a working solution which I thought I would share.
    > 
    > In order to do this, you first need to configure airflow for custom logging using
steps 1-4 here:
    > 
    > https://airflow.readthedocs.io/en/stable/howto/write-logs.html#writing-logs-to-azure-blob-storage
    > 
    > (note that although the document is for Azure remote logging you don’t actually
need azure for this)
    > 
    > Next, modify the log_config.py script created in the step above as follows:
    > 
    > 
    >  1.  Import logging
    >  2.  Define the filter class:
    > 
    > 
    > 
    > class DeprecationWarningFilter(logging.Filter):
    > 
    >    def filter(self, record):
    > 
    >        allow = 'DeprecationWarning' not in record.msg
    > 
    >        return allow
    > 
    > 
    >  1.  Add a “filters” section to the LOGGING_CONFIG beneath “formatters:
    > 
    > 
    > 
    > 'filters': {
    > 
    >        'noDepWarn': {
    > 
    >            '()': DeprecationWarningFilter,
    > 
    >        }
    > 
    >    },
    > 
    > 
    >  1.  For each of the handlers where you want to suppress the warnings (console, task,
processor, or any of the remote log handlers you may be using) add the following line to its
configuration:
    > 
    > 
    > 
    > 'filters': ['noDepWarn'],
    > 
    > Restart airflow and your logs should be clean.
    > 
    > 
    > Sean Carey
    > 
    
    

Mime
View raw message