airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Palmer <ch...@crpalmer.com>
Subject Re: Solved: suppress PendingDeprecationWarning messages in airflow logs
Date Sat, 29 Sep 2018 00:56:32 GMT
Doesn't this warning come from the BaseOperator class -
https://github.com/apache/incubator-airflow/blob/master/airflow/models.py#L2511
?

Are you passing unused arguments to the QuboleOperator, or do you not
control the instantiation of those operators?

Chris

On Fri, Sep 28, 2018 at 7:39 PM Sean Carey <sean.carey@aurishealth.com>
wrote:

> Haha yes, good point.  Thanks!
>
> Sean
>
>
> On 9/28/18, 6:13 PM, "Ash Berlin-Taylor" <ash@apache.org> wrote:
>
>     Sounds good for your use, certainly.
>
>     I mainly wanted to make sure other people knew before blindly
> equipping a foot-canon :)
>
>     -ash
>
>     > On 29 Sep 2018, at 00:09, Sean Carey <sean.carey@aurishealth.com>
> wrote:
>     >
>     > Thanks, Ash.  I understand what you're saying.  The warnings are
> coming from the Qubole operator.  We get a lot of this:
>     >
>     > PendingDeprecationWarning: Invalid arguments were passed to
> QuboleOperator. Support for passing such arguments will be dropped in
> Airflow 2.0. Invalid arguments were:
>     > *args: ()
>     > **kwargs: {...}
>     >  category=PendingDeprecationWarning
>     >
>     > We've spoken to Qubole about this and they plan to address it.  In
> the meantime, it generates a ton of noise in our logs which we'd like to
> suppress -- we are aware of the issue, and don't need to be told about it
> every 5 seconds.
>     >
>     > As for it suddenly breaking, being that this is pending Airflow 2.0
> I feel the risk is low and when we do upgrade it will be thoroughly tested.
>     >
>     > Thanks!
>     >
>     > Sean
>     >
>     >
>     > On 9/28/18, 5:01 PM, "Ash Berlin-Taylor" <ash@apache.org> wrote:
>     >
>     >    What deprecation warnings are you getting? Are they from Airflow
> itself (i.e. things Airflow is calling like flask_wtf, etc) or of your use
> of Airflow?
>     >
>     >    If it is the form could you check and see if someone has already
> reported a Jira issue so we can fix them?
> https://issues.apache.org/jira/issues/?jql=project%3DAIRFLOW
>     >
>     >    If it is the latter PLEASE DO NOT IGNORE THESE.
>     >
>     >    Deprecation warnings are how we, the Airflow community tell users
> that you need to make a change to your DAG/code/config to upgrade things.
> If you silence these warnings you will have a much harder time upgrading to
> new versions of Airflow (read: you might suddenly find that things stop
> working because you turned of the warnings.)
>     >
>     >    -ash
>     >
>     >> On 28 Sep 2018, at 22:52, Sean Carey <sean.carey@aurishealth.com>
> wrote:
>     >>
>     >> Hello,
>     >>
>     >> I’ve been looking for a way to suppress the
> PendingDeprecationWarning messages cluttering our airflow logs and I have a
> working solution which I thought I would share.
>     >>
>     >> In order to do this, you first need to configure airflow for custom
> logging using steps 1-4 here:
>     >>
>     >>
> https://airflow.readthedocs.io/en/stable/howto/write-logs.html#writing-logs-to-azure-blob-storage
>     >>
>     >> (note that although the document is for Azure remote logging you
> don’t actually need azure for this)
>     >>
>     >> Next, modify the log_config.py script created in the step above as
> follows:
>     >>
>     >>
>     >> 1.  Import logging
>     >> 2.  Define the filter class:
>     >>
>     >>
>     >>
>     >> class DeprecationWarningFilter(logging.Filter):
>     >>
>     >>   def filter(self, record):
>     >>
>     >>       allow = 'DeprecationWarning' not in record.msg
>     >>
>     >>       return allow
>     >>
>     >>
>     >> 1.  Add a “filters” section to the LOGGING_CONFIG beneath
> “formatters:
>     >>
>     >>
>     >>
>     >> 'filters': {
>     >>
>     >>       'noDepWarn': {
>     >>
>     >>           '()': DeprecationWarningFilter,
>     >>
>     >>       }
>     >>
>     >>   },
>     >>
>     >>
>     >> 1.  For each of the handlers where you want to suppress the
> warnings (console, task, processor, or any of the remote log handlers you
> may be using) add the following line to its configuration:
>     >>
>     >>
>     >>
>     >> 'filters': ['noDepWarn'],
>     >>
>     >> Restart airflow and your logs should be clean.
>     >>
>     >>
>     >> Sean Carey
>     >>
>     >
>     >
>     >
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message