airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rui Wang <rui.w...@airbnb.com.INVALID>
Subject Re: Xcom related security issue
Date Wed, 01 Mar 2017 22:34:55 GMT
Thanks Jeremiah for your response!

After discussion with experienced airflow contributors. One gently way is, not deprecate it,
but has a conf setting to show if use Pickle Type.
Let’s say if the conf says does not use Pickle Type, before the object to be pickled, it
can first be parsed as JSON. And if retrieve  from Pickle Type column, it could recover from
JSON as well.

Thanks,
Rui Wang


On 2017-02-19 09:37 (-0800), Jeremiah Lowin <j...@apache.org> wrote: 
> Rui,> 
> 
> Thanks for pointing this out, it's a valid concern.> 
> 
> I personally have no issue with swapping Pickle -> JSON, but there may be> 
> many Airflow users relying on the current behavior and I don't want to> 
> invalidate their DAGs with a PR.> 
> 
> On the other hand, I'm not sure of a way to "gently" deprecate the> 
> PickleType. Perhaps step 1 is to check if an XCom can be JSON serialized> 
> and if it can't, print a warning? Then step 2 is to enforce JSON> 
> serialization at a future date.> 
> 
> Any suggestions of how to implement this?> 
> 
> J> 
> 
> On Sat, Feb 18, 2017 at 10:16 AM Rui Wang <rui.wang@airbnb.com.invalid>> 
> wrote:> 
> 
> > Hi,> 
> >> 
> > I created an JIRA issue: https://issues.apache.org/jira/browse/AIRFLOW-855> 
> > .> 
> >> 
> >> 
> > The JIRA task above gives pretty rich context. Briefly speaking, PickleType>

> > gives the possible that run code/command on remote machines. This type can> 
> > serialize objects, which is a wide scope. I am wondering what kind of use> 
> > cases you have for using Xcom and its PickleType. If the use cases show the>

> > possibility that replacing PickleType with JSON type, the probably this> 
> > security issue can be solved by using JSON type instead,> 
> >> 
> >> 
> > Thanks,> 
> > Rui Wang> 
> >> 
> 
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message