From commits-return-12786-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Thu Feb 22 18:35:05 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 0029F18064E for ; Thu, 22 Feb 2018 18:35:04 +0100 (CET) Received: (qmail 83038 invoked by uid 500); 22 Feb 2018 17:35:04 -0000 Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list commits@airflow.incubator.apache.org Received: (qmail 83029 invoked by uid 99); 22 Feb 2018 17:35:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Feb 2018 17:35:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 8FD69180414 for ; Thu, 22 Feb 2018 17:35:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.511 X-Spam-Level: X-Spam-Status: No, score=-109.511 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id pvJ_qNZzn-ia for ; Thu, 22 Feb 2018 17:35:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 0E1D75F216 for ; Thu, 22 Feb 2018 17:35:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 5E926E01AD for ; Thu, 22 Feb 2018 17:35:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 1A65427135 for ; Thu, 22 Feb 2018 17:35:00 +0000 (UTC) Date: Thu, 22 Feb 2018 17:35:00 +0000 (UTC) From: "Soundar (JIRA)" To: commits@airflow.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (AIRFLOW-2141) Cannot create airflow variables when there is a list of dictionary as a value MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/AIRFLOW-2141?page=3Dcom.atlass= ian.jira.plugin.system.issuetabpanels:all-tabpanel ] Soundar updated AIRFLOW-2141: ----------------------------- Description:=20 I'm trying to create Airflow variables using a json file. I am trying to im= port airflow variables using UI(webserver) when I upload the json file I ge= t this error "Missing file or syntax error" and when I try to upload using = airflow cli not all the variables gets uploaded properly. The catch is that= I have a list of dictionary in my json file, say ex: { "demo_archivedir": "/home/ubuntu/folders/archive", "demo_filepattern": [ { "id": "reference", "pattern": "Sample Data.xlsx" } , { "id": "sale", "pattern": "Sales.xlsx" } ], "demo_sourcepath": "/home/ubuntu/folders/input", "demo_workdir": "/home/ubuntu/folders/working" } I've attached two images img1. Using airflow variables cli command I was able to create partial vari= ables from my json file(airflow_cli.png)img2. After inserting logs in the "= airflow/bin/cli.py" file, I got this error. (airflow_cli2_crop.png) The thing is I gave this value through the Admin UI one by one and it worke= d. Then I exported those same variable using "airflow variables" cli comman= d and tried importing them, still it failed and the above mentioned error s= till occurs. Note: =C2=A0 I am using Python 3.5 with Airflow version 1.8 The stack trace is as follows .compute-1.amazonaws.com:22] out: 0 of 4 variables successfully updated. .compute-1.amazonaws.com:22] out: Traceback (most recent call last): .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/bin/ai= rflow", line 28, in .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 args.func(args) .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/py= thon3.5/site-packages/airflow/bin/cli.py", line 242, in variables .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 import_helper(imp= ) .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/py= thon3.5/site-packages/airflow/bin/cli.py", line 273, in import_helper .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 Variable.set(k, v= ) .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/py= thon3.5/site-packages/airflow/utils/db.py", line 53, in wrapper .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 result =3D func(*= args, **kwargs) .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/py= thon3.5/site-packages/airflow/models.py", line 3615, in set .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 session.add(Varia= ble(key=3Dkey, val=3Dstored_value)) .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "", line 4, in _= _init__ .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/py= thon3.5/site-packages/sqlalchemy/orm/state.py", line 417, in _initialize_in= stance .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 manager.dispatch.= init_failure(self, args, kwargs) .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/py= thon3.5/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 compat.reraise(ex= c_type, exc_value, exc_tb) .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/py= thon3.5/site-packages/sqlalchemy/util/compat.py", line 187, in reraise .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 raise value .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/py= thon3.5/site-packages/sqlalchemy/orm/state.py", line 414, in _initialize_in= stance .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 return manager.or= iginal_init(*mixed[1:], **kwargs) .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/py= thon3.5/site-packages/sqlalchemy/ext/declarative/base.py", line 700, in _de= clarative_constructor .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 setattr(self, k, = kwargs[k]) compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "", line 1, in __= set__ .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/py= thon3.5/site-packages/airflow/models.py", line 3550, in set_val .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 self._val =3D FER= NET.encrypt(bytes(value, 'utf-8')).decode() .compute-1.amazonaws.com:22] out: TypeError: encoding without a string argu= ment .compute-1.amazonaws.com:22] out: was: I'm trying to create Airflow variables using a json file. I am trying to im= port airflow variables using UI(webserver) when I upload the json file I ge= t this error "Missing file or syntax error" and when I try to upload using = airflow cli not all the variables gets uploaded properly. The catch is that= I have a list of dictionary in my json file, say ex: { "demo_archivedir": "/home/ubuntu/folders/archive", "demo_filepattern": [ { "id": "reference", "pattern": "Sample Data.xlsx" } , { "id": "sale", "pattern": "Sales.xlsx" } ], "demo_sourcepath": "/home/ubuntu/folders/input", "demo_workdir": "/home/ubuntu/folders/working" } I've attached two images=20 img1. Using airflow variables cli command I was able to create partial vari= ables from my json file(airflow_cli.png)img2. After inserting logs in the "= airflow/bin/cli.py" file, I got this error. (airflow_cli2_crop.png) The thing is I gave this value through the Admin UI one by one and it worke= d. Then I exported those same variable using "airflow variables" cli comman= d and tried importing them, still it failed and the above mentioned error s= till occurs. Note: =C2=A0 I am using Python 3.5 with Airflow version 1.8 > Cannot create airflow variables when there is a list of dictionary as a v= alue > -------------------------------------------------------------------------= ---- > > Key: AIRFLOW-2141 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2141 > Project: Apache Airflow > Issue Type: Bug > Components: aws > Affects Versions: Airflow 1.8 > Reporter: Soundar > Priority: Major > Labels: beginner, newbie > Attachments: airflow_cli.png, airflow_cli2_crop.png > > > I'm trying to create Airflow variables using a json file. I am trying to = import airflow variables using UI(webserver) when I upload the json file I = get this error "Missing file or syntax error" and when I try to upload usin= g airflow cli not all the variables gets uploaded properly. The catch is th= at I have a list of dictionary in my json file, say > ex: > { > "demo_archivedir": "/home/ubuntu/folders/archive", > "demo_filepattern": [ > { "id": "reference", "pattern": "Sample Data.xlsx" } > , > { "id": "sale", "pattern": "Sales.xlsx" } > ], > "demo_sourcepath": "/home/ubuntu/folders/input", > "demo_workdir": "/home/ubuntu/folders/working" > } > I've attached two images > img1. Using airflow variables cli command I was able to create partial va= riables from my json file(airflow_cli.png)img2. After inserting logs in the= "airflow/bin/cli.py" file, I got this error. (airflow_cli2_crop.png) > The thing is I gave this value through the Admin UI one by one and it wor= ked. Then I exported those same variable using "airflow variables" cli comm= and and tried importing them, still it failed and the above mentioned error= still occurs. > Note: > =C2=A0 I am using Python 3.5 with Airflow version 1.8 > The stack trace is as follows > .compute-1.amazonaws.com:22] out: 0 of 4 variables successfully updated. > .compute-1.amazonaws.com:22] out: Traceback (most recent call last): > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/bin/= airflow", line 28, in > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 args.func(args) > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/= python3.5/site-packages/airflow/bin/cli.py", line 242, in variables > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 import_helper(i= mp) > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/= python3.5/site-packages/airflow/bin/cli.py", line 273, in import_helper > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 Variable.set(k,= v) > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/= python3.5/site-packages/airflow/utils/db.py", line 53, in wrapper > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 result =3D func= (*args, **kwargs) > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/= python3.5/site-packages/airflow/models.py", line 3615, in set > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 session.add(Var= iable(key=3Dkey, val=3Dstored_value)) > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "", line 4, in= __init__ > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/= python3.5/site-packages/sqlalchemy/orm/state.py", line 417, in _initialize_= instance > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 manager.dispatc= h.init_failure(self, args, kwargs) > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/= python3.5/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit= __ > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 compat.reraise(= exc_type, exc_value, exc_tb) > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/= python3.5/site-packages/sqlalchemy/util/compat.py", line 187, in reraise > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 raise value > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/= python3.5/site-packages/sqlalchemy/orm/state.py", line 414, in _initialize_= instance > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 return manager.= original_init(*mixed[1:], **kwargs) > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/= python3.5/site-packages/sqlalchemy/ext/declarative/base.py", line 700, in _= declarative_constructor > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 setattr(self, k= , kwargs[k]) > compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "", line 1, in = __set__ > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0 File "/home/ubuntu/Env/lib/= python3.5/site-packages/airflow/models.py", line 3550, in set_val > .compute-1.amazonaws.com:22] out:=C2=A0=C2=A0=C2=A0=C2=A0 self._val =3D F= ERNET.encrypt(bytes(value, 'utf-8')).decode() > .compute-1.amazonaws.com:22] out: TypeError: encoding without a string ar= gument > .compute-1.amazonaws.com:22] out: -- This message was sent by Atlassian JIRA (v7.6.3#76005)