Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 22418200BC2 for ; Thu, 17 Nov 2016 09:15:47 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 20ACA160B0B; Thu, 17 Nov 2016 08:15:47 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 677EC160AFF for ; Thu, 17 Nov 2016 09:15:46 +0100 (CET) Received: (qmail 54995 invoked by uid 500); 17 Nov 2016 08:15:45 -0000 Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list commits@airflow.incubator.apache.org Received: (qmail 54986 invoked by uid 99); 17 Nov 2016 08:15:45 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Nov 2016 08:15:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 09CF2180538 for ; Thu, 17 Nov 2016 08:15:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -6.218 X-Spam-Level: X-Spam-Status: No, score=-6.218 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-2.999, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id EGe46tMbXUDW for ; Thu, 17 Nov 2016 08:15:42 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id F061F5F47F for ; Thu, 17 Nov 2016 08:15:41 +0000 (UTC) Received: (qmail 54960 invoked by uid 99); 17 Nov 2016 08:15:41 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Nov 2016 08:15:41 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id EF20BE0C0A; Thu, 17 Nov 2016 08:15:40 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: sanand@apache.org To: commits@airflow.incubator.apache.org Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: incubator-airflow git commit: [AIRFLOW-565] Fixes DockerOperator on Python3.x Date: Thu, 17 Nov 2016 08:15:40 +0000 (UTC) archived-at: Thu, 17 Nov 2016 08:15:47 -0000 Repository: incubator-airflow Updated Branches: refs/heads/master d91082124 -> e1bc51691 [AIRFLOW-565] Fixes DockerOperator on Python3.x The issue is that `self.cli.pull()` returns `bytes()`, and not a string. Then, when we try to pass that to `json.loads()`, it raises an exception. The fix is to convert the bytes to a string by decoding it as "utf-8". We're hardcoding the encoding because, by the JSON schema, a JSON should encoded in UTF-8, UTF-16 or UTF-32. Considering we're only pulling images from Docker servers, we can be relatively safe that they'll behave correctly. Closes #1832 from vitorbaptista/bug/fixes- AIRFLOW-565 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/e1bc5169 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/e1bc5169 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/e1bc5169 Branch: refs/heads/master Commit: e1bc5169181e7fcf6b8068f7da309c35b87a05b7 Parents: d910821 Author: Vitor Baptista Authored: Thu Nov 17 00:15:30 2016 -0800 Committer: Siddharth Anand Committed: Thu Nov 17 00:15:30 2016 -0800 ---------------------------------------------------------------------- airflow/operators/docker_operator.py | 2 +- tests/operators/docker_operator.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/e1bc5169/airflow/operators/docker_operator.py ---------------------------------------------------------------------- diff --git a/airflow/operators/docker_operator.py b/airflow/operators/docker_operator.py index d17d154..c75e776 100644 --- a/airflow/operators/docker_operator.py +++ b/airflow/operators/docker_operator.py @@ -152,7 +152,7 @@ class DockerOperator(BaseOperator): if self.force_pull or len(self.cli.images(name=image)) == 0: logging.info('Pulling docker image ' + image) for l in self.cli.pull(image, stream=True): - output = json.loads(l) + output = json.loads(l.decode('utf-8')) logging.info("{}".format(output['status'])) cpu_shares = int(round(self.cpus * 1024)) http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/e1bc5169/tests/operators/docker_operator.py ---------------------------------------------------------------------- diff --git a/tests/operators/docker_operator.py b/tests/operators/docker_operator.py index 5a405ad..cdfae94 100644 --- a/tests/operators/docker_operator.py +++ b/tests/operators/docker_operator.py @@ -44,7 +44,7 @@ class DockerOperatorTestCase(unittest.TestCase): client_mock.create_host_config.return_value = host_config client_mock.images.return_value = [] client_mock.logs.return_value = ['container log'] - client_mock.pull.return_value = ['{"status":"pull log"}'] + client_mock.pull.return_value = [b'{"status":"pull log"}'] client_mock.wait.return_value = 0 client_class_mock.return_value = client_mock