From dev-return-6479-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Mon Sep 10 10:20:08 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 60A9C180656 for ; Mon, 10 Sep 2018 10:20:07 +0200 (CEST) Received: (qmail 4267 invoked by uid 500); 10 Sep 2018 08:20:05 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Received: (qmail 4157 invoked by uid 99); 10 Sep 2018 08:20:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Sep 2018 08:20:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 036EAC7CC6 for ; Mon, 10 Sep 2018 08:20:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.859 X-Spam-Level: ** X-Spam-Status: No, score=2.859 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_HUGEIMGSRC=0.2, KAM_INFOUSMEBIZ=0.75, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, T_REMOTE_IMAGE=0.01] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=getyourguide.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id VVZH_8zvBqXi for ; Mon, 10 Sep 2018 08:20:02 +0000 (UTC) Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 8651C5F48E for ; Mon, 10 Sep 2018 08:20:01 +0000 (UTC) Received: by mail-pl1-f172.google.com with SMTP id b30-v6so7097260pla.0 for ; Mon, 10 Sep 2018 01:20:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=getyourguide.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=eFNiFCLHLkzIY7FEbA1cG9hKjWnnQwt1/RxqJhbA0UY=; b=adJa3pgGS+v9ji8wW3hpFH/wVC00odB5eRrhncrB7T9ToXD8L4oLMtWjgnWRx73cAh Q1xsD28fHO7SPA67BoyjLXh3v7A2A2G8QWh9Ktwl4Dz9ZfNcCHV1Kyh+WLyaHYMOolDw 5X/O1esHEt9byg7A3DMJKX/ttZzRkh7oqaVxE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=eFNiFCLHLkzIY7FEbA1cG9hKjWnnQwt1/RxqJhbA0UY=; b=HKKd5b2Iv0NtFC6gs359Jyxi8/rLNWUPfisvcsYv/j+URaL2UQIJYuZKdo9/UeBuG3 XPuTMjlJ6q84uUueTi5VG+3SWC9XK95pEcTeLzKA4M//bJ84WN0tDQmzVVSzmMD1TZS3 t4/2u+fcsnVlMt92EKgpNL9Nvl5lYKN/Iq3wwx3kwSfMwf6+1N45KcImFaEQnjLjzkSC m+jgz8WaMWeNveFBxf2Mj6GaEoZYrmaab+sfsS68Gd5c7cwgWF6/ECnwniB4y7ylNFVS xT/v1J+b9BhIeko6gVePWoXByY6uwB2DbrvJsI150p/Op9a6TlxEK3oB8ck7tKxhjvL3 Knqg== X-Gm-Message-State: APzg51Bgcl/YVgULID+Qz/ZCAt1OtGL+AnBzIyndtjOGFPdmOvqsdP1g hSmr+YZCOIKa3CqVSdjw993a2y2A2EdQS6NEuDl+mc91r7SZFGj5gIZPbJOO2eZiyGR94lagZSn Qp8+wT/B3fR6s2yI71LAvGTDv+HsN08YayVeJNLrFjc4= X-Google-Smtp-Source: ANB0Vda0pTTvHQ6Gsvka9t2O12zIHzOHDGtFvHy17Hah6OT1wApuD/BN5bxxt2dD1GG/AStrLMqDpVyTh3aEMwlTPvs= X-Received: by 2002:a17:902:8542:: with SMTP id d2-v6mr20893283plo.285.1536567599762; Mon, 10 Sep 2018 01:19:59 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Emmanuel Brard Date: Mon, 10 Sep 2018 10:19:48 +0200 Message-ID: Subject: Re: Use xcom in task retry To: dev@airflow.incubator.apache.org Content-Type: multipart/alternative; boundary="000000000000f5220f057580056c" --000000000000f5220f057580056c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello Mishika, I think we had the same use case in which we wanted an operator to use the xcom values pushed from a previous try (after a retry) and also found out that it is cleared before an execution. What we did is to extend Airflow data model to hold the data we wanted to persist between retries in another table. That is a bit hacky but it's been working fine so far. Bye, E On Mon, Sep 10, 2018 at 9:05 AM Mishika Singh wrote: > Hi Taylor, > > I am providing the code to show how I am using xcom pull and push, yes th= e > push is succeeding to the database. > But, the behaviour that I observed is when execute() is called for retry, > xcom values are deleted from the table for combination of execution_date> > Then I searched for this in the Airflow code and I found that > clear_xcom_data() is called before the execute() starts.(providing this > code as well). We are using Airflow 1.9 > > def execute(self, context, **kwargs): > > pull_response =3D ti.xcom_pull(key=3D'computeResponse', task_ids=3DNo= ne) > logging.info("pull pull_response %s", pull_response) > > ti.xcom_push('computeResponse', self.compute_response) > > > ################ > > def clear_xcom_data(self, session=3DNone): > """ > Clears all XCom data from the database for the task instance > """ > self.log.info('------------------->>>>> c') > session.query(XCom).filter( > XCom.dag_id =3D=3D self.dag_id, > XCom.task_id =3D=3D self.task_id, > XCom.execution_date =3D=3D self.execution_date > ).delete() > session.commit() > > > On Sun, Sep 9, 2018 at 1:20 AM Taylor Edmiston > wrote: > > > XComs push and pull should work as expected when a task is retried. It > > shouldn't make a difference, but are you using XComs with explicit keys > or > > the implicit return-based style? Is the push succeeding to the database= ? > > > > Also can you show a simplified example of your code for this DAG? > > > > Taylor > > > > On Thu, Sep 6, 2018 at 12:48 PM Ben Gregory wrote: > > > > > Hi Mishika -- > > > > > > Posting this question on StackOverflow with some code examples you're > > using > > > will likely be the fastest way to have this addressed. > > > > > > Just make sure to tag it with "airflow" so people can find it. > > > > > > - Ben > > > > > > On Thu, Sep 6, 2018 at 12:12 AM Mishika Singh > > wrote: > > > > > > > I am pushing some [key,value] using xcom in an operator, which I wa= nt > > to > > > > use when that task fails and comes for retry, for which I am using > > > > xcom_pull in that operator. But it is returning None instead of tha= t > > > value. > > > > Any pointer around this will be helpful. > > > > > > > > > > > > -- > > > > Regards > > > > Mishika Singh > > > > > > > > > > > > > -- > > > > > > [image: Astronomer Logo] > > > > > > *Ben Gregory* > > > Data Engineer > > > > > > Mobile: +1-615-483-3653 =E2=80=A2 Online: astronomer.io < > > > https://www.astronomer.io/> > > > > > > Download our new ebook. From > > > Volume > > > to Value - A Guide to Data Engineering. > > > > > -- > > *Taylor Edmiston* > > Blog | CV > > | LinkedIn > > | AngelList > > | Stack Overflow > > > > > > > -- > Regards > Mishika Singh > --=20 GetYourGuide AG Stampfenbachstrasse 48 =C2=A0 8006 Z=C3=BCrich Switzerland =20 =20 =20 =20 --000000000000f5220f057580056c--