From dev-return-4593-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Wed Feb 21 21:58:04 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 5838518061A for ; Wed, 21 Feb 2018 21:58:04 +0100 (CET) Received: (qmail 8444 invoked by uid 500); 21 Feb 2018 20:58:03 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Received: (qmail 7873 invoked by uid 99); 21 Feb 2018 20:58:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Feb 2018 20:58:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id D44D3180414 for ; Wed, 21 Feb 2018 20:58:01 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.379 X-Spam-Level: ** X-Spam-Status: No, score=2.379 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 6Rj9YGFzqK3f for ; Wed, 21 Feb 2018 20:58:01 +0000 (UTC) Received: from mail-ua0-f170.google.com (mail-ua0-f170.google.com [209.85.217.170]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 5FFD95F27B for ; Wed, 21 Feb 2018 20:58:00 +0000 (UTC) Received: by mail-ua0-f170.google.com with SMTP id e25so1939267uan.5 for ; Wed, 21 Feb 2018 12:58:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=/FFMB+hSZQgfTrG/5mz5u2QJ33RlWo1PjxT9p7p6R6U=; b=OhTmAzfiBCzg8tNN8drt9EL9M55tTkrkGKizufvESBtBCrjF5vIgpQlPTEpahNDJBb MkuL0RVDDo5S927QBbj0WzixUEdrBkG2rhkqOQfCHms0t///yqIcWsefzPnhWnqHilrp d6eaaXsxg5sP2xWCMJzRNJqDOSvQMn9K5spfEg8zALNcUxb7uSaqyrm0TEG7l9z1N8Kc jCuze8s4gYz9LaGpOVM9uNKhnOwKiFV4haWLdcc8L1vMQY946wzso5+HL9Z512/0E/rk BnKKodI6bmBO8zA2486AdZ56gG2Lt+qBkp3hE6f5rQX7OB1oKzGNN+bY13IwpSKHBLUe vJBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=/FFMB+hSZQgfTrG/5mz5u2QJ33RlWo1PjxT9p7p6R6U=; b=c0bSWdReFbzneweLQJVTRGlP1CsnNu1tod96bn4fGJeHbodAQ53IrUYjIkAKZJTD2v oFOIYgfTaaC/t0jG4KK2eEhT7JzfJru2pkmPDNW8VTpB30wd/IdAahutsTpMrXeHG70p sFcCzWFcQScWgiLOuzRjuoGGzUlJkOBj/maR2teHoJqtoMXTR0RdooobR+MaXsv3xK8t QrDYbIJ60Z1o9f2YdxUIN6w4WBtfnYHzvkJDSX0Q9O4TfDfsAroLqH0vbEHHMDaGrG9L uYqRCI6kAc22i3Sy6YTx8cbFGS0pIU7slS6f0fOt+zUJsBEiFe5YG1BnwyqT9rVQaQRD GHGw== X-Gm-Message-State: APf1xPAHeX4j0sY9cFOgZNtGcuKGLWwXjbEEDSHzZ/xTFGwiG2vF5vqf uczJ9/baqTHHzj5chRP/QU+rmhyjHV3gATwnY8sNcQ== X-Google-Smtp-Source: AH8x227HMssl2rVUXsfnOB6q/lUUJwxA9i6deKgcEZ8PEYfQm24LOk4J7p3ZHD5Ih7vrqBk80FEHQ3l2H4CJt3ufuBY= X-Received: by 10.176.76.129 with SMTP id y1mr3191125uaf.152.1519246673087; Wed, 21 Feb 2018 12:57:53 -0800 (PST) MIME-Version: 1.0 From: Matthew Housley Date: Wed, 21 Feb 2018 20:57:42 +0000 Message-ID: Subject: Re: troubleshooting hung docker airflow containers on 1.9.0 To: dev@airflow.apache.org Content-Type: multipart/alternative; boundary="f403045f8bd64690550565bf2ea0" --f403045f8bd64690550565bf2ea0 Content-Type: text/plain; charset="UTF-8" > We don't have a lot of dags and they are very simple, but one of them runs every minute> > and others run every 15 or every hour.> I would start by looking at your scheduler container. You should be able to pull the container logs and look for evidence that it hangs and stops looping over dags. How are you hosting the containers, e.g., Docker Compose, Docker Swarm, ECS, Kubernetes? --f403045f8bd64690550565bf2ea0--