Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id B4E22200CBB for ; Tue, 4 Jul 2017 10:52:08 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id B378B160BE1; Tue, 4 Jul 2017 08:52:08 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id D3941160BEF for ; Tue, 4 Jul 2017 10:52:07 +0200 (CEST) Received: (qmail 25569 invoked by uid 500); 4 Jul 2017 08:52:07 -0000 Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list commits@airflow.incubator.apache.org Received: (qmail 25428 invoked by uid 99); 4 Jul 2017 08:52:06 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Jul 2017 08:52:06 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id CD4A1D0CCA for ; Tue, 4 Jul 2017 08:52:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.011 X-Spam-Level: X-Spam-Status: No, score=-100.011 tagged_above=-999 required=6.31 tests=[SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id Ub9en18fqKc0 for ; Tue, 4 Jul 2017 08:52:03 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 7BD5F5FBDF for ; Tue, 4 Jul 2017 08:52:03 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id D66AFE0D7D for ; Tue, 4 Jul 2017 08:52:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 303782460D for ; Tue, 4 Jul 2017 08:52:00 +0000 (UTC) Date: Tue, 4 Jul 2017 08:52:00 +0000 (UTC) From: "Maneesh Sharma (JIRA)" To: commits@airflow.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (AIRFLOW-1370) Scheduler is crashing because of IntegrityError MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 04 Jul 2017 08:52:08 -0000 Maneesh Sharma created AIRFLOW-1370: --------------------------------------- Summary: Scheduler is crashing because of IntegrityError Key: AIRFLOW-1370 URL: https://issues.apache.org/jira/browse/AIRFLOW-1370 Project: Apache Airflow Issue Type: Bug Components: celery, scheduler Affects Versions: Airflow 1.8 Reporter: Maneesh Sharma Scheduler is crashing with multiple task running on Celery Executor. It is = throwing `{color:red}IntegrityError: (psycopg2.IntegrityError) duplicate ke= y value violates unique constraint "task_instance_pkey"{color}`. Below is t= he complete stack trace of error -- Process DagFileProcessor490-Process: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _boots= trap self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/jobs.py", l= ine 348, in helper pickle_dags) File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/utils/db.py= ", line 53, in wrapper result =3D func(*args, **kwargs) File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/jobs.py", l= ine 1587, in process_file self._process_dags(dagbag, dags, ti_keys_to_schedule) File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/jobs.py", l= ine 1176, in _process_dags self._process_task_instances(dag, tis_out) File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/jobs.py", l= ine 880, in _process_task_instances run.verify_integrity(session=3Dsession) File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/utils/db.py= ", line 53, in wrapper result =3D func(*args, **kwargs) File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/models.py",= line 4117, in verify_integrity session.commit() File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/orm/sess= ion.py", line 906, in commit self.transaction.commit() File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/orm/sess= ion.py", line 461, in commit self._prepare_impl() File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/orm/sess= ion.py", line 441, in _prepare_impl self.session.flush() File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/orm/sess= ion.py", line 2171, in flush self._flush(objects) File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/orm/sess= ion.py", line 2291, in _flush transaction.rollback(_capture_exception=3DTrue) File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/util/lan= ghelpers.py", line 66, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/orm/sess= ion.py", line 2255, in _flush flush_context.execute() File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/orm/unit= ofwork.py", line 389, in execute rec.execute(self) File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/orm/unit= ofwork.py", line 548, in execute uow File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/orm/pers= istence.py", line 181, in save_obj mapper, table, insert) File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/orm/pers= istence.py", line 799, in _emit_insert_statements execute(statement, multiparams) File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/b= ase.py", line 945, in execute return meth(self, multiparams, params) File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/sql/elem= ents.py", line 263, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/b= ase.py", line 1053, in _execute_clauseelement compiled_sql, distilled_params File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/b= ase.py", line 1189, in _execute_context context) File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/b= ase.py", line 1402, in _handle_dbapi_exception exc_info File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/util/com= pat.py", line 203, in raise_from_cause reraise(type(exception), exception, tb=3Dexc_tb, cause=3Dcause) File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/b= ase.py", line 1159, in _execute_context context) File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/engine/d= efault.py", line 467, in do_executemany cursor.executemany(statement, parameters) IntegrityError: (psycopg2.IntegrityError) duplicate key value violates uniq= ue constraint "task_instance_pkey" DETAIL: Key (task_id, dag_id, execution_date)=3D(Hello_World_task-21, Hell= o_World_Tasks, 2017-07-04 06:59:40) already exists. [SQL: 'INSERT INTO task_instance (task_id, dag_id, execution_date, start_d= ate, end_date, duration, state, try_number, hostname, unixname, job_id, poo= l, queue, priority_weight, operator, queued_dttm, pid) VALUES (%(task_id)s,= %(dag_id)s, %(execution_date)s, %(start_date)s, %(end_date)s, %(duration)s= , %(state)s, %(try_number)s, %(hostname)s, %(unixname)s, %(job_id)s, %(pool= )s, %(queue)s, %(priority_weight)s, %(operator)s, %(queued_dttm)s, %(pid)s)= '] [parameters: ({'task_id': 'Hello_World_task-21', 'unixname': 'ubuntu', '= job_id': None, 'end_date': None, 'pool': None, 'queued_dttm': None, 'execut= ion_date': datetime.datetime(2017, 7, 4, 6, 59, 40), 'pid': None, 'try_numb= er': 0, 'queue': 'default', 'duration': None, 'state': None, 'start_date': = None, 'operator': None, 'priority_weight': 2, 'hostname': u'', 'dag_id': 'H= ello_World_Tasks'}, {'task_id': 'Hello_World_task-20', 'unixname': 'ubuntu'= , 'job_id': None, 'end_date': None, 'pool': None, 'queued_dttm': None, 'exe= cution_date': datetime.datetime(2017, 7, 4, 6, 59, 40), 'pid': None, 'try_n= umber': 0, 'queue': 'default', 'duration': None, 'state': None, 'start_date= ': None, 'operator': None, 'priority_weight': 2, 'hostname': u'', 'dag_id':= 'Hello_World_Tasks'}, {'task_id': 'Hello_World_task-23', 'unixname': 'ubun= tu', 'job_id': None, 'end_date': None, 'pool': None, 'queued_dttm': None, '= execution_date': datetime.datetime(2017, 7, 4, 6, 59, 40), 'pid': None, 'tr= y_number': 0, 'queue': 'default', 'duration': None, 'state': None, 'start_d= ate': None, 'operator': None, 'priority_weight': 2, 'hostname': u'', 'dag_i= d': 'Hello_World_Tasks'}, {'task_id': 'Hello_World_task-22', 'unixname': 'u= buntu', 'job_id': None, 'end_date': None, 'pool': None, 'queued_dttm': None= , 'execution_date': datetime.datetime(2017, 7, 4, 6, 59, 40), 'pid': None, = 'try_number': 0, 'queue': 'default', 'duration': None, 'state': None, 'star= t_date': None, 'operator': None, 'priority_weight': 2, 'hostname': u'', 'da= g_id': 'Hello_World_Tasks'}, {'task_id': 'Hello_World_task-25', 'unixname':= 'ubuntu', 'job_id': None, 'end_date': None, 'pool': None, 'queued_dttm': N= one, 'execution_date': datetime.datetime(2017, 7, 4, 6, 59, 40), 'pid': Non= e, 'try_number': 0, 'queue': 'default', 'duration': None, 'state': None, 's= tart_date': None, 'operator': None, 'priority_weight': 2, 'hostname': u'', = 'dag_id': 'Hello_World_Tasks'}, {'task_id': 'Hello_World_task-24', 'unixnam= e': 'ubuntu', 'job_id': None, 'end_date': None, 'pool': None, 'queued_dttm'= : None, 'execution_date': datetime.datetime(2017, 7, 4, 6, 59, 40), 'pid': = None, 'try_number': 0, 'queue': 'default', 'duration': None, 'state': None,= 'start_date': None, 'operator': None, 'priority_weight': 2, 'hostname': u'= ', 'dag_id': 'Hello_World_Tasks'}, {'task_id': 'Hello_World_task-27', 'unix= name': 'ubuntu', 'job_id': None, 'end_date': None, 'pool': None, 'queued_dt= tm': None, 'execution_date': datetime.datetime(2017, 7, 4, 6, 59, 40), 'pid= ': None, 'try_number': 0, 'queue': 'default', 'duration': None, 'state': No= ne, 'start_date': None, 'operator': None, 'priority_weight': 2, 'hostname':= u'', 'dag_id': 'Hello_World_Tasks'}, {'task_id': 'Hello_World_task-26', 'u= nixname': 'ubuntu', 'job_id': None, 'end_date': None, 'pool': None, 'queued= _dttm': None, 'execution_date': datetime.datetime(2017, 7, 4, 6, 59, 40), '= pid': None, 'try_number': 0, 'queue': 'default', 'duration': None, 'state':= None, 'start_date': None, 'operator': None, 'priority_weight': 2, 'hostnam= e': u'', 'dag_id': 'Hello_World_Tasks'} ... displaying 10 of 2002 total bo= und parameter sets ... {'task_id': 'Hello_World_task-1783', 'unixname': 'u= buntu', 'job_id': None, 'end_date': None, 'pool': None, 'queued_dttm': None= , 'execution_date': datetime.datetime(2017, 7, 4, 6, 59, 40), 'pid': None, = 'try_number': 0, 'queue': 'default', 'duration': None, 'state': None, 'star= t_date': None, 'operator': None, 'priority_weight': 2, 'hostname': u'', 'da= g_id': 'Hello_World_Tasks'}, {'task_id': 'Hello_World_task-1782', 'unixname= ': 'ubuntu', 'job_id': None, 'end_date': None, 'pool': None, 'queued_dttm':= None, 'execution_date': datetime.datetime(2017, 7, 4, 6, 59, 40), 'pid': N= one, 'try_number': 0, 'queue': 'default', 'duration': None, 'state': None, = 'start_date': None, 'operator': None, 'priority_weight': 2, 'hostname': u''= , 'dag_id': 'Hello_World_Tasks'})] =20 -- This message was sent by Atlassian JIRA (v6.4.14#64029)