Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 72228200BE7 for ; Tue, 20 Dec 2016 14:14:07 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 70C28160B29; Tue, 20 Dec 2016 13:14:07 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 9553F160B1B for ; Tue, 20 Dec 2016 14:14:06 +0100 (CET) Received: (qmail 71724 invoked by uid 500); 20 Dec 2016 13:14:00 -0000 Mailing-List: contact users-help@apex.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@apex.apache.org Delivered-To: mailing list users@apex.apache.org Received: (qmail 71712 invoked by uid 99); 20 Dec 2016 13:14:00 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Dec 2016 13:14:00 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 0974518C367 for ; Tue, 20 Dec 2016 13:14:00 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.011 X-Spam-Level: X-Spam-Status: No, score=0.011 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_NONE=-0.0001, RP_MATCHES_RCVD=-2.999, T_KAM_HTML_FONT_INVALID=0.01] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 5RN54WD9Bzkn for ; Tue, 20 Dec 2016 13:13:58 +0000 (UTC) Received: from EMP-EXED102.leidos.com (emp-exed102.leidos.com [149.8.144.42]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 092E85FC0E for ; Tue, 20 Dec 2016 13:13:58 +0000 (UTC) Received: from EMP-EXMR101.corp.leidos.com (10.128.180.232) by EMP-EXED102.leidos.com (149.8.144.42) with Microsoft SMTP Server (TLS) id 14.3.224.2; Tue, 20 Dec 2016 07:13:51 -0600 Received: from EMP-EXMR102.corp.leidos.com ([fe80::982d:e179:81a5:4a9]) by EMP-EXMR101.corp.leidos.com ([fe80::c93a:36dc:3bae:c8b1%28]) with mapi id 14.03.0224.002; Tue, 20 Dec 2016 07:13:50 -0600 From: "Doyle, Austin O." To: "users@apex.apache.org" Subject: RE: Data duplication between operators Thread-Topic: Data duplication between operators Thread-Index: AdJaLZ6aPWwkd2nnQtqOfIWRkdWWeAAVec8AAA/Gd+A= Date: Tue, 20 Dec 2016 13:13:50 +0000 Message-ID: <034E9AAEC5E6C8439F2C151754F62BBD22E6D72B@EMP-EXMR102.corp.leidos.com> References: <034E9AAEC5E6C8439F2C151754F62BBD22E6D641@EMP-EXMR102.corp.leidos.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.226.129.23] Content-Type: multipart/alternative; boundary="_000_034E9AAEC5E6C8439F2C151754F62BBD22E6D72BEMPEXMR102corpl_" MIME-Version: 1.0 archived-at: Tue, 20 Dec 2016 13:14:07 -0000 --_000_034E9AAEC5E6C8439F2C151754F62BBD22E6D72BEMPEXMR102corpl_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable The downstream operator doesn't seem to be failing and through some local t= ests I can confirm that each operator works separately. Could it be someth= ing else? -Austin From: Vlad Rozov [mailto:v.rozov@datatorrent.com] Sent: Monday, December 19, 2016 6:40 PM To: users@apex.apache.org Subject: Re: Data duplication between operators This will be a bug unless the downstream operator constantly fails and is r= estored to a checkpoint in which case it is expected that it may get the sa= me tuple multiple times. Thank you, Vlad On 12/19/16 11:33, Doyle, Austin O. wrote: I'm trying to send some sequential data between an S3 Input Operator and a = CSV Parser operator. I added logging to see what the outputPort is emittin= g and it seems to be straightforward (data points 1 to 1000). I added logg= ing on the input of the CSV Parser which receives 1000 data points but not = the correct data points. It actually receives random data points multiple = times (like point 57 twenty or so times). Has anyone seen anything like th= is? Thanks, Austin --_000_034E9AAEC5E6C8439F2C151754F62BBD22E6D72BEMPEXMR102corpl_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

The downstream operato= r doesn’t seem to be failing and through some local tests I can confi= rm that each operator works separately.  Could it be something else?

 

-Austin

 

From: Vlad Rozov [mailto:v.rozov@datatorrent.com]
Sent: Monday, December 19, 2016 6:40 PM
To: users@apex.apache.org
Subject: Re: Data duplication between operators

 

This will be a bug un= less the downstream operator constantly fails and is restored to a checkpoi= nt in which case it is expected that it may get the same tuple multiple tim= es.

Thank you,

Vlad

On 12/19/16 11:33, Doyle, Austin O. wrote:

I’m trying to send some sequential data betwee= n an S3 Input Operator and a CSV Parser operator.  I added logging to = see what the outputPort is emitting and it seems to be straightforward (dat= a points 1 to 1000).  I added logging on the input of the CSV Parser which receives 1000 data points but not the correc= t data points.  It actually receives random data points multiple times= (like point 57 twenty or so times).  Has anyone seen anything like th= is?

 

Thanks,

Austin

 

--_000_034E9AAEC5E6C8439F2C151754F62BBD22E6D72BEMPEXMR102corpl_--