From commits-return-58811-archive-asf-public=cust-asf.ponee.io@beam.apache.org Fri Feb 16 11:03:04 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id DEB7E18067B for ; Fri, 16 Feb 2018 11:03:03 +0100 (CET) Received: (qmail 86894 invoked by uid 500); 16 Feb 2018 10:03:03 -0000 Mailing-List: contact commits-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list commits@beam.apache.org Received: (qmail 86859 invoked by uid 99); 16 Feb 2018 10:03:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Feb 2018 10:03:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 8336B1A00FE for ; Fri, 16 Feb 2018 10:03:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -102.311 X-Spam-Level: X-Spam-Status: No, score=-102.311 tagged_above=-999 required=6.31 tests=[RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 64JMRaaMGxWA for ; Fri, 16 Feb 2018 10:03:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 13DB45F183 for ; Fri, 16 Feb 2018 10:03:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 989D7E00A6 for ; Fri, 16 Feb 2018 10:03:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 5DEC121E5A for ; Fri, 16 Feb 2018 10:03:00 +0000 (UTC) Date: Fri, 16 Feb 2018 10:03:00 +0000 (UTC) From: =?utf-8?Q?Isma=C3=ABl_Mej=C3=ADa_=28JIRA=29?= To: commits@beam.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Resolved] (BEAM-2582) KinesisIO incorrectly handles closed shards MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/BEAM-2582?page=3Dcom.atlassian= .jira.plugin.system.issuetabpanels:all-tabpanel ] Isma=C3=ABl Mej=C3=ADa resolved BEAM-2582. -------------------------------- Resolution: Invalid Fix Version/s: Not applicable Closing this because it is now invalid after the merge of BEAM-2469 > KinesisIO incorrectly handles closed shards > ------------------------------------------- > > Key: BEAM-2582 > URL: https://issues.apache.org/jira/browse/BEAM-2582 > Project: Beam > Issue Type: Bug > Components: io-java-kinesis > Reporter: adam gray > Priority: Major > Fix For: Not applicable > > > The KinesisIO throws an exception when consuming closed Kinesis shards, w= hich return null from `GetShardIterator`, as it tries to call `GetRecords` = with the null `shardIterator` value instead of abandoning the closed shard. > This means KinesisIO fails after re-sharding a stream with an exception l= ike the following: > {noformat} > Exception in thread "main" java.lang.RuntimeException: Kinesis client sid= e failure > =09at org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.wrapExceptio= ns(SimplifiedKinesisClient.java:151) > =09at org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.getRecords(S= implifiedKinesisClient.java:115) > =09at org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.getRecords(S= implifiedKinesisClient.java:102) > =09at org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readMoreIfNeces= sary(ShardRecordsIterator.java:79) > =09at org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.next(ShardRecor= dsIterator.java:64) > =09at org.apache.beam.sdk.io.kinesis.KinesisReader.advance(KinesisReader.= java:86) > =09at org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$Unboun= dedReadEvaluator.startReader(UnboundedReadEvaluatorFactory.java:190) > =09at org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$Unboun= dedReadEvaluator.processElement(UnboundedReadEvaluatorFactory.java:128) > =09at org.apache.beam.runners.direct.TransformExecutor.processElements(Tr= ansformExecutor.java:139) > =09at org.apache.beam.runners.direct.TransformExecutor.run(TransformExecu= tor.java:107) > =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:= 511) > =09at java.util.concurrent.FutureTask.run(FutureTask.java:266) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1142) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:617) > =09at java.lang.Thread.run(Thread.java:745) > Caused by: com.amazonaws.AmazonServiceException: 1 validation error detec= ted: Value null at 'shardIterator' failed to satisfy constraint: Member mus= t not be null (Service: AmazonKinesis; Status Code: 400; Error Code: Valida= tionException; Request ID: d764e747-9616-5db3-86ba-08a0bc44cb39) > =09at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpC= lient.java:1378) > =09at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpCli= ent.java:924) > =09at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.= java:702) > =09at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java= :454) > =09at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClie= nt.java:416) > =09at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:3= 65) > =09at com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonK= inesisClient.java:2016) > =09at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKin= esisClient.java:1986) > =09at com.amazonaws.services.kinesis.AmazonKinesisClient.getRecords(Amazo= nKinesisClient.java:985) > =09at org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient$3.call(Simpl= ifiedKinesisClient.java:118) > =09at org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient$3.call(Simpl= ifiedKinesisClient.java:115) > =09at org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.wrapExceptio= ns(SimplifiedKinesisClient.java:140) > =09... 14 more > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)