From user-return-1267-archive-asf-public=cust-asf.ponee.io@kudu.apache.org Tue Feb 27 03:09:46 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 1466118064A for ; Tue, 27 Feb 2018 03:09:45 +0100 (CET) Received: (qmail 45921 invoked by uid 500); 27 Feb 2018 02:09:45 -0000 Mailing-List: contact user-help@kudu.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@kudu.apache.org Delivered-To: mailing list user@kudu.apache.org Received: (qmail 45880 invoked by uid 99); 27 Feb 2018 02:09:44 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Feb 2018 02:09:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id BF96DC6185 for ; Tue, 27 Feb 2018 02:09:43 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.148 X-Spam-Level: ** X-Spam-Status: No, score=2.148 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id SvreQkUcUoiB for ; Tue, 27 Feb 2018 02:09:43 +0000 (UTC) Received: from mail-qk0-f172.google.com (mail-qk0-f172.google.com [209.85.220.172]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id A8EB55FAD9 for ; Tue, 27 Feb 2018 02:09:42 +0000 (UTC) Received: by mail-qk0-f172.google.com with SMTP id 130so16878197qkd.13 for ; Mon, 26 Feb 2018 18:09:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=NQaQlpSZnvBZuAeJx3NX38RVW26v0Xv+OI70JEX5Mvo=; b=ij6IohNZ8cuPs4zffyyDuDSzlvKFbwByYlmo5b6A2t+Kep+p+GEsXG/Gl4rpCiYnWL JOYuYdehj0jFe5TuCw1JwWDSahSo3v5XoLg31D5egrw22LiaXSLWx66uUHGlGnkGyfLP EJstuzeo5jwwajgp2I6K2yO3TTAsyHzqoO6fLpDTOqzhURWzjPYF2Qaf1UZtRuG3E8D6 IsvNjyzpVcUxLY3mYV6IMDlief7TGE08V2QWVoljekx6zoz4lH/kbfI5OU8C1fNJD1G3 DQFgMtBLCTb5DiwVyh+w2DfExQtZfhg36gnyOU2Z1d/msV+oC77yH1Bet7FHJD71/No5 foSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=NQaQlpSZnvBZuAeJx3NX38RVW26v0Xv+OI70JEX5Mvo=; b=OQbSg0KMXud68Ra5RmdIjqvktujvI1NkHzBNKTcHYNGpqsz3Z1v7Pb0jxoMRow8Ckx pif+ceuibW1RZPq3ZeOyaaOph4rAcjnEZXBW7HB6BAiSvxIfRIKenUQIBLbQ8w5WEe5R a5pVC8G8UTlkkMLAjSPC36i5GR6zqbxV4MT5tptGECSdrK69lFlNccizXlVvZ70a0BaQ V2ctLw8vgncxnb9BURr2WFAQfCaXFvtWxgIBhhuSr2dZiN6GGYxEEM9G5STXonPMt2qR +BiQsJ+hBUbgr4NUw6ENm9zdJqZVc109FO39VAJ6HY6/0AoFNJzsAxNlWMbgtVqJxWHa 0gXQ== X-Gm-Message-State: APf1xPDY+E1dM250VneqL/L0dIpIcy3UEFi//eIXQaPzZM1prxp5ZnyZ gkr4N5b9s7t2uElD0IsN/Ut3Ec2O6ld/wq4/FQo= X-Google-Smtp-Source: AG47ELv6Tb4V5xnSFvpUxN3ta9JE8G+pR246sOM5JMxpxRHps/B/TzYvdwDDXgFrCTDwMMOOA5HrIqrFwUMab8vwPHE= X-Received: by 10.55.167.143 with SMTP id q137mr20664900qke.27.1519697382082; Mon, 26 Feb 2018 18:09:42 -0800 (PST) MIME-Version: 1.0 Received: by 10.140.41.42 with HTTP; Mon, 26 Feb 2018 18:09:41 -0800 (PST) In-Reply-To: References: From: Ravi Kanth Date: Mon, 26 Feb 2018 18:09:41 -0800 Message-ID: Subject: Re: Spark Streaming + Kudu To: user@kudu.apache.org Content-Type: multipart/alternative; boundary="001a114d8c86a00b670566281ef1" --001a114d8c86a00b670566281ef1 Content-Type: text/plain; charset="UTF-8" Cliff, Thanks for the response. Well, I do agree that its simple and seamless. In my case, I am able to upsert ~25000 events/sec into Kudu. But, I am facing the problem when any of the Kudu Tablet or master server is down. I am not able to get a hold of the exception from client. The client is going into an infinite loop trying to connect to Kudu. Meanwhile, I am loosing my records. I tried handling the errors through getPendingErrors() but still it is helpless. I am using AsyncKuduClient to establish the connection and retrieving the syncClient from the Async to open the session and table. Any help? Thanks, Ravi On 26 February 2018 at 18:00, Cliff Resnick wrote: > While I can't speak for Spark, we do use the client API from Flink > streaming and it's simple and seamless. It's especially nice if you require > an Upsert semantic. > > On Feb 26, 2018 7:51 PM, "Ravi Kanth" wrote: > >> Hi, >> >> Anyone using Spark Streaming to ingest data into Kudu and using Kudu >> Client API to do so rather than the traditional KuduContext API? I am stuck >> at a point and couldn't find a solution. >> >> Thanks, >> Ravi >> > --001a114d8c86a00b670566281ef1 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Cliff,

Thanks for the response. Well, I= do agree that its simple and seamless. In my case, I am able to upsert ~25= 000 events/sec into Kudu. But, I am facing the problem when any of the Kudu= Tablet or master server is down. I am not able to get a hold of the except= ion from client. The client is going into an infinite loop trying to connec= t to Kudu. Meanwhile, I am loosing my records. I tried handling the errors = through getPendingErrors() but still it is helpless. I am using AsyncKuduCl= ient to establish the connection and retrieving the syncClient from the Asy= nc to open the session and table. Any help?=C2=A0

= Thanks,
Ravi

On 26 February 2018 at 18:00, Cliff Resnick <cresny@gmail.= com> wrote:
While I can't speak for Spark, we do use the client API from Flink s= treaming and it's simple and seamless. It's especially nice if you = require an Upsert semantic.

On Feb 26, 2018 7:5= 1 PM, "Ravi Kanth" <ravikanth.4b0@gmail.com> wrote:
Hi,

Anyone using Spark Streaming to ingest data into Kudu and using Kudu Cli= ent API to do so rather than the traditional KuduContext API? I am stuck at= a point and couldn't find a solution.=C2=A0

T= hanks,
Ravi

--001a114d8c86a00b670566281ef1--