Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 1A5B6200CB3 for ; Mon, 26 Jun 2017 18:18:15 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 19428160BDE; Mon, 26 Jun 2017 16:18:15 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 39609160BD9 for ; Mon, 26 Jun 2017 18:18:14 +0200 (CEST) Received: (qmail 90838 invoked by uid 500); 26 Jun 2017 16:18:13 -0000 Mailing-List: contact user-help@kudu.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@kudu.apache.org Delivered-To: mailing list user@kudu.apache.org Received: (qmail 90828 invoked by uid 99); 26 Jun 2017 16:18:13 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jun 2017 16:18:13 +0000 Received: from mail-ua0-f173.google.com (mail-ua0-f173.google.com [209.85.217.173]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 937791A00A2 for ; Mon, 26 Jun 2017 16:18:12 +0000 (UTC) Received: by mail-ua0-f173.google.com with SMTP id 70so3848146uau.0 for ; Mon, 26 Jun 2017 09:18:11 -0700 (PDT) X-Gm-Message-State: AKS2vOykWWTZlkeXGwrhqwWB3vChx/3B+q+oUq65TvdytaV+uPvcxmy/ GTBxzaaB0eLV3IQgc9nhOLCWeaxA3g== X-Received: by 10.176.27.81 with SMTP id n17mr573568uai.58.1498493889391; Mon, 26 Jun 2017 09:18:09 -0700 (PDT) MIME-Version: 1.0 Received: by 10.176.86.26 with HTTP; Mon, 26 Jun 2017 09:18:08 -0700 (PDT) In-Reply-To: References: From: Jean-Daniel Cryans Date: Mon, 26 Jun 2017 09:18:08 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Spark locality issue To: user@kudu.apache.org Content-Type: multipart/alternative; boundary="f403043651a6f9b8f90552df4bca" archived-at: Mon, 26 Jun 2017 16:18:15 -0000 --f403043651a6f9b8f90552df4bca Content-Type: text/plain; charset="UTF-8" On Mon, Jun 26, 2017 at 8:53 AM, Jean-Daniel Cryans wrote: > Hi Pavel, > > I think the whole Kudu/Spark story needs more attention, for example Spark > SQL query plans don't have access to any Kudu stats so you can end up with > some really bad join decisions. > > It feels like KUDU-1454 should be really easy to solve at this point. What > we need is to get the RDD to use CLOSEST_REPLICA and to set a propagated > timestamp like Todd says in the jira. This is all stuff that's done in > Impala's integration for Kudu. If you wanted to see if that solves your > problem you could add the following code on this line http://github.mtv. > cloudera.com/CDH/kudu/blob/cdh5-trunk/java/kudu-client/ > src/main/java/org/apache/kudu/client/KuduScanToken.java#L226 > Of course I meant a link more like this https://github.com/apache/kudu/blob/master/java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java#L226 > > builder.replicaSelection(ReplicaSelection.CLOSEST_REPLICA); > > The propagated timestamp part is also needed but only for consistency > purposes, it won't affect the locality. > > J-D > > On Mon, Jun 26, 2017 at 12:59 AM, Pavel Martynov > wrote: > >> Hi, guys! >> >> I working on replacing proprietary analytic platform Microsoft PDW (aka >> Microsoft APS) in my company with open source alternative. Currently, I >> experimenting with Mesos/Spark/Kudu stack and it looks promising. >> >> Recently I discovered very strange behavior. Situation: I have table on >> 5-servers cluster with 50 tablets and run simple Spark rdd.count() against >> it. If table has no replication - all is fine, every server run count >> aggregation on local data. But, if that table have replication > 1, I see >> (with iftop util) that Spark scans remote tablets and Spark UI still shows >> me tasks with locality NODE_LOCAL, what is not true. >> >> I found issue https://issues.apache.org/jira/browse/KUDU-1454 "Spark and >> MR jobs running without scan locality" which looks like my problem. >> >> IMHO Kudu-Spark can't be considered as production-ready with such an >> issue. Are there fundamental problems with fixing of that issue? >> >> -- >> with best regards, Pavel Martynov >> > > --f403043651a6f9b8f90552df4bca Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Mon, Jun 26, 2017 at 8:53 AM, Jean-Daniel Cryans &= lt;jdcryans@apache= .org> wrote:
Hi Pavel,

I think the whole Kudu/S= park story needs more attention, for example Spark SQL query plans don'= t have access to any Kudu stats so you can end up with some really bad join= decisions.

It feels like KUDU-1454 should be real= ly easy to solve at this point. What we need is to get the RDD to use CLOSE= ST_REPLICA and to set a propagated timestamp like Todd says in the jira. Th= is is all stuff that's done in Impala's integration for Kudu. If yo= u wanted to see if that solves your problem you could add the following cod= e on this line=C2=A0http://github.mtv.cloudera.com/CDH/k= udu/blob/cdh5-trunk/java/kudu-client/src/main/java/org/apache/kud= u/client/KuduScanToken.java#L226

=
=C2=A0

builder.replicaSelection(ReplicaSelection.CLOSEST_REPLICA);

The= propagated timestamp part is also needed but only for consistency purposes= , it won't affect the locality.

J-D
On Mon, Jun 26, 2017 at 12:59 AM, Pavel Martyno= v <mr.xkurt@gmail.com> wrote:
Hi, guys!

I wor= king on replacing proprietary analytic platform Microsoft PDW (aka Microsof= t APS) in my company with open source alternative. Currently, I experimenti= ng with Mesos/Spark/Kudu stack and it looks promising.

=
Recently I discovered very strange=C2=A0behavior. Situation: I have ta= ble on 5-servers cluster with 50 tablets and run simple Spark rdd.count() a= gainst it. If table has no replication - all is fine, every server run coun= t aggregation on local data. But, if that table have=C2=A0replication > = 1, I see (with iftop=C2=A0util) that Spark scans remote tablets and Spark U= I still shows me tasks with locality NODE_LOCAL, what is not true.

I found issue=C2=A0https://issues.apache.org/jira/= browse/KUDU-1454 "Spark and MR jobs running without scan locality&= quot; which looks like my problem.

IMHO Kudu-Spark= can't be considered as production-ready with such an issue. Are there = fundamental problems with fixing of that issue?

--
with best regards, Pavel Martynov


--f403043651a6f9b8f90552df4bca--