From dev-return-493-archive-asf-public=cust-asf.ponee.io@zipkin.apache.org Tue Mar 19 06:43:49 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id F1E6D180791 for ; Tue, 19 Mar 2019 07:43:48 +0100 (CET) Received: (qmail 40906 invoked by uid 500); 19 Mar 2019 06:43:48 -0000 Mailing-List: contact dev-help@zipkin.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@zipkin.apache.org Delivered-To: mailing list dev@zipkin.apache.org Received: (qmail 40745 invoked by uid 99); 19 Mar 2019 06:43:47 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Mar 2019 06:43:47 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id C8331C588D for ; Tue, 19 Mar 2019 06:43:46 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.597 X-Spam-Level: X-Spam-Status: No, score=0.597 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 0GbgdCeLDFff for ; Tue, 19 Mar 2019 06:43:45 +0000 (UTC) Received: from mail-wr1-f68.google.com (mail-wr1-f68.google.com [209.85.221.68]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 25EA45F3D0 for ; Tue, 19 Mar 2019 06:35:51 +0000 (UTC) Received: by mail-wr1-f68.google.com with SMTP id j9so5477187wrn.6 for ; Mon, 18 Mar 2019 23:35:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :content-transfer-encoding; bh=5lVcHdbS4LbmZsMjyDa6u+rwlKBrekj7MFcRQRvwFX0=; b=KF2vxn7YMHnz705LIyc5q8zj+xtvQRLgKXlkpDsL4Q7Q4WZOsscLch51ciN+WWoBoW pLzJudxVoWiK6HhUD3Xvbu4SxSM9y34hsL0oJkUY4WOi8fEVWYpE/cAbBf5SWsxTx3D4 bCqCeTW5GiN8GfWjmyiikeac+e4kzVaMdqAvVy60OV3q5LrdNkSwPscFfOd/nGR0PcrQ DCKU1GUFUtNvB//bOBYjw7+pukX49DFK4FJ9phmG8zibHpaxhZSDjNPdrIPSZiTNvSmp 8LuX56r51lfwYUL3wxN5a82YAwTfHhE1rsjxBlzOhJ/1TwPjfrzl2Noge8mOmfeQ2G7x qjBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:content-transfer-encoding; bh=5lVcHdbS4LbmZsMjyDa6u+rwlKBrekj7MFcRQRvwFX0=; b=t1D97cepWBp25qt/Ew5bfvCaKKiC2BsywmKOIMUXIUOILhaS8zz+2O8EgRTfFirWm5 NWmmu1Ska1EwNxkR/ZE3eSqwCMmJ4LVcnwrLiMGQ+EAQFSfeSeSwPcoUBzq3BHu8humt 6faHsfe8v1cI4e/Xo/SJ75qbrUnos6Q8zbVJaNdVzTUWre4YKQus7i2HzPk6goWFOJSM /IbazANWetjExInqCe5v2TCeormos3v0nDPefdQmIbw2mA4DoPhKsUAFhJ2Y/DDmHyDI cR3u5seSNWTf5oDjtCp5NQKVrDj+S/46A2FPR9T70Xgx2L+jEs9C1xcc1TKi6BVtYoZH ZwiQ== X-Gm-Message-State: APjAAAVB4c3M0OKc1DPmCWR1B8pkyrFPPF36TM9C7gPByAg8SlQbcpTx Qr5Xcij+5eSdirvpq9faZGEcEFYS4AKqewI/bcIYxoHWf8c= X-Google-Smtp-Source: APXvYqxucuqqJP3y8UIKCFXjGVwd995yr8VtGjORWscap4XtI18/FAsl0RsBP3jNAUfak4tjM1YlqSWgyr725ekZyg0= X-Received: by 2002:adf:f611:: with SMTP id t17mr14899604wrp.248.1552977349810; Mon, 18 Mar 2019 23:35:49 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Willem Jiang Date: Tue, 19 Mar 2019 14:35:38 +0800 Message-ID: Subject: Re: Perpetual support problems using Spark for dependency link aggregation To: dev@zipkin.apache.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable My original through is we need to fix the code issue of the spark job, but it turns out it's the support issue after reading about the README change you made. It's make sense that we are not the expert of the troubleshooting of spark cluster or the elasticsearch cluster. But we could provide the information that we know to help the user. I guess we can provide a FAQ to help them to the basic troubleshooting or just point to the right community to ask question. Willem Jiang Twitter: willemjiang Weibo: =E5=A7=9C=E5=AE=81willem On Tue, Mar 19, 2019 at 11:55 AM Adrian Cole wrot= e: > > Hi, Willem. > > Considering most people struggle with running spark or don't have idea > why jobs don't distribute... (second problems being the nature of how > elasticsearch always dropping support for things).. > > What exactly are you thinking to be put into a wiki or blog? We > actually have no answers for spark... so usually a wiki is where we'd > put some answers or such, although we tend to use README (and we've > put the little we know there). I don't quite understand the content > you are suggesting us write even if I agreed that it is likely to > solve the problem. > > -A > > On Tue, Mar 19, 2019 at 11:34 AM Willem Jiang wr= ote: > > > > Hi Adrian, > > > > Thanks for the briefing of the support problem of Sparks. > > We could always ask help from the community by providing enough > > context information. > > Maybe we can add a page in the zipkin wiki page and write twitter > > about it, to see if we could attract some contributors. > > From my experience, if we have a great idea, it won't take a long > > time to find help from the open source community. > > > > Regards, > > > > Willem Jiang > > > > Twitter: willemjiang > > Weibo: =E5=A7=9C=E5=AE=81willem > > > > On Tue, Mar 19, 2019 at 8:51 AM Adrian Cole w= rote: > > > > > > Hi, team. > > > > > > A long time ago, we arbitrarily used spark for dependency link > > > aggregation (porting the work from Eirik's hadoop job). The initial > > > spark job was created incomplete then abandoned by the author. I've > > > tried a lot to support it, but it has been perpetual maintenance and > > > most of us have no idea how to support it. Yet, we get a lot of user > > > questions about it and the support load is higher than most of our > > > projects. > > > > > > The Elasticsearch part is landmines from the "wan only" stuff, to the= m > > > having a narrow supported range of versions. It is rev-locked to a JR= E > > > (even if will change later). We've had users complain about CVE > > > maintenance and actively ask for a non-spark option. General support > > > comes in questions about cluster distribution which no-one knows the > > > answer to. I've recently in desperation added a change to help show > > > where Spark support is. > > > > > > https://github.com/openzipkin/zipkin-dependencies/pull/133 > > > > > > All this said, despite the problems running distributed or with > > > elasticsearch, most can start the zipkin-dependencies job as a > > > one-shot cron job without much help. > > > > > > I think we have to be honest about the fact that since this project > > > started, we've rarely had anyone able to support it. I hope we can ge= t > > > out of the mutually disappointing support swamp. Does anyone have any > > > ideas? > > > > > > I would like to think someone could come in and save us, but seems we > > > should also consider other tools as that usually doesn't happen, and > > > one person saving us isn't sustainable (usually we need a few people > > > to know a tool in order to realistically support it). It is possible > > > to recruit for this, but we need significant close buy-in from people > > > who know spark imho, like actually helping with support, if we want t= o > > > continue this path. > > > > > > I know there's a Kafka streaming option [1]. I also know some have > > > used Flink, and some have had interest in Pulsar. I think we should > > > have streaming options, but fact is many don't use any buffer like > > > Kafka (direct http), which leads me to think we still need an > > > after-the-fact option (pull from storage). Moreover spark's embedded > > > mode is nice as it can be treated as a dumb cron job. > > > > > > Looking for ideas, > > > -A > > > > > > [1] https://github.com/sysco-middleware/zipkin-dependencies-streaming > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscribe@zipkin.apache.org > > > For additional commands, e-mail: dev-help@zipkin.apache.org > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscribe@zipkin.apache.org > > For additional commands, e-mail: dev-help@zipkin.apache.org > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscribe@zipkin.apache.org > For additional commands, e-mail: dev-help@zipkin.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@zipkin.apache.org For additional commands, e-mail: dev-help@zipkin.apache.org