From dev-return-940-archive-asf-public=cust-asf.ponee.io@fluo.apache.org Sun Dec 15 16:17:13 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id D1B52180636 for ; Sun, 15 Dec 2019 17:17:12 +0100 (CET) Received: (qmail 49649 invoked by uid 500); 15 Dec 2019 16:17:12 -0000 Mailing-List: contact dev-help@fluo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@fluo.apache.org Delivered-To: mailing list dev@fluo.apache.org Received: (qmail 49637 invoked by uid 99); 15 Dec 2019 16:17:11 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Dec 2019 16:17:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 1853A1A325C for ; Sun, 15 Dec 2019 16:17:11 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 0XMEF3U4mBYQ for ; Sun, 15 Dec 2019 16:17:09 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::835; helo=mail-qt1-x835.google.com; envelope-from=mjwall@gmail.com; receiver= Received: from mail-qt1-x835.google.com (mail-qt1-x835.google.com [IPv6:2607:f8b0:4864:20::835]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id DCE237DDD3 for ; Sun, 15 Dec 2019 16:17:08 +0000 (UTC) Received: by mail-qt1-x835.google.com with SMTP id g17so2435725qtp.11 for ; Sun, 15 Dec 2019 08:17:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Zt9+tG085v8cXrtXOL2Tw7i+ks16xxGkxvzN2IWyqeo=; b=Buqld9oENcynULhiSEK3uj+in3gjAT+0iKSrwjzK/qOzaYt0DwCUTJyGMbuifuxp/Y GNVbM31iCrCT3JcnX586waYB1PL5GicZLhI/xeiqM1KGRmS+QwTyxZJ2lIy/Rkx5J0ZS y55vAuEHj8Q9JgFFmAlGoHOuP2PkGdPL82jTP+qC8ZSI04YXBuT2Qr7FdF6fm8rKIjhB HsmQtv0pkL3xQJIEDrYKNsNURSevPwnaBufDlkhVycDBmy6Mgn/TNXDYvvDQPKnDdAQs KKzM3cg0HWoB8WwcobGjAUdP2SGAS538+Yp+vjBR+oy5oGpU0yCVqgPbc1ILNUCGSsA4 rXwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Zt9+tG085v8cXrtXOL2Tw7i+ks16xxGkxvzN2IWyqeo=; b=ReL0YQO8BgQ2IiI4TwmvUsaMTzvluG4vN1qRHAWWiZlmcmof8CzDX7tpN+zu/UG7ID yTy7SrrJOlJdtejpTEz6R03sBPgrB34bdty+tvTGuJPQLUI8ApE4UAdVLC2+CCMm/EHX +mfqHhAfsw94ev6glAWvC9zPuAwCfG2bnnjS9EQFk+FSI1c9c6Xo4HrRbDgQV75Wx/h+ 43TfTwnwDsGmy1ElpErFwE04yrKyHZMomQ61bCVKwePRRc3rnhMOuskkv80uJTWVx0MK FzV9RcZg77nZBcUAQA+QzK0RfJfu7Tut4G0wixDJacN7xQhySu4A7jMMjYCVjN2ImoLR Z/5Q== X-Gm-Message-State: APjAAAUU/OYo2ghGqbi+EF84YbV+sjiecKBqyJMDR7ycabvGT3/PLWsP NvukP6nLKGCu7VHXTJqjKwK/QxZXVHws1NVo5xnU X-Google-Smtp-Source: APXvYqxmT+H3G1k7mD8sQYADH3TfD8Zhr9bqTUtRVwc90Hx3zWznOh9JVs9QM8Bb0c7akXyjwVzOPPsgQtYfx1ld7oM= X-Received: by 2002:ac8:602:: with SMTP id d2mr21310502qth.245.1576426627183; Sun, 15 Dec 2019 08:17:07 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Michael Wall Date: Sun, 15 Dec 2019 11:16:56 -0500 Message-ID: Subject: Re: Run Accumulo and Hadoop services under systemd To: dev@fluo.apache.org, Aishwarya.Thangappa@microsoft.com Cc: Arvind Shyamsundar , Billie Rinaldi Content-Type: multipart/alternative; boundary="00000000000020b7aa0599c06d27" --00000000000020b7aa0599c06d27 Content-Type: text/plain; charset="UTF-8" Hi Aishwarya, I didn't get any attachments on this. Thanks Mike On Fri, Dec 13, 2019 at 5:46 PM Aishwarya Thangappa wrote: > Hello everyone, > > I have not subscribed to the dev mailing list earlier and missed on some > of your questions. I will address them here. > > @Christopher > Most of the changes except the actual installation of the systemd units > could possibly go into Accumulo. These would be the systemd units for > various accumulo services, modification to cluster-wide scripts in accumulo > to use systemd instead of directly starting/stopping the processes. We > would be happy to accommodate/answer any suggestions or follow-up questions > you may have. > > Attached the accumulo_cluster and accumulo_service scripts with systemd > changes. > > > @Keith Turner > Once we determine where the different pieces should land, I can post PRs > accordingly. In our current setup, in muchos.properties file I have added a > `use_systemd` flag which when set to true, will overwrite the accumulo > cluster-wide scripts in the nodes with the attached ones. These files > currently reside at ansible/roles/accumulo/files. If we determine that > these scripts and the systemd unit files will instead go to Accumulo > project, I will have to make changes accordingly. > > @Michael Wall > Systemd units internally call the same scripts that accumulo_cluster > commands currently use. The change is that accumulo_cluster commands would > call systemd start/stop which inturn would call accumulo_service commands. > Attached a sample systemd_unit template. Can you please elaborate if this > is still an issue? > > ------------------------------ > *From:* Aishwarya Thangappa > *Sent:* Thursday, December 12, 2019 11:25 AM > *To:* dev@fluo.apache.org > *Cc:* Arvind Shyamsundar ; Billie Rinaldi < > Billie.Rinaldi@microsoft.com> > *Subject:* Run Accumulo and Hadoop services under systemd > > Hi everyone, > > While using fluo-muchos to deploy an Accumulo cluster, we recognized the > need for various Accumulo and Hadoop services to be run under a service > manager like systemd which will ensure that all these services are brought > up correctly in the event of VM / OS reboots / cold starts. We have made > the required changes for this and would like to contribute it back to the > community if there is any interest around it. > > Summarizing what we have done: > > - Crafted separate systemd unit files for Accumulo (master, monitor, > gc, traser, tserver), Hadoop (journalnode, namenode, datanode, > resourcemanager, nodemanager, zkfc) and Zookeeper services. > - All of these unit files will be copied to the respective nodes' > /etc/systemd/system directory; the services will then be started and > enabled by ansible systemd module. > - In case of num_tservers > 1, multiple tserver systemd units will be > copied to the node and each will be independently managed. > - Also made necessary changes to the existing cluster-wide scripts > including accumulo_cluster, accumulo_service, start_dfs, start_yarn etc., > to have them work seamlessly with sytemd. > > Is there an appetite to look at the details? If so, we can post a PR or if > there are any feedbacks and other considerations, please let us know and we > can discuss them. > > Thanks, > Aishwarya > > --00000000000020b7aa0599c06d27--