From dev-return-18820-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Thu Feb 21 17:51:56 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id BFC0018064C for ; Thu, 21 Feb 2019 18:51:55 +0100 (CET) Received: (qmail 24016 invoked by uid 500); 21 Feb 2019 17:51:54 -0000 Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list dev@nifi.apache.org Received: (qmail 24002 invoked by uid 99); 21 Feb 2019 17:51:54 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Feb 2019 17:51:54 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id BD40DC286C for ; Thu, 21 Feb 2019 17:51:53 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.799 X-Spam-Level: * X-Spam-Status: No, score=1.799 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id dWKVAfCaJEHu for ; Thu, 21 Feb 2019 17:51:51 +0000 (UTC) Received: from mail-oi1-f181.google.com (mail-oi1-f181.google.com [209.85.167.181]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 83F6C6238B for ; Thu, 21 Feb 2019 17:51:50 +0000 (UTC) Received: by mail-oi1-f181.google.com with SMTP id w66so2102547oia.4 for ; Thu, 21 Feb 2019 09:51:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=2/Aa8ktII27EP3gbJiPE4AD/vIuYG84ePKcvnP7Twkw=; b=dzmOrSo8qd0IbMA9QYk7fB/MjvfjKa4X2o/0xLi0YZl40a6V9A8ayQT1mEPwGDnWim 7TqqtMB+j9RMNQrdr7iKmlOPN8JeFFuQJdAAqfJ2AlMoR7sbDOGBdevtQBEAvIza3THs bIHIs6b1lYN2Z4sEPfJrbWSPy7hQbZg0vPfCRyZEzMouyLn/EAgs+1nx6L/JBzM98BM8 H20n+3EFad9Qp5bpbaDIXirR8vCOsVcmlQ4Na/naK+Xr8tWdyWV1U7Vo3Z4kmcmpdUnk mGngZsuMuUY+jIZ3626J1ERqPQ0uefKWcu5FKB3NgEg9qVTSnOZbAYr0iXaJ9KOjg6Ef qdSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=2/Aa8ktII27EP3gbJiPE4AD/vIuYG84ePKcvnP7Twkw=; b=JCf9cjC2qn6V6GHZpl7PJ9rDbmia2+w94uyyNrSsDmBjRifZuz94QbguCWTzs5Ribp kEtCS4FnfgBpVVsasnjesPVUmxkaRCsruvOWbwJOmEDUJC30PpxVi5cWz1v5qsmQ4n9F b6sfy+gYHdTK87ZQ6WpX2eDIGZd2RDmiAtGotduXfSn2PauoxwJLnziBGobigjs5dnRs yEJlo7XqFxUQd4KhyNGoOJ3Yrwsj9tGYqSnUlGMnl2adrGQiDg3Ap+ybpJXW4LiGnp7z 0exKQMyAyKDnvAGoqRd3cpic3kxyXMBk8GM67Sz7ODpmW/7wG/mPks5elHpsLwI21v04 tSSw== X-Gm-Message-State: AHQUAuY02NK5vTXbI7lNuQMl5QWhyC7PoaU7HmvoursGwycA1u/Uk2n6 ubQ5yjwJbJ7cQ6sNfFH1FLMmCDwE+VEfGHZqOgAmiw== X-Google-Smtp-Source: AHgI3IYh1ZlEypt6mmRv420/UNne8qDlPHNBZIYTPJMPxA/iYW5UQNT+IL7yp5tJYG/f77BUVQxPhokac6Ssv5sUV9k= X-Received: by 2002:aca:f4d3:: with SMTP id s202mr10087948oih.178.1550771509134; Thu, 21 Feb 2019 09:51:49 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Mike Thomsen Date: Thu, 21 Feb 2019 12:51:37 -0500 Message-ID: Subject: Re: NiFi, Docker Environment Variable for enabling debugging of NiFi inside a Docker container To: dev@nifi.apache.org Content-Type: multipart/alternative; boundary="000000000000ee09aa05826b202c" --000000000000ee09aa05826b202c Content-Type: text/plain; charset="UTF-8" Erik, I think a lot of people would benefit from having both the JVM debugger option AND your debug logging. Go ahead and send us a PR for your debug logging and I'll do a ticket to add the JVM debugger. I just merged a change to the Docker setup that will make the JVM debugger much easier to enable. Thanks, Mike On Wed, Feb 20, 2019 at 9:48 AM Erik Anderson wrote: > Andrew, > > Sorry, was skiing. > > > A user only had to specify an > > additional -debug flag for the main shell script which would in turn take > > care of any bootstrap rewriting/generation for allowing for the remote > JVM > > debug session to connect. > > So you dont want logging, you want a remote debugger session? > > 1) I expect many environments wont let devs just connect a debugger into a > production network. In a prod network, we start with logs. > 2) What about NiFi clusters? Distributed debugging, wow, now that gives me > a headache. How would you connect a java debugger into a 3 node cluster? > 3) Maybe you want to trap/breakpoint a list of exceptions and stop the > java process to wait for a debugger to be attached? > > As our NiFi usage grows, the #1 most important feature is debug-ability of > an issue in any environment. > > Andrew, as you have a lot more experience with issues you have seen in the > past, how about a 30 minute call? This way can I figure out what debugging > means to everyone, what information they need, and the best way to > bootstrap it. > > Erik Anderson > Bloomberg > https://www.linkedin.com/in/erikanderson/ > > Sent from my iPad > > > On Feb 14, 2019, at 12:47 AM, Andrew Grande wrote: > > > > Here's what I did previously, Erik. A user only had to specify an > > additional -debug flag for the main shell script which would in turn take > > care of any bootstrap rewriting/generation for allowing for the remote > JVM > > debug session to connect. Maybe it could give a few ideas. > > > > Andrew > > > > On Wed, Feb 13, 2019, 7:34 AM Erik Anderson wrote: > > > >>> I was reading that email and was thinking of JVM debug options, with > >>> suspend y/n. I guess it just shows we meant very different things by > >> debug > >>> mode. Maybe you could incorporate those into a PR too? > >>> > >>> Andrew > >> > >> Good point Andrew. and sorry for the slow response. I had to look at how > >> NiFi set the JVM properties. > >> > >> We had issues with Java and needed to manually set env Java flags. > >> > >> It seems all of the Java debugging flags are set in the bootstrap.conf, > >> located here. > >> > >> > >> > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/bootstrap.conf > >> > >> For me, I chained together the Dockerfile > >> ENTRYPOINT ["my_company_entry_point.sh"] > >> then had it call the original entry point > >> In short, all of NiFi properties are set via environment variables in a > >> /etc/profile.d/nifi.sh on the host and propogate throughout the NiFi > >> system. Like I listed in > >> > >> > https://github.com/dprophet/nifi/blob/master/nifi-docker/dockerhub/CONFIGURATION.md > >> > >> I normalized all NIFI specific environment variables to start with NIFI_ > >> > >> Why? Because you set the > >> > >> export NIFI_FOO="my foo" > >> > >> and its directly passed into the container and into the NiFi role > account > >> used to start the container. You now have access to the environment > >> variables throughout the NiFi system. > >> > >> Example: > >> docker run --name nifi --env-file <(env | grep NIFI_) --hostname nifi > >> > >> IMO, the Dockerfile entry point should allow a plug and play script so > you > >> can set these custom behaviors (both enterprise behaviors and custom > >> developer/debugging). I doubt any enterprise will blindly pull a > DockerHub > >> container and run it. From my experience, a public container isnt > >> enterprise friendly. > >> > >> Andrew, define what you want for JVM debugging, what you would want to > set > >> (and unset), and I will take a look. > >> > >> Erik Anderson > >> Bloomberg > >> https://www.linkedin.com/in/erikanderson/ > >> > --000000000000ee09aa05826b202c--