From dev-return-16228-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Sat Jan 13 01:44:21 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 7803F180621 for ; Sat, 13 Jan 2018 01:44:21 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 68220160C42; Sat, 13 Jan 2018 00:44:21 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 84C5F160C20 for ; Sat, 13 Jan 2018 01:44:20 +0100 (CET) Received: (qmail 83309 invoked by uid 500); 13 Jan 2018 00:44:19 -0000 Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list dev@nifi.apache.org Received: (qmail 83297 invoked by uid 99); 13 Jan 2018 00:44:19 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Jan 2018 00:44:19 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id CC8EF180985 for ; Sat, 13 Jan 2018 00:44:18 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.129 X-Spam-Level: ** X-Spam-Status: No, score=2.129 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id X5QPY_L4mTrJ for ; Sat, 13 Jan 2018 00:44:16 +0000 (UTC) Received: from mail-vk0-f52.google.com (mail-vk0-f52.google.com [209.85.213.52]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 8AEC55F39D for ; Sat, 13 Jan 2018 00:44:16 +0000 (UTC) Received: by mail-vk0-f52.google.com with SMTP id v70so4625143vkd.8 for ; Fri, 12 Jan 2018 16:44:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=drwqPukNaF5qnNzAwERaprWpvN6OS+UumWdp84/Oyos=; b=EgkYXYt7RGNPd1ohnRJbd9D6dTfHbnmV6AamlzqrtiYVKn9LtwyA5ILAIPlAFsYqP3 zkDmf/CdkZrWPDmDJluPa3lCl2uHFN20awxLgzLGR5WLrZnYYpjA3PhtRNyb3zQXPY/6 yd3NBOg+SGHDr5dpTUaMPOLxQ9fnx/1AIdp9zlcaoGtFLOEImrBBVzeYyLTZr548xsG8 LNxCiw9AOZCpVCdAgIOu8YBU1jTOZyGxiFDR5/kAYeU2Xkb67SiFOneLJ/wqCbg3FyVP ZLL+38nSvQOSEwfNWpKCdKE+8eD09WOgzwje4PkgzYTV4jdFT7qByP3t3fJvuoC4khOW fiLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=drwqPukNaF5qnNzAwERaprWpvN6OS+UumWdp84/Oyos=; b=EHGknyyQUnc1q9ds2RhqAKkGE//WwR/q/E5mRbLlwL5cxedXHIlsqe+24prT9ao6lT T1uvLEFTMHwAhYOWF1XdVND7ZQJPUMOFoei/GZ1ii1xTJN0dNjRuAgQbMLaChf6krPIk 4k6MwTotb7fNwpvwpXGtr/znbAkjQLgrtC3YeXBasMFalzA8hjjk0g/mvdUw1g+aXQ/x wtM7RceEagKCz8YrdwIDQfTywzeONK7CKwFIL6hjN6mOstIM5rUgxT3X3MNcc/0mL2Ko dbYh3sZJ1JuuHoz9b0RtqcrnDcafwTYTG6FfuqqN9+V2J6qsgwhpFDQTPwrcQ7+AaDJn jn0g== X-Gm-Message-State: AKwxytdg4uDYz8XBCiKEweqcq2H/VMv7XKQR0ea6o3DAuKQKhzWkQi6M G8ZIrFacu3LHopAr3Haqp05otwyty/Uy2nYZwbM= X-Google-Smtp-Source: ACJfBoudqNHkO/ZlI0Y75sVpexcv6O3gruGVCNjFanM3KfsZwgdNBNUCXhX6ULoLPn6fd3gcFMCGxdd5DpQfhSIqRqY= X-Received: by 10.31.86.65 with SMTP id k62mr25836231vkb.168.1515804256027; Fri, 12 Jan 2018 16:44:16 -0800 (PST) MIME-Version: 1.0 Received: by 10.159.43.202 with HTTP; Fri, 12 Jan 2018 16:43:35 -0800 (PST) In-Reply-To: <307678c3-7acc-e43f-55d9-0763a41d4396@windofkeltia.com> References: <307678c3-7acc-e43f-55d9-0763a41d4396@windofkeltia.com> From: =?UTF-8?B?5bC55paH5omN?= Date: Sat, 13 Jan 2018 08:43:35 +0800 Message-ID: Subject: Re: clear all flowfiles in all queues upon NiFi restart To: dev@nifi.apache.org Content-Type: multipart/alternative; boundary="001a114e58303ac7bf05629daed9" --001a114e58303ac7bf05629daed9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks Mark, Andrew and Russell, I think using the Volatile-implementations repositories mentioned by Mark should be sufficient for me. Regards, Ben 2018-01-12 22:59 GMT+08:00 Russell Bateman : > Andrew just meant that if you smoke the contents of all the repository > subdirectories under ${NIFI_ROOT}, it will result in what you seem to be > asking for. > > Hope this helps. > > > On 01/11/2018 09:48 PM, =E5=B0=B9=E6=96=87=E6=89=8D wrote: > >> Hi Andrew, sorry I didn't follow your idea, could you please elaborate >> with >> more details? >> What I want to do is to be able to clear all the FlowFiles when NiFi die= s >> unexpectedly and restarts itself. >> >> Regards, >> Ben >> >> 2018-01-12 12:44 GMT+08:00 Andrew Grande : >> >> Perhaps you could delete the repository directories when you need to >>> restart with no data? >>> >>> On Thu, Jan 11, 2018, 9:16 PM =E5=B0=B9=E6=96=87=E6=89=8D wrote: >>> >>> Hi Mark, forgot to ask about VolatileFlowFileRepository you mentioned, >>>> >>> if I >>> >>>> switch to use VolatileFlowFileRepository, will NiFi swap out all the >>>> >>> other >>> >>>> FlowFiles to disk if a queue is already full? >>>> Is it just simply keeping all FlowFiles in memory? >>>> >>>> Regards, >>>> Ben >>>> >>>> 2018-01-12 12:07 GMT+08:00 =E5=B0=B9=E6=96=87=E6=89=8D : >>>> >>>> Thanks Mark, my case is that I'm using NiFi to do some ETL work and >>>>> >>>> it's >>> >>>> possible that NiFi dies unexpectedly due to lack of system resources. >>>>> >>>> After >>>> >>>>> NiFi restarts itself, >>>>> I will re-extract all the data from database and re-perform all the >>>>> operations, so I need to clear all possible FlowFiles that might exis= t >>>>> >>>> in >>> >>>> any queue. >>>>> >>>>> Regards, >>>>> Ben >>>>> >>>>> 2018-01-12 11:49 GMT+08:00 Mark Payne : >>>>> >>>>> Ben, >>>>>> >>>>>> I have to admit - that=E2=80=99s kind of an odd request :) I=E2=80= =99m curious what >>>>>> >>>>> the >>> >>>> use case is, if you can share? >>>>>> >>>>>> Regardless, the easiest way would be to update nifi.properties so th= at >>>>>> the FlowFile repo that is used is the VolatileFlowFileRepository. Th= is >>>>>> would avoid writing the FlowFile state to disk, so ok restart you wi= ll >>>>>> >>>>> lose >>>> >>>>> all FlowFiles. The content will still be present, but nifi will delet= e >>>>>> >>>>> it >>>> >>>>> all on startup because there is no FlowFile associated with it. >>>>>> >>>>>> I=E2=80=99m on my phone right now so can=E2=80=99t easily tell you t= he exact name of >>>>>> >>>>> the >>> >>>> property to change but you=E2=80=99ll probably find it pretty quickly.= The >>>>>> >>>>> Admin >>> >>>> Guide may well explain the different repositories as well. >>>>>> >>>>>> Thanks >>>>>> -Mark >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>> On Jan 11, 2018, at 10:31 PM, =E5=B0=B9=E6=96=87=E6=89=8D wrote: >>>>>>> >>>>>>> Hi guys, I'm trying to clear all FlowFIles in all queues when NiFi >>>>>>> >>>>>> is >>> >>>> restarted, but I don't know the correct way to do this. I checked >>>>>>> >>>>>> all >>> >>>> NiFi's guide documentation, >>>>>>> it seems there're 2 possible solutions: >>>>>>> 1. write a custom notification service: a notification service coul= d >>>>>>> >>>>>> be >>>> >>>>> notified when NiFi is restarted and then inside the service, delete >>>>>>> >>>>>> all >>>> >>>>> the >>>>>> >>>>>>> files inside content_repository, flowfile_repository and >>>>>>> provenance_repository. >>>>>>> I know there're now 2 existing services: email and http. But I'm >>>>>>> >>>>>> not >>> >>>> quite sure how to correctly write one and deploy it into my NiFi >>>>>>> environment, is there a tutorial on writing one notification >>>>>>> >>>>>> service? >>> >>>> 2. I know from the developer guide that by using the annotation >>>>>>> >>>>>> @Shutdown >>>>>> >>>>>>> in a custom processor, the method could be called when NiFi is >>>>>>> >>>>>> successfully >>>>>> >>>>>>> shut down. The problem with this approach is the method could >>>>>>> not be guaranteed to be called when NiFi dies unexpectedly. >>>>>>> >>>>>>> Does anyone know what is the correct way to implement it? Thanks. >>>>>>> >>>>>>> Regards, >>>>>>> Ben >>>>>>> >>>>>> >>>>> > --001a114e58303ac7bf05629daed9--