From dev-return-18811-archive-asf-public=cust-asf.ponee.io@manifoldcf.apache.org Mon Dec 3 17:51:28 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 8B529180645 for ; Mon, 3 Dec 2018 17:51:27 +0100 (CET) Received: (qmail 78532 invoked by uid 500); 3 Dec 2018 16:51:26 -0000 Mailing-List: contact dev-help@manifoldcf.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@manifoldcf.apache.org Delivered-To: mailing list dev@manifoldcf.apache.org Received: (qmail 78504 invoked by uid 99); 3 Dec 2018 16:51:25 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Dec 2018 16:51:25 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 834D018BE52 for ; Mon, 3 Dec 2018 16:51:25 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.297 X-Spam-Level: *** X-Spam-Status: No, score=3.297 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=2, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id IGtt0Zt2-K5d for ; Mon, 3 Dec 2018 16:51:23 +0000 (UTC) Received: from mail-wr1-f68.google.com (mail-wr1-f68.google.com [209.85.221.68]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 7F25660E3B for ; Mon, 3 Dec 2018 16:51:22 +0000 (UTC) Received: by mail-wr1-f68.google.com with SMTP id x10so12929522wrs.8 for ; Mon, 03 Dec 2018 08:51:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=EPc/K7qHoXMH9EslynVOmk9aNpTOMR32AtMpga0SRjQ=; b=orosvKkFLR4L9HFMD+z2LS0MosZT1tH/0gh3ZfxKbvEaxDYSFbBo2lK8PSxg4csElX yo28wYY639G//9pQ8R3EX5+EwPiZ/UfZaCCoXdnXFe0bSowhCl9iejUViOPGexmmOcMf +DEEAnIw31nKQCWrgsEURdd8EADLI1XfFAvi6C2/6aF4Xm2a3mfc1+PZeeSYaHsR1Hxt 4Zuct8HPjBj3+E0TF7tLA5AZ8jeh5VboiPW1L2d/F81olDt2m6o7zfyyiLMV+jc6RRuv f3ivHuWGNESzfZgU+WlF28xISDyxnGuqynvb+VgQT3KvOQ092jtBm4zhN7p0N5VxcMKg VtkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=EPc/K7qHoXMH9EslynVOmk9aNpTOMR32AtMpga0SRjQ=; b=XFRvrrY1CA+Of6A5eME0i+N2TozzRlxa4DE7+NPPs73VX2A4HJsw1IOhveIbk7FptI 7jhcM8qrGeZ0ot8ZIKKAAGpsBVZjfOeXP7eUPH6r3iAcA4os/qkur2T8Si9v9b4mzLvQ vofCLWCXbr+M6KJCoMwoTvTtU919Gbq6LpFjVOc3BMuSTVM6ahZtgLAH2CY9JPje/z/H BOsYoVtExiGGPZ4N8Ij9iOAlG7Kruil8uPdILJ58LNO08L8qFAYJcJeBTf2uy34DLYaM yJ0UgzqNJwpCMpPOVZCQbCJTgpcroRrx4JsH1aX1c44KrZU9h9rgy7SU+rs/ShwaIEw1 j8QQ== X-Gm-Message-State: AA+aEWb2mFoKj6jlsUvDUcpNMehh5ADmWm1zhEj6Mbv8TYIhIFeawyrK HJCy+b3mw8ypo/vX1+GnsqFaLbgTmOi7JxuNqPJE5w== X-Google-Smtp-Source: AFSGD/UBI3oFBG1Vzfyzh6Xxx/MuUYc9jCuTcEfd7dALNW1U80/qCuNI+nie77moiJPNYFnnGCLK3qLH71DyFGRGuaI= X-Received: by 2002:adf:b3c3:: with SMTP id x3mr14875593wrd.294.1543855881663; Mon, 03 Dec 2018 08:51:21 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Karl Wright Date: Mon, 3 Dec 2018 11:51:09 -0500 Message-ID: Subject: Re: Apache Manifold 2.10 To: dev Content-Type: multipart/alternative; boundary="000000000000693b9f057c20f544" --000000000000693b9f057c20f544 Content-Type: text/plain; charset="UTF-8" You can just change the setup provided you point to the same database. Thanks, Karl On Mon, Dec 3, 2018 at 9:57 AM krishna agrawal wrote: > thanks Karl, > > I have deployed in my local as simple example and in Dev and QA with the > recomendation of Dev Ops team we deployed as multiprocess file example we > had brief discussion about considering multiprocess-zk-example and at that > time we were unsure of multiprocess-zk-example. > > But i will check and let you know if we can change the set up now? > > One question do we need to do fresh install or can we upgrade to > multiprocess-zk-example? > > Thanks for anticipation. > > Thanks, > Krishna A > > On Sat, Dec 1, 2018 at 3:05 PM Karl Wright wrote: > > > Another thing: it's quite important to guarantee a working setup here, > > otherwise you're just wasting everyone's time. So, please base your > > installation on the multiprocess-zk-example. Start off by running the > > example as is, on a small test crawl. Once you know how it works, then > > move next to changing only what you have to -- namely, the database > > properties in the global properties file, to point to your MySQL > instance. > > Try that also on a small test case (crawl some files for instance), > before > > trying it on your large case. Every step of the way should work, and if > it > > doesn't, figure out why not before you move onto the next step. > > > > Thanks, > > Karl > > > > > > On Sat, Dec 1, 2018 at 2:59 PM Karl Wright wrote: > > > > > Zookeeper does not require a locking directory. It is a process that > > > synchronizes other processes, and they connect to it by port. > > > > > > Karl > > > > > > > > > On Sat, Dec 1, 2018 at 2:55 PM krishna agrawal > > > wrote: > > > > > >> Thanks for the information. > > >> if we use Zookeeper how can we make sure all our ManifoldCF processes > > use > > >> same locking directory does it can be done at the configuration level > > >> while > > >> installing. > > >> > > >> thanks, > > >> Krishna A > > >> > > >> On Sat, Dec 1, 2018 at 1:39 PM Karl Wright > wrote: > > >> > > >> > That error is the result of the database not managing transactions > > >> > properly. It can occur if the locking system is not set up > properly, > > >> or if > > >> > you are using multiple agents processes and each process does not > have > > >> its > > >> > own ID. We have also seen it reported before just because MySQL > seems > > >> to > > >> > have bugs and sometimes writes are delayed or don't go through. > > >> > > > >> > My recommendation would be to: > > >> > (1) use zookeeper, not file locking > > >> > (2) Make sure all your ManifoldCF processes use the SAME locking > > >> directory > > >> > or Zookeeper instance > > >> > (3) If you are using multiple agents process, be certain that each > > such > > >> > process gets its own ID (as is done in the examples). > > >> > > > >> > Karl > > >> > > > >> > > > >> > On Sat, Dec 1, 2018 at 11:43 AM krishna agrawal < > krish.agwl@gmail.com > > > > > >> > wrote: > > >> > > > >> > > Thanks Karl, > > >> > > > > >> > > I will take a look at it > > >> > > > > >> > > But there is the error keep on tossing at manifold log > > >> > > > > >> > > ERROR 2018-12-01T11:13:26,297 (Job reset thread) - Exception > tossed: > > >> > > Unexpected job status encountered: 33 > > >> > > org.apache.manifoldcf.core.interfaces.ManifoldCFException: > > Unexpected > > >> job > > >> > > status encountered: 33 > > >> > > at > > >> > > > > >> > > org.apache.manifoldcf.crawler.jobs.Jobs.returnJobToActive(Jobs.java:2145) > > >> > > ~[mcf-pull-agent.jar:?] > > >> > > at > > >> > > > > >> > > > > >> > > > >> > > > org.apache.manifoldcf.crawler.jobs.JobManager.resetJobs(JobManager.java:8449) > > >> > > ~[mcf-pull-agent.jar:?] > > >> > > at > > >> > > > > >> > > > > >> > > > >> > > > org.apache.manifoldcf.crawler.system.JobResetThread.run(JobResetThread.java:77) > > >> > > [mcf-pull-agent.jar:?] > > >> > > > > >> > > Thanks, > > >> > > Krishna A > > >> > > > > >> > > > > >> > > On Fri, Nov 30, 2018 at 7:00 PM Karl Wright > > >> wrote: > > >> > > > > >> > > > Hi Krishna, > > >> > > > > > >> > > > First of all I suggest that you *not* use > > multiprocess-file-example, > > >> > and > > >> > > > instead use multiprocess-zk-example. > > >> > > > > > >> > > > Your symptoms suggest many possibilities. But if you move to > > >> Zookeeper > > >> > > we > > >> > > > will be able to eliminate dangling file locks as a complication. > > So > > >> > > please > > >> > > > do that first. > > >> > > > > > >> > > > Karl > > >> > > > > > >> > > > > > >> > > > On Fri, Nov 30, 2018 at 6:29 PM krishna agrawal < > > >> krish.agwl@gmail.com> > > >> > > > wrote: > > >> > > > > > >> > > > > Yeah in our local set up we did Simple example but in server > we > > >> did > > >> > > > > multiprocess-file-example are you suggesting us to upgrade > from > > >> 2.10 > > >> > to > > >> > > > > 2.11 ? > > >> > > > > > > >> > > > > and we are using MY Sql database , > > >> > > > > > > >> > > > > So most of time i saw nothing is running and still it say job > is > > >> > > running > > >> > > > > and you have to wait for it to complete. > > >> > > > > > > >> > > > > and restarting also not helping. > > >> > > > > > > >> > > > > Any other solution woould be greatly appreciated. > > >> > > > > > > >> > > > > Thanks, > > >> > > > > Krishna A > > >> > > > > > > >> > > > > On Fri, Nov 30, 2018 at 10:50 AM Karl Wright < > > daddywri@gmail.com> > > >> > > wrote: > > >> > > > > > > >> > > > > > It also may be useful to start with the simple example, > which > > is > > >> > not > > >> > > > > > multiprocess, and get familiar with using ManifoldCF that > way, > > >> > before > > >> > > > you > > >> > > > > > try to go to a more complicated setup. > > >> > > > > > > > >> > > > > > Thanks, > > >> > > > > > Karl > > >> > > > > > > > >> > > > > > > > >> > > > > > On Fri, Nov 30, 2018 at 9:46 AM Karl Wright < > > daddywri@gmail.com > > >> > > > >> > > > wrote: > > >> > > > > > > > >> > > > > > > "simplified multi-process"? There is no such example. > > >> > > > > > > > > >> > > > > > > These are the examples available. Which one are you > using? > > >> > > > > > > > > >> > > > > > > 11/15/2018 03:40 AM example > > >> > > > > > > 11/15/2018 03:40 AM example-proprietary > > >> > > > > > > 11/15/2018 03:40 AM > > >> multiprocess-file-example > > >> > > > > > > 11/15/2018 03:40 AM > > >> > > > > > > multiprocess-file-example-proprietary > > >> > > > > > > 11/15/2018 03:40 AM > > multiprocess-zk-example > > >> > > > > > > 11/15/2018 03:40 AM > > >> > > > > > multiprocess-zk-example-proprietary > > >> > > > > > > > > >> > > > > > > Cleaning locks makes no sense unless you are using the > > >> > > > > multiprocess-file > > >> > > > > > > setup. This is deprecated, by the way, in favor of the > > >> Zookeeper > > >> > > > > setup. > > >> > > > > > > > > >> > > > > > > As for the buttons, please read: > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > https://manifoldcf.apache.org/release/release-2.11/en_US/end-user-documentation.html#outputs > > >> > > > > > > > > >> > > > > > > The buttons in question are "Reindex all..." and "Remove > > >> all..." > > >> > > > > > > > > >> > > > > > > Karl > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > On Fri, Nov 30, 2018 at 9:36 AM krishna agrawal < > > >> > > > krish.agwl@gmail.com> > > >> > > > > > > wrote: > > >> > > > > > > > > >> > > > > > >> We have deployed the Manifold using > > >> > > > > > >> > > >> > > > > > >> - Simplified multi-process model > > >> > > > > > >> > > >> > > > > > >> We did try clean up of lock Sh but that also did not > work. > > >> > > > > > >> > > >> > > > > > >> I dont have forget all document button in output > connector. > > >> > > > > > >> > > >> > > > > > >> [image: image.png] > > >> > > > > > >> > > >> > > > > > >> On Thu, Nov 29, 2018 at 6:52 PM Karl Wright < > > >> daddywri@gmail.com > > >> > > > > >> > > > > wrote: > > >> > > > > > >> > > >> > > > > > >>> Hi Krishna, > > >> > > > > > >>> > > >> > > > > > >>> Please give us some background as to how you've deployed > > >> > > > ManifoldCF. > > >> > > > > > Are > > >> > > > > > >>> you using one of the examples? If so, which one? > > >> > > > > > >>> > > >> > > > > > >>> The detailed answer to your question is: the job must > > delete > > >> > all > > >> > > > > > >>> documents > > >> > > > > > >>> it indexed before it can be deleted. That is the > typical > > >> way > > >> > > jobs > > >> > > > > > work. > > >> > > > > > >>> Thus, if you shut down the target of your output > > connection, > > >> > you > > >> > > > may > > >> > > > > be > > >> > > > > > >>> blocked in deleting your job. > > >> > > > > > >>> > > >> > > > > > >>> At that point, you can either (a) restart the target of > > your > > >> > > output > > >> > > > > > >>> connection, or (b) go to the "view" page for the output > > >> > > connection > > >> > > > > and > > >> > > > > > >>> click both of the "forget all documents" buttons on it. > > >> (b) is > > >> > > not > > >> > > > > > >>> recommended unless you really want to start over fresh > on > > >> your > > >> > > > output > > >> > > > > > >>> index. > > >> > > > > > >>> > > >> > > > > > >>> Thanks, > > >> > > > > > >>> Karl > > >> > > > > > >>> > > >> > > > > > >>> > > >> > > > > > >>> On Thu, Nov 29, 2018 at 3:21 PM krishna agrawal < > > >> > > > > krish.agwl@gmail.com> > > >> > > > > > >>> wrote: > > >> > > > > > >>> > > >> > > > > > >>> > Hi We are facing issue of action button is not > available > > >> > > > > > >>> > > > >> > > > > > >>> > [image: image.png] > > >> > > > > > >>> > > > >> > > > > > >>> > I have stop the agent process but still i am not able > > to > > >> > > remove > > >> > > > > the > > >> > > > > > >>> job > > >> > > > > > >>> > it say it > > >> > > > > > >>> > > > >> > > > > > >>> > there should be some way to forcefully restart and > stop > > >> the > > >> > > > running > > >> > > > > > >>> > process ? > > >> > > > > > >>> > > > >> > > > > > >>> > Job 1542835910915 is busy; you must wait and/or shut > it > > >> down > > >> > > > before > > >> > > > > > >>> > deleting it > > >> > > > > > >>> > but there is no job running, and i am seeing this > > message > > >> > from > > >> > > > > past 3 > > >> > > > > > >>> days. > > >> > > > > > >>> > > > >> > > > > > >>> > is there any ways to clear this? > > >> > > > > > >>> > > > >> > > > > > >>> > > > >> > > > > > >>> > Any help in this matter will be appreciated. > > >> > > > > > >>> > > > >> > > > > > >>> > Thanks, > > >> > > > > > >>> > Krishna A > > >> > > > > > >>> > > > >> > > > > > >>> > > >> > > > > > >> > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > > --000000000000693b9f057c20f544--