From dev-return-18803-archive-asf-public=cust-asf.ponee.io@manifoldcf.apache.org Sat Dec 1 20:55:31 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 2A36B18064E for ; Sat, 1 Dec 2018 20:55:31 +0100 (CET) Received: (qmail 75793 invoked by uid 500); 1 Dec 2018 19:55:30 -0000 Mailing-List: contact dev-help@manifoldcf.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@manifoldcf.apache.org Delivered-To: mailing list dev@manifoldcf.apache.org Received: (qmail 75777 invoked by uid 99); 1 Dec 2018 19:55:29 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Dec 2018 19:55:29 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 354111869D9 for ; Sat, 1 Dec 2018 19:55:29 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.297 X-Spam-Level: *** X-Spam-Status: No, score=3.297 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=2, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id hHGyqei7R2Y7 for ; Sat, 1 Dec 2018 19:55:27 +0000 (UTC) Received: from mail-ed1-f65.google.com (mail-ed1-f65.google.com [209.85.208.65]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 09CFF5FACC for ; Sat, 1 Dec 2018 19:55:27 +0000 (UTC) Received: by mail-ed1-f65.google.com with SMTP id d39so7522554edb.12 for ; Sat, 01 Dec 2018 11:55:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=uJhoKANLVHHlEfdiL5lZm2JGUhOoLgKinaP4tyMwxgQ=; b=PRajcg/Ub7euiCtx8lPbEpCVXGOKcNhkZgWMD6UomUmWicZ3Hcu08pYN+Uoj7yhKkG QxuYbSnadah+ltkjDVVLxi0MYexmGTTfPqlPpC7CDvsyrOzj/4yM7FljfoMbTshnSmNT THI6KPtdwmFw0NvVzcIhitPt5vYR3SMnc0DVEnhLHPSw35Oj2yA08NZ3671Tj6+5w5nG OvkNO2N5Gzz9rP8Ki9l7FDG4b/GWqXIvt6oBtnMgHLaz68KdvffpvMwQU2g8iTAnznSi e/zyAR58W+mHg2lFFb70jLltr5wPXPbCkRRKIekq35HunpbQj3oN4zS9VS8KzCWLm15j wShQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=uJhoKANLVHHlEfdiL5lZm2JGUhOoLgKinaP4tyMwxgQ=; b=r0YePOBgHVJY3UZdOvKEmN6iQPgjy3ygnu2D1BBkHlNkjMIAVvTv21AHlsmDMYN1Di 8Jjmo1zhndk5r+Ra63nh2vQD85tEl8Cb2T4dSfZoHLEyuILQoUmz4l4G1ljHsd2aYYSY KVvwrJV4CNwoqkgDjdT70IBcPcUPeMKPQ0aOybT3t4B3mW1+gdmd38XkqSWCSowGgOTx EU4YjnLyQjkulgJsSTLw1RwAX8IneClSVjNQX3135hWWUZZ3MDqmmU/JSMK8Dx7P1ynt th74lgk26UiP1CdzrFWPm53IY38s8yT0dEpmuXRCd+J2d30hJNTmYw3qsJOQx2sXu/oT QqCw== X-Gm-Message-State: AA+aEWbu11+LQF+kvwem76sakI5eksVCqecYEW5NYESMuYtgZ8AcDN8I 8i4mOB9Tg2iR/h8XMOn1iA/yl3bAGLC92z+/LL6r9w== X-Google-Smtp-Source: AFSGD/WHJesy5f6NnVVEp/pLY/a1Rze3lP6RpaoczypFyQjfCIVbrNe56NmPM+JklO5cv7AXbUdvFWWVowXLX+bwhrQ= X-Received: by 2002:a17:906:849:: with SMTP id f9-v6mr8414901ejd.49.1543694125606; Sat, 01 Dec 2018 11:55:25 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: krishna agrawal Date: Sat, 1 Dec 2018 14:55:14 -0500 Message-ID: Subject: Re: Apache Manifold 2.10 To: dev@manifoldcf.apache.org Content-Type: multipart/alternative; boundary="000000000000ffaa6d057bfb4b3d" --000000000000ffaa6d057bfb4b3d Content-Type: text/plain; charset="UTF-8" Thanks for the information. if we use Zookeeper how can we make sure all our ManifoldCF processes use same locking directory does it can be done at the configuration level while installing. thanks, Krishna A On Sat, Dec 1, 2018 at 1:39 PM Karl Wright wrote: > That error is the result of the database not managing transactions > properly. It can occur if the locking system is not set up properly, or if > you are using multiple agents processes and each process does not have its > own ID. We have also seen it reported before just because MySQL seems to > have bugs and sometimes writes are delayed or don't go through. > > My recommendation would be to: > (1) use zookeeper, not file locking > (2) Make sure all your ManifoldCF processes use the SAME locking directory > or Zookeeper instance > (3) If you are using multiple agents process, be certain that each such > process gets its own ID (as is done in the examples). > > Karl > > > On Sat, Dec 1, 2018 at 11:43 AM krishna agrawal > wrote: > > > Thanks Karl, > > > > I will take a look at it > > > > But there is the error keep on tossing at manifold log > > > > ERROR 2018-12-01T11:13:26,297 (Job reset thread) - Exception tossed: > > Unexpected job status encountered: 33 > > org.apache.manifoldcf.core.interfaces.ManifoldCFException: Unexpected job > > status encountered: 33 > > at > > org.apache.manifoldcf.crawler.jobs.Jobs.returnJobToActive(Jobs.java:2145) > > ~[mcf-pull-agent.jar:?] > > at > > > > > org.apache.manifoldcf.crawler.jobs.JobManager.resetJobs(JobManager.java:8449) > > ~[mcf-pull-agent.jar:?] > > at > > > > > org.apache.manifoldcf.crawler.system.JobResetThread.run(JobResetThread.java:77) > > [mcf-pull-agent.jar:?] > > > > Thanks, > > Krishna A > > > > > > On Fri, Nov 30, 2018 at 7:00 PM Karl Wright wrote: > > > > > Hi Krishna, > > > > > > First of all I suggest that you *not* use multiprocess-file-example, > and > > > instead use multiprocess-zk-example. > > > > > > Your symptoms suggest many possibilities. But if you move to Zookeeper > > we > > > will be able to eliminate dangling file locks as a complication. So > > please > > > do that first. > > > > > > Karl > > > > > > > > > On Fri, Nov 30, 2018 at 6:29 PM krishna agrawal > > > wrote: > > > > > > > Yeah in our local set up we did Simple example but in server we did > > > > multiprocess-file-example are you suggesting us to upgrade from 2.10 > to > > > > 2.11 ? > > > > > > > > and we are using MY Sql database , > > > > > > > > So most of time i saw nothing is running and still it say job is > > running > > > > and you have to wait for it to complete. > > > > > > > > and restarting also not helping. > > > > > > > > Any other solution woould be greatly appreciated. > > > > > > > > Thanks, > > > > Krishna A > > > > > > > > On Fri, Nov 30, 2018 at 10:50 AM Karl Wright > > wrote: > > > > > > > > > It also may be useful to start with the simple example, which is > not > > > > > multiprocess, and get familiar with using ManifoldCF that way, > before > > > you > > > > > try to go to a more complicated setup. > > > > > > > > > > Thanks, > > > > > Karl > > > > > > > > > > > > > > > On Fri, Nov 30, 2018 at 9:46 AM Karl Wright > > > wrote: > > > > > > > > > > > "simplified multi-process"? There is no such example. > > > > > > > > > > > > These are the examples available. Which one are you using? > > > > > > > > > > > > 11/15/2018 03:40 AM example > > > > > > 11/15/2018 03:40 AM example-proprietary > > > > > > 11/15/2018 03:40 AM multiprocess-file-example > > > > > > 11/15/2018 03:40 AM > > > > > > multiprocess-file-example-proprietary > > > > > > 11/15/2018 03:40 AM multiprocess-zk-example > > > > > > 11/15/2018 03:40 AM > > > > > multiprocess-zk-example-proprietary > > > > > > > > > > > > Cleaning locks makes no sense unless you are using the > > > > multiprocess-file > > > > > > setup. This is deprecated, by the way, in favor of the Zookeeper > > > > setup. > > > > > > > > > > > > As for the buttons, please read: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://manifoldcf.apache.org/release/release-2.11/en_US/end-user-documentation.html#outputs > > > > > > > > > > > > The buttons in question are "Reindex all..." and "Remove all..." > > > > > > > > > > > > Karl > > > > > > > > > > > > > > > > > > On Fri, Nov 30, 2018 at 9:36 AM krishna agrawal < > > > krish.agwl@gmail.com> > > > > > > wrote: > > > > > > > > > > > >> We have deployed the Manifold using > > > > > >> > > > > > >> - Simplified multi-process model > > > > > >> > > > > > >> We did try clean up of lock Sh but that also did not work. > > > > > >> > > > > > >> I dont have forget all document button in output connector. > > > > > >> > > > > > >> [image: image.png] > > > > > >> > > > > > >> On Thu, Nov 29, 2018 at 6:52 PM Karl Wright > > > > > wrote: > > > > > >> > > > > > >>> Hi Krishna, > > > > > >>> > > > > > >>> Please give us some background as to how you've deployed > > > ManifoldCF. > > > > > Are > > > > > >>> you using one of the examples? If so, which one? > > > > > >>> > > > > > >>> The detailed answer to your question is: the job must delete > all > > > > > >>> documents > > > > > >>> it indexed before it can be deleted. That is the typical way > > jobs > > > > > work. > > > > > >>> Thus, if you shut down the target of your output connection, > you > > > may > > > > be > > > > > >>> blocked in deleting your job. > > > > > >>> > > > > > >>> At that point, you can either (a) restart the target of your > > output > > > > > >>> connection, or (b) go to the "view" page for the output > > connection > > > > and > > > > > >>> click both of the "forget all documents" buttons on it. (b) is > > not > > > > > >>> recommended unless you really want to start over fresh on your > > > output > > > > > >>> index. > > > > > >>> > > > > > >>> Thanks, > > > > > >>> Karl > > > > > >>> > > > > > >>> > > > > > >>> On Thu, Nov 29, 2018 at 3:21 PM krishna agrawal < > > > > krish.agwl@gmail.com> > > > > > >>> wrote: > > > > > >>> > > > > > >>> > Hi We are facing issue of action button is not available > > > > > >>> > > > > > > >>> > [image: image.png] > > > > > >>> > > > > > > >>> > I have stop the agent process but still i am not able to > > remove > > > > the > > > > > >>> job > > > > > >>> > it say it > > > > > >>> > > > > > > >>> > there should be some way to forcefully restart and stop the > > > running > > > > > >>> > process ? > > > > > >>> > > > > > > >>> > Job 1542835910915 is busy; you must wait and/or shut it down > > > before > > > > > >>> > deleting it > > > > > >>> > but there is no job running, and i am seeing this message > from > > > > past 3 > > > > > >>> days. > > > > > >>> > > > > > > >>> > is there any ways to clear this? > > > > > >>> > > > > > > >>> > > > > > > >>> > Any help in this matter will be appreciated. > > > > > >>> > > > > > > >>> > Thanks, > > > > > >>> > Krishna A > > > > > >>> > > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > --000000000000ffaa6d057bfb4b3d--