Return-Path: Delivered-To: apmail-jakarta-lucene-user-archive@apache.org Received: (qmail 37967 invoked from network); 29 Jun 2002 08:15:48 -0000 Received: from unknown (HELO nagoya.betaversion.org) (192.18.49.131) by 209.66.108.5 with SMTP; 29 Jun 2002 08:15:48 -0000 Received: (qmail 26407 invoked by uid 97); 29 Jun 2002 08:16:05 -0000 Delivered-To: qmlist-jakarta-archive-lucene-user@jakarta.apache.org Received: (qmail 26373 invoked by uid 97); 29 Jun 2002 08:16:05 -0000 Mailing-List: contact lucene-user-help@jakarta.apache.org; run by ezmlm Precedence: bulk List-Unsubscribe: List-Subscribe: List-Help: List-Post: List-Id: "Lucene Users List" Reply-To: "Lucene Users List" Delivered-To: mailing list lucene-user@jakarta.apache.org Received: (qmail 26358 invoked by uid 98); 29 Jun 2002 08:16:04 -0000 X-Antivirus: nagoya (v4198 created Apr 24 2002) Reply-To: From: "Nader S. Henein" To: "Lucene Users List" Subject: RE: Stress Testing Lucene Date: Sat, 29 Jun 2002 12:16:39 +0400 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 In-Reply-To: <3D1B30CE.1030003@lucene.com> Importance: Normal X-Spam-Rating: 209.66.108.5 1.6.2 0/1000/N X-Spam-Rating: 209.66.108.5 1.6.2 0/1000/N That's the weird thing I wasn't writing to the index at the time I was searching (hardcore searching) 20 clients each one issuing 20 simultaneous search request .. it was going fine until it started throwing errors at me and when I looked at the logs, I found a set of "Too many files open" error. Previously this only happened if their was a crash on the server while indexing leaving an un-optimized index with 800+ files. -----Original Message----- From: Doug Cutting [mailto:cutting@lucene.com] Sent: Thursday, June 27, 2002 7:36 PM To: Lucene Users List Subject: Re: Stress Testing Lucene It's very hard to leave an index in a bad state. Updating the "segments" file atomically updates the index. So the only way to corrupt things is to only partly update the segments file. But that too is hard, since it's first written to a temporary file, which is then renamed "segments". The only vulnerability I know if is that in Java on Win32 you can't atomically rename a file to something that already exists, so Lucene has to first remove the old version. So if you were to crash between the time that the old version of "segments" is removed and the new version is moved into place, then the index would be corrupt, because it would have no "segments" file. Doug Scott Ganyo wrote: > Which came first--the out of file handles error or the corruption? I > haven't looked, but I would guess that if you ran into the file handles > exception while writing, that might leave Lucene in a bad state. Lucene > isn't transactional and doesn't really have the ACID properties of a > database... > > >>-----Original Message----- >>From: Nader S. Henein [mailto:nsh@bayt.net] >>Sent: Wednesday, June 26, 2002 11:45 PM >>To: Lucene Users List >>Subject: RE: Stress Testing Lucene >> >> >>I rebooted my machine and still the same issue .. if I know >>what caused that to happen, I would be able to solve it with >>some source tweaking, and it's not the files handles on the machine I >>got over that problem months ago. Let's consider worst case >>scenario and >>that >>corruption did occur what could be the reasons, I'm goig to need some >>insider >>help to get through this one. >> >>N. >> >>-----Original Message----- >>From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com] >>Sent: Wednesday, June 26, 2002 7:15 PM >>To: 'Lucene Users List' >>Subject: RE: Stress Testing Lucene >> >> >>1) Are you sure that the index is corrupted? Maybe the file >>handles just >>haven't been released yet. Did you try to reboot and try again? >> >>2) To avoid the too-many files problem: a) increase the >>system file handle >>limits, b) make sure that you reuse IndexReaders as much as >>you can rather >>across requests and client rather than opening and closing them. >> >> >>>-----Original Message----- >>>From: Nader S. Henein [mailto:nsh@bayt.net] >>>Sent: Wednesday, June 26, 2002 10:11 AM >>>To: lucene-user@jakarta.apache.org >>>Subject: Stress Testing Lucene >>>Importance: High >>> >>> >>> >>>Hey people, >>> >>>I'm running a Lucene (v1.2) servlet on resin and I must say >>>compared to >>>Oracle Intermedia >>>it's working beautifully. BUT today, I started stress testing and I >>>downloaded a program called >>>Web Roller, witch simulates clients, requests , >>>multi-threading .. the works >>>and I was testing >>>I was doing something like 50 simultaneous requests and I was >>>repeating that >>>10 times in a row. >>> >>>but then something happened and the index got corrupted, >>>every time I try >>>opening the index >>>with the reader to search or open with the writer to optimize >>>I get that >>>damned too-many files >>>open error. I can imagine that every application on the market has a >>>breaking point and these breaking >>>points have side effects, so is the corruption of the index a >>>side effect >>>and if so is there a way that >>>I configure my web server to crash before the corruption >>>occurs, I'd rather >>>re-start the web server and >>>throw some people off wack rather that have to re-build the >>>index or revert >>>to an older version. >>> >>>Do you know of any way to safeguard against this ? >>> >>>General Info: >>>The index is about 45 MB with 60 000 XML files each >>>containing 18-25 fields. >>> >>> >>>Nader S. Henein >>>Bayt.com , Dubai Internet City >>>Tel. +9714 3911900 >>>Fax. +9714 3911915 >>>GSM. +9715 05659557 >>>www.bayt.com >>> >>> >>>-- >>>To unsubscribe, e-mail: >>> >>>For additional commands, e-mail: >>> >>> >> >>-- >>To unsubscribe, e-mail: >> >>For additional commands, e-mail: >> >> > -- To unsubscribe, e-mail: For additional commands, e-mail: -- To unsubscribe, e-mail: For additional commands, e-mail: