From java-dev-return-14313-apmail-lucene-java-dev-archive=lucene.apache.org@lucene.apache.org Thu Jun 01 23:38:11 2006 Return-Path: Delivered-To: apmail-lucene-java-dev-archive@www.apache.org Received: (qmail 80040 invoked from network); 1 Jun 2006 23:38:10 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 1 Jun 2006 23:38:10 -0000 Received: (qmail 32470 invoked by uid 500); 1 Jun 2006 23:38:10 -0000 Delivered-To: apmail-lucene-java-dev-archive@lucene.apache.org Received: (qmail 31863 invoked by uid 500); 1 Jun 2006 23:38:08 -0000 Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-dev@lucene.apache.org Delivered-To: mailing list java-dev@lucene.apache.org Received: (qmail 31851 invoked by uid 99); 1 Jun 2006 23:38:08 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jun 2006 16:38:08 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=RCVD_IN_BL_SPAMCOP_NET,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: domain of simon.willnauer@googlemail.com designates 66.249.92.169 as permitted sender) Received: from [66.249.92.169] (HELO ug-out-1314.google.com) (66.249.92.169) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jun 2006 16:38:07 -0700 Received: by ug-out-1314.google.com with SMTP id u2so297838uge for ; Thu, 01 Jun 2006 16:37:45 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=googlemail.com; h=received:message-id:date:from:reply-to:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=englM0/PGfPwQ/jmnn7B/6A2w3R8zJs48PK1H1ArNavmls02In/qYiNM1HxasSnCB6oZ6Otlx69AhbiuunwXgZdOjyPOHmQJdWZcfIl8Van/RdsK8xXFHF1/NeYFYURFfR+ao3RtXgQ9atHuAoLf1H8S7RPAbA3tES7f3vzuaH8= Received: by 10.67.100.12 with SMTP id c12mr235894ugm; Thu, 01 Jun 2006 16:37:44 -0700 (PDT) Received: by 10.66.236.1 with HTTP; Thu, 1 Jun 2006 16:37:44 -0700 (PDT) Message-ID: Date: Fri, 2 Jun 2006 01:37:44 +0200 From: "Simon Willnauer" Reply-To: simon.willnauer@gmail.com To: java-dev@lucene.apache.org Subject: GData Server - Lucene storage MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Hello folks, as I'm the only developer on the project due to the SummerOfCode program it is quiet a tough task to discuss all the architecture with you on the mailing list. For this reason I decided to create UML diagrams to discuss the main components. I will not attach the uml to the mails rather upload it to a server so you can download an study it. Well, the next thing I have to implement is a storage to store the entries in. I will provide 2 kinds of storage's (lucene and BerkleyDB based). The first will be a lucene index to store the entries identified by the entry ID and feed ID stored in the index as a Keyword (used to be Field.Keyword). The underlaying lucene storage will only be used to store the entries compressed. Which feed entries to retrieve from the lucene storage will be based on results of the indexing/search component as every client request to a gdata server is a query to the index. So the results of the search are entry ids and a corresponding feed. These entries will be retrieved from the storage and send back to the client. The storage component does also provide delete / update and insert functionality (wouldn't be a storage without these). The biggest problem with the lucene storage is to achieve a transactional state. Imagine the following scenario: An update request comes in. -> the entry to update will be added to the lucene writer who writes the update. But another delete request has locked the index and an IOException will be thrown. So the update request will queue the entry and retries to obtain the lock. No problem so far. But if the index writer can not open the index due to some other error (the index could not be found) the exception will also be an IOExc. Is there any way to figure out whether the IOException is caused due to a lock which would be alright or due to some other serious reasons? I added some comments on the UML to describe the arch. to you in more detail. So please download the file and have a look at it. http://www.javawithchopsticks.de/webaccess/lucenestorage.pdf I will appreciate all your comments!! regards Simon --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org For additional commands, e-mail: java-dev-help@lucene.apache.org