Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of earwin@gmail.com designates
 72.14.220.155 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=tEEvNMkpYA96WbsHM3/jr0ggec1MmbfefFbmeeS1wIrlY3Q3ALSWN/FXaCEjj2oKHk
         dUB8MYATk22pD+PBX+2zxwIWt42dLzkKhWvAQP9qPp7rpqGB764G42IoDJZMYOLx9PAf
         GLIpriwwWOJndTYJpXotLOvDgzg95XbsOjBuw=
MIME-Version: 1.0
In-Reply-To: <85d3c3b60908081539y4b36d15ub376012646af20e7@mail.gmail.com>
References: <028AB4BD-37D4-4C11-8481-CD579C88A5A7@lucidimagination.com>
	 <8A9FCECC-587D-4F33-BCB4-866F31A2A915@apache.org>
	 <786fde50907300756t2a6bfbd0oe209ed479260d489@mail.gmail.com>
	 <9ac0c6aa0907300916y1eceeecavfd9074caab469101@mail.gmail.com>
	 <4A7741A8.5070004@gmail.com>
	 <85d3c3b60908081539y4b36d15ub376012646af20e7@mail.gmail.com>
Date: Sun, 9 Aug 2009 03:26:39 +0400
Message-ID: <59b3eb370908081626o4c4b232eqb314600a4e10297f@mail.gmail.com>
Subject: Re: ConcurrentMergeScheduler and MergePolicy question
From: Earwin Burrfoot <earwin@gmail.com>
To: java-dev@lucene.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

> Perhaps the ideal search system architecture that requires
> optimizing is to dedicate a server to it, copy the index to the
> optimize server, do the optimize, copy the index off (to a
> search server) and start again for the next optimize task.
>
> I wonder how/if this would work with Hadoop/HDFS as copying
> 100GB around would presumably tie up the network? Also, I've
> found rsyncing large optimized indexes to be time consuming and
> wreaks havoc on the searcher server's IO subsystem. Usually this
> is unacceptable for the user as the queries will suddenly
> degrade.
You don't have to copy. You can have one machine optimize your indexes
whilst other serves user requests, then they switch roles, rinse,
repeat. This approach also works with sharding, and more than 2-way
mirroring.

--=20
Kirill Zakharenko/=D0=9A=D0=B8=D1=80=D0=B8=D0=BB=D0=BB =D0=97=D0=B0=D1=85=
=D0=B0=D1=80=D0=B5=D0=BD=D0=BA=D0=BE (earwin@gmail.com)
Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423
ICQ: 104465785

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org