Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=sbcglobal.net;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=cHEF20SC/WXuo0OYNA2Kq2ermzMpyuv0XDi+oq1kJwgLr65CxP38pezm/DAAYfUaVT0fsD3APVCphHR1knOkkfCGMpQaV+SeL7AhU/298OR3zJhvrQKFwh7CQsnqbutlffkfcMXi3QdBjhCSg7vVt+iCao+Dl1guDGW0NSBGQgo=;
Message-ID: <501613.31980.qm@web82105.mail.mud.yahoo.com>
Date: Mon, 1 Nov 2010 21:41:43 -0700 (PDT)
From: Dennis Gearon <gearond@sbcglobal.net>
Subject: RE: Ensuring stable timestamp ordering
To: solr-user@lucene.apache.org
In-Reply-To: 
 <2E6A89A648463A4EBF093A9062C16683032C20309521@SBMAILBOX1.sb.statsbiblioteket.dk>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

how about a timrstamp with either a GUID appended on  the end of it?=0A=0A=
=0ADennis Gearon=0A=0ASignature Warning=0A----------------=0AIt is always a=
 good idea to learn from your own mistakes. It is usually a better idea to =
learn from others=E2=80=99 mistakes, so you do not have to make them yourse=
lf. from 'http://blogs.techrepublic.com.com/security/?p=3D4501&tag=3Dnl.e03=
6'=0A=0AEARTH has a Right To Life,=0A  otherwise we all die.=0A=0A=0A--- On=
 Sun, 10/31/10, Toke Eskildsen <te@statsbiblioteket.dk> wrote:=0A=0A> From:=
 Toke Eskildsen <te@statsbiblioteket.dk>=0A> Subject: RE: Ensuring stable t=
imestamp ordering=0A> To: "solr-user@lucene.apache.org" <solr-user@lucene.a=
pache.org>=0A> Date: Sunday, October 31, 2010, 12:18 PM=0A> Dennis Gearon [=
gearond@sbcglobal.net]=0A> wrote:=0A> > Even microseconds may not be enough=
 on some really=0A> good, fast machine.=0A> =0A> True, especially since the=
 timer might not provide=0A> microsecond granularity although the returned =
value is in=0A> microseconds. However, an unique timestamp generator should=
=0A> keep track of the previous timestamp to guard against=0A> duplicates. =
Uniqueness can thus be guaranteed by waiting a=0A> bit or cheating on the d=
ecimals. With microseconds can=0A> produce 1 million timestamps / second. W=
hile I agree that=0A> duplicates within microseconds can occur on a fast ma=
chine,=0A> guaranteeing uniqueness by waiting should only be a=0A> performa=
nce problem when the number of duplicates is high.=0A> That's still a few y=
ears off, I think.=0A> =0A> As Michael pointed out, using normal timestamps=
 as unique=0A> IDs might not be such a great idea as it effectively locks=
=0A> index-building to a single JVM. By going the ugly route and=0A> expres=
sing the time in nanos with only microsecond=0A> granularity and use the la=
st 3 decimals for a builder ID=0A> this could be fixed. Not very clean thou=
gh, as the contract=0A> is not expressed in the data themselves but must=0A=
> nevertheless be obeyed by all builders to avoid collisions.=0A> It also r=
aises the question of who should assign the builder=0A> IDs. Not trivial in=
 an anarchistic setup where new builders=0A> can be added by different cont=
rollers.=0A> =0A> Pragmatists might use the PID % 1000 or similar for the=
=0A> builder ID as it does not require coordination, but this is=0A> where =
the Birthday Paradox hits us again: The chance of two=0A> processes on diff=
erent machines having the same PID is 10%=0A> if just 15 machines are used =
(1% for 5 machines, 50% for 37=0A> machines). I don't like those odds and t=
hat's assuming that=0A> the PIDs will be randomly distributed, which they w=
on't. It=0A> could be lowered by reserving more decimals for the salt,=0A> =
but then we would decrease the maximum amount of timestamps=0A> / second, s=
till without guaranteed uniqueness. Guys a lot=0A> smarter than me has spen=
d time on the unique ID problem and=0A> it's clearly not easy: Java's UUID =
takes up 128 bits.=0A> =0A> - Toke