Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of lasizoillo@gmail.com designates
 209.85.218.163 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=QP3pe7/Jt3eE6AwIfpVYAdJwlR9HWeOrp3DMrgBhPi5kEunLTe+AULckyIQlQ+ZWTA
         PD2JjFdVC15wWSrLh9vS51ZSSsFmJFXChGApuJv7z3QKJlW12SIltFTdBjpVlcOLkULl
         V9cowUR8ve87E4R75ZhuEiT4J1goS1KzNYIXo=
MIME-Version: 1.0
In-Reply-To: <4535f6cd0902260534r33db6665mfdaf9fc535219c19@mail.gmail.com>
References: <4535f6cd0902252002j79241e14ncd6e4c9cfade8555@mail.gmail.com>
	 <7D5CCC6A-B29A-47FF-B4E1-4261A844EC96@mooseyard.com>
	 <4535f6cd0902252030p526843fdve16305b850c6c114@mail.gmail.com>
	 <8C1FCA44-C63F-40BC-AA73-C464ABE92959@apache.org>
	 <4535f6cd0902260534r33db6665mfdaf9fc535219c19@mail.gmail.com>
Date: Thu, 26 Feb 2009 17:22:25 +0100
Message-ID: <49414f570902260822kd9561b0na6df0e0ab44f45e7@mail.gmail.com>
Subject: Re: What's the speed(performance) of couchdb?
From: lasizoillo <lasizoillo@gmail.com>
To: user@couchdb.apache.org
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

2009/2/26 Scott Zhang <getyourcontacts@gmail.com>:
> Thanks. Jan.
> I am using 0.8.1 beta installer download from couchDB wiki.
>
> ---------------------------------------------------
> The question is what you need you system to look like eventually. If this=
 is
> an initial data-import and after that you get mostly read requests, the
> longer
> insertion time will amortize over time.
> ---------------------------------------------------
> Yes. I am trying to transfer the keyword-index database from SqlServer to=
 a
> database to banlance the pressure on SqlServer database. =A0The I will do
> search use the keyword-index database. =A0So the first import process is =
very
> important to me. After initial data-import process, I can slowly add new
> keywords in.
>
> My candidates are Mnesia(prefer at first time), couchDB, postgresql, mysq=
l.
>

Maybe a Berkeley DB like database is a good option. Tokyo Tyrant
http://tokyocabinet.sourceforge.net/benchmark.pdf

> The mnesia's performance(insert) is good. But there is a weird issue abou=
t
> Http_client in erlang windows release. After reporting that bugs on erlan=
g
> mailing list but get no reponse. =A0I finally have to give up mnesia.
>
> CouchDB is my second try, but the problem is as I showed in the mail.
>
> Now I am working with postgresql. At least I see it works as good as
> SqlServer.
>
>
> I will check couchDB back soon later after your 1.0 release. But as I see=
,
> for user play with huge records, I can see when I saw couchDB 1.0. I will=
 be
> playing with 1 billion records. So speed is the most important thing I ca=
re.
>

If you want do a very heavy migration is better develop a script that
generate de db file directly. The net layers consume a lot of time.
Develop an aplication that write bytes directly to disk can save you a
lot of time.

I don't know how to do profiling in erlang to do a estimation of the
time saving.


Regards,

Javi

PD: Excuse me for my bad english

> Cheers.
> Thanks for your hard working.
>
>
> Regards.
> Scott
>
>
>
>
>
>
>
> On Thu, Feb 26, 2009 at 6:04 PM, Jan Lehnardt <jan@apache.org> wrote:
>
>> Hi Scott,
>>
>> thanks for your feedback. As a general note, you can't expect any magic
>> from CouchDB. It is bound by the same constraint all other programmes
>> are. To get the most out of CouchDB or SqlServer or MySQL, you need
>> to understand how it works.
>>
>>
>> On 26 Feb 2009, at 05:30, Scott Zhang wrote:
>>
>> =A0Hi. Thanks for replying.
>>> But what a database is for if it is slow? Every database has the featur=
e
>>> to
>>> make cluster to improve speed and capacity (Don't metion "access" thing=
s).
>>>
>>
>> The point of CouchDB is allowing high numbers of concurrent requests. Th=
is
>> gives you more throughput for a single machine but not necessarily faste=
r
>> single query execution speed.
>>
>>
>> =A0I was expecting couchDB is as fast as SqlServer or mysql. At least I =
know,
>>> mnesia is much faster than SqlServer. But mnesia always throw harmless
>>> "overload" message.
>>>
>>
>> CouchDB is not nearly as old as either of them. Did you really expect a
>> software in alpha stages to be faster than fine-tuned systems that have
>> been used in production for a decade or longer?
>>
>>
>> =A0I will try bulk insert now. But be =A0fair, I was inserting =A0into s=
qlserver
>>> one insert one time.
>>>
>>
>> Insert speed can be speed up in numerous ways:
>>
>> =A0- Use sequential descending document ids on insert.
>> =A0- Use bulk insert.
>> =A0- Bypass the HTTP API and insert native Erlang terms and skip JSON
>> conversion.
>>
>> The question is what you need you system to look like eventually. If thi=
s
>> is
>> an initial data-import and after that you get mostly read requests, the
>> longer
>> insertion time will amortize over time.
>>
>> What version is the Windows binary you are using? If it is still 0.8, yo=
u
>> should
>> try trunk (which most likely means switching to some UNIXy system).
>>
>> Cheers
>> Jan
>> --
>>
>>
>>
>>
>>
>>
>>
>>> Regards.
>>>
>>>
>>>
>>>
>>> On Thu, Feb 26, 2009 at 12:18 PM, Jens Alfke <jens@mooseyard.com> wrote=
:
>>>
>>>
>>>> On Feb 25, 2009, at 8:02 PM, Scott Zhang wrote:
>>>>
>>>> But the performance is as bad as I can image, After several minutes ru=
n,
>>>> I
>>>>
>>>>> only inserted into 120K records. I saw the speed is ~20 records each
>>>>> second.
>>>>>
>>>>>
>>>> Use the bulk-insert API to improve speed. The way you're doing it, eve=
ry
>>>> record being added is a separate transaction, which requires a separat=
e
>>>> HTTP
>>>> request and flushing the file.
>>>>
>>>> (I'm a CouchDB newbie, but I don't think the point of CouchDB is speed=
.
>>>> What's exciting about it is the flexibility and the ability to build
>>>> distributed systems. If you're looking for a traditional database with
>>>> speed, have you tried MySQL?)
>>>>
>>>> =97Jens
>>>>
>>>
>>
>