Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of elf2001@gmail.com designates
 209.85.210.180 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAHqJuML_HvJxHq77-ZrAnsmy1qQeaYF_0TArir-=gz7F0L-GKg@mail.gmail.com>
References: 
 <CAHqJuML_HvJxHq77-ZrAnsmy1qQeaYF_0TArir-=gz7F0L-GKg@mail.gmail.com>
Date: Wed, 28 Dec 2011 20:16:08 +0200
Message-ID: 
 <CAHqJuMLTaeeHMMr-0pyjVy3=9VVbB+76XnYt_VGBP3paKLWVrw@mail.gmail.com>
Subject: Re: couchdb disk storage format - why so large overhead?
From: Alexey Loshkarev <elf2001@gmail.com>
To: user@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1

>
> Also, I just realized here
> (http://www.erlang.org/doc/apps/erts/erl_ext_dist.html), cite:
> ===============
> A float is stored in string format. the format used in sprintf to
> format the float is "%.20e" (there are more bytes allocated than
> necessary)
> ===============
> So, every float requires 33 bytes off disk space. Not so efficient.

Reading specs I realized that using minor_version = 1 in
term_to_binary options makes floats be 9-bytes long instead 33.
It just search/replace in few files.

I tested my data blob with this map function:

fun({Doc}) ->
   Emit(<<"raw">>, size(term_to_binary(Doc))),
   Emit(<<"raw_1">>, size(term_to_binary(Doc, [{minor_version, 1}])))
end.

And received ~10% space (~600MB instead of ~700MB) usage decrease.

Do I need to file a bug in Jira for it?


-- 
----------------
Best regards
Alexey Loshkarev
mailto:elf2001@gmail.com