Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Received-SPF: pass (nike.apache.org: domain of eric.newton@gmail.com
 designates 209.85.216.45 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAGUtCHpPs5T_-nj6hkNcjnqZQu3h4kHDN=vqzJmyBDj7AkNqgw@mail.gmail.com>
References: 
 <AC78983C72177B4D9D1C14F7F4AEBA2144915A15D3@aplesstripe.dom1.jhuapl.edu>
	<CAGUtCHpPs5T_-nj6hkNcjnqZQu3h4kHDN=vqzJmyBDj7AkNqgw@mail.gmail.com>
Date: Tue, 29 Oct 2013 23:05:11 -0400
Message-ID: 
 <CADxc9BnrR=9tVgSUhcnT_yism2o=Amcccmmfi0tdLvJBRbzxiA@mail.gmail.com>
Subject: Re: sum of mutation.numBytes() significantly different from rfile
 size
From: Eric Newton <eric.newton@gmail.com>
To: "user@accumulo.apache.org" <user@accumulo.apache.org>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

For comparison, I posted this some time ago:

http://tinyurl.com/k28bkbg

I was surprised that RFile was smaller than a gzip'd CSV file, too.

On Tue, Oct 29, 2013 at 6:35 PM, Keith Turner <keith@deenlo.com> wrote:
>
>
>
> On Tue, Oct 29, 2013 at 5:50 PM, Slater, David M. <David.Slater@jhuapl.ed=
u>
> wrote:
>>
>> Hello,
>>
>>
>>
>> I=92m seeing about an order of magnitude difference between the number o=
f
>> bytes returned by mutation.numBytes() and the size of the rfiles on disk
>> (Accumulo 1.4.2). Note that all of my mutations are new entries, and the=
re
>> are no combiners running.
>>
>>
>>
>> While I understand that there is some compression on the rfile, I would =
be
>> really surprised if it was 10:1.
>>
>>
>>
>> My entries are composed of a row ID (most of which is equivalent to the
>> previous row ID), an empty column family, a nonempty column qualifier (w=
hich
>> likely shares a lot with the previous qualifier), and an empty value. An
>> example of the rowID and column qualifier might be:
>
>
> In 1.4 if a field (row, col fam, etc) in key is the same as the previous,
> then its not written again.  So if the row is the same in 10 consecutive
> keys, its only written once.   Maybe this explains the difference. Scan t=
he
> table to make sure all of the data you expect to be there is there.
>
>>
>>
>>
>> (forward table)
>>
>> 0000000000000|9|fa19                 IP|127.000.000.001
>>
>> 0000000000000|9|fa19                  PORT|00080
>>
>> =85
>>
>> 0000000000000|9|fa22                  IP|128.032.144.139
>>
>> =85
>>
>> <timeblock>|<hash>|<uid>       <index>|<textual value>
>>
>>
>>
>> OR
>>
>> (reverse table)
>>
>> 0000000000000|IP|127.000.000.001         fa19
>>
>> 0000000000000|IP|127.000.000.001         fd02
>>
>> 0000000000000|IP|127.000.000.002         123
>>
>> =85
>>
>> 0000000000000|PORT|00080                      fa19
>>
>>
>>
>> The numBytes() method appears to return a number of bytes equal to the
>> string length of the row ID and column qualifiers, plus 26 * # of column
>> qualifiers.
>>
>>
>>
>> Is there something else that I=92m missing, or would this possibly compr=
ess
>> by that much?
>>
>>
>>
>> Thanks,
>>
>> David
>
>