Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ignite.apache.org
MIME-Version: 1.0
In-Reply-To: <CAGt6o8mrGKD=yPR_cVpvkYtCFbrqeL0gss83P-7VY-DWV4dP=A@mail.gmail.com>
References: <000201d23022$319dc450$94d94cf0$@neulion.com.cn>
 <1477598139155-8563.post@n6.nabble.com> <CAGt6o8kV58m0sXDb39-=tFUpJoW14kRjgOrGrEL_NTDnLiAqPA@mail.gmail.com>
 <CABuYRcpz+5O7jHZNEP-C2GiPMpLkOPkOaiD_LTVRYt5Sju9hrQ@mail.gmail.com>
 <CAGt6o8k8HH0C0+iu+z8s1jibpPyc5tZKpA7DCLuO7BL4LkLeTw@mail.gmail.com>
 <CABuYRcpYaJJVKX5CKKiGpUh3GB5dcCpe4rSsmsu8ju-ercc6VQ@mail.gmail.com>
 <CAGt6o8krV0x_X7MOGcLZJK33fZE7e9mWPCV_bg5bpx_Qz49aKg@mail.gmail.com>
 <CABuYRcrQMecCiNPRD6moTxP3=UdgMnVW+xA_WEu-TNKFpU0TMA@mail.gmail.com> <CAGt6o8mrGKD=yPR_cVpvkYtCFbrqeL0gss83P-7VY-DWV4dP=A@mail.gmail.com>
From: Igor Sapego <isapego@gridgain.com>
Date: Mon, 31 Oct 2016 20:38:34 +0300
Message-ID: <CALOGAow6Leem62XDq-ELPCPq6kAdqrVRMqN=GDVeYeFCzJX0sg@mail.gmail.com>
Subject: Re: BinaryObject pros/cons
To: dev@ignite.apache.org
Cc: Valentin Kulichenko <valentin.kulichenko@gmail.com>
Content-Type: multipart/alternative; boundary=001a114b253c67d1e905402cade5
archived-at: Mon, 31 Oct 2016 17:44:20 -0000

--001a114b253c67d1e905402cade5
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Vladimir,

How about some reserved value? I.e -1 offset means a default/null value
should be used?

Best Regards,
Igor

On Mon, Oct 31, 2016 at 5:05 PM, Vladimir Ozerov <vozerov@gridgain.com>
wrote:

> Valya,
>
> Do you have any ideas how to implement this? We write field offsets in th=
e
> footer. If field is not written, then what should be used for its offset?
>
> On Mon, Oct 31, 2016 at 4:56 PM, Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
> > Vladimir,
> >
> > These are good points, but I'm not suggesting to change the schema. If
> one
> > writes five fields, the schema should have five fields in any case,
> > regardless of values. I only suggest to change the internal
> representation
> > of the object and do not save fields with default values in the byte
> array
> > as we don't really need them there.
> >
> > -Val
> >
> > On Sun, Oct 30, 2016 at 12:24 PM, Vladimir Ozerov <vozerov@gridgain.com=
>
> > wrote:
> >
> >> Valya,
> >>
> >> I have several concerns:
> >> 1) Correctness: hasField() will not work properly. But probably we can
> >> fix that by adding this info to schema.
> >> 2) Performance: we have lots optimizations which depend on either
> >> "stable" object schema, or low number of schemas. We will effectively
> turn
> >> them off.
> >> But what concerns me even more, is that we may end up in enormous numb=
er
> >> of schemas. E.g. consider an object with 10 number fields. If all fiel=
ds
> >> could be zero, we may end up in something like 2^10 schemas.
> >>
> >> Vladimir.
> >>
> >> 29 =D0=BE=D0=BA=D1=82. 2016 =D0=B3. 0:37 =D0=BF=D0=BE=D0=BB=D1=8C=D0=
=B7=D0=BE=D0=B2=D0=B0=D1=82=D0=B5=D0=BB=D1=8C "Valentin Kulichenko" <
> >> valentin.kulichenko@gmail.com> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=
=BB:
> >>
> >> Vova,
> >>>
> >>> Why do we need to write zeros and nulls in the first place? What's th=
e
> >>> value of having them in the byte array?
> >>>
> >>> -Val
> >>>
> >>> On Fri, Oct 28, 2016 at 1:18 AM, Vladimir Ozerov <vozerov@gridgain.co=
m
> >
> >>> wrote:
> >>>
> >>>> Valya,
> >>>>
> >>>> Currently null value is written as one byte, while zero value of lon=
g
> >>>> type is written as 9 bytes. I want to improve that and write zeros a=
s
> one
> >>>> byte as well.
> >>>>
> >>>> As per var-length encoding, I am strongly against it. It saves IO an=
d
> >>>> memory at the cost of CPU. If we encode numbers in this way we will
> >>>> slowdown SQL (which is already not very fast, to be honest). Because
> >>>> instead of a single read memory read, we will have to perform multip=
le
> >>>> reads and then apply some mechanics to restore original value. We
> already
> >>>> have such problem with Strings - Java stores them as UTF-16, but we
> encode
> >>>> them as UTF-8. As a result every read of a string field in SQL
> results in
> >>>> decoding overhead.
> >>>>
> >>>> Vladimir.
> >>>>
> >>>> On Fri, Oct 28, 2016 at 6:07 AM, Valentin Kulichenko <
> >>>> valentin.kulichenko@gmail.com> wrote:
> >>>>
> >>>>> Cross-posting this to dev list.
> >>>>>
> >>>>> Vladimir,
> >>>>>
> >>>>> To be honest, I don't see much difference between null values for
> >>>>> objects and zero values for primitives. From BinaryObject semantics
> >>>>> standpoint, both are default values for corresponding types. These
> values
> >>>>> will be returned from the BinaryObject.field() method regardless of
> whether
> >>>>> we actually save then in the byte array or not. Having said that,
> why don't
> >>>>> we just skip them during write?
> >>>>>
> >>>>> You optimization will be still useful though, because there are oft=
en
> >>>>> a lot of ints and longs that are not zeros, but still small and can
> fit 1-2
> >>>>> bytes. We already added such compaction in direct message marshalin=
g
> and it
> >>>>> reduced overall traffic by around 30%.
> >>>>>
> >>>>> -Val
> >>>>>
> >>>>>
> >>>>> On Thu, Oct 27, 2016 at 2:21 PM, Vladimir Ozerov <
> vozerov@gridgain.com
> >>>>> > wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I am not very concerned with null fields overhead, because usually
> it
> >>>>>> won't be significant. However, there is a problem with zeros. User
> object
> >>>>>> might have lots of int/long zeros, this is not uncommon. And each
> zero will
> >>>>>> consume 4-8 additional bytes. We probably will implement special
> >>>>>> optimization which will write such fields in special compact forma=
t.
> >>>>>>
> >>>>>> Vladimir.
> >>>>>>
> >>>>>> On Thu, Oct 27, 2016 at 10:55 PM, vkulichenko <
> >>>>>> valentin.kulichenko@gmail.com> wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> Yes, null values consume memory. I believe this can be optimized,
> >>>>>>> but I
> >>>>>>> haven't seen issues with this so far. Unless you have hundreds of
> >>>>>>> fields
> >>>>>>> most of which are nulls (very rare case), the overhead is minimal=
.
> >>>>>>>
> >>>>>>> -Val
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> View this message in context: http://apache-ignite-users.705
> >>>>>>> 18.x6.nabble.com/BinaryObject-pros-cons-tp8541p8563.html
> >>>>>>> Sent from the Apache Ignite Users mailing list archive at
> Nabble.com.
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >
>

--001a114b253c67d1e905402cade5--