Mailing-List: contact river-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: river-dev@incubator.apache.org
Received-SPF: neutral (athena.apache.org: 209.147.113.130 is neither permitted
 nor denied by domain of mmcgrady@topiatechnology.com)
Subject: Re: datastructure classes
Mime-Version: 1.0 (Apple Message framework v1082)
Content-Type: text/plain; charset=us-ascii
From: MICHAEL MCGRADY <mmcgrady@topiatechnology.com>
In-Reply-To: <4D07E442.5040908@acm.org>
Date: Tue, 14 Dec 2010 13:49:36 -0800
Content-Transfer-Encoding: quoted-printable
Message-Id: <39AED645-40CC-4314-9956-25BE1325B2F5@topiatechnology.com>
References: <4CF89BD3.3030103@acm.org> <4CF909A0.5070409@wonderly.org>
 <4D056E41.7020102@acm.org> <4D057B73.3090905@zeus.net.au>
 <4D05BA22.4000002@acm.org> <4D06672A.10203@wonderly.org>
 <4D066FA2.8050207@wonderly.org> <4D069D21.5070908@zeus.net.au>
 <4D06BA8D.4040602@simulexinc.com> <4D06C523.5000006@acm.org>
 <5E44AB11-EC78-476E-8A6D-C9AEBEFCA3D2@topiatechnology.com>
 <4D079D60.4070607@wonderly.org> <4D07E442.5040908@acm.org>
To: river-dev@incubator.apache.org


On Dec 14, 2010, at 1:40 PM, Patricia Shanahan wrote:

> On 12/14/2010 8:37 AM, Gregg Wonderly wrote:
>> On 12/14/2010 1:36 AM, MICHAEL MCGRADY wrote:
>>> I would say that in addition to just be a fast data structure the =
data
>>> structure
>> > must be fast and accommodate synchronous and asynchronous backups,
>> partitions,
>> > and transactions.
>>=20
>> This is an important issue from the perspective that there are two
>> scenarios that used to be supported by outrigger. A persistent and an
>> non-persistent version used to exist. The persistent version used PSE
>> for serialization to disk. That was a simple yet powerful mechanism. =
Due
>> to licensing (Sun paid for a distribution license), it was in a =
sense,
>> deprecated at the point of River being started.
>>=20
>> For those that don't know about PSE, it used a post compilation =
bytecode
>> manipulator that looked for calls to a "start transaction" method, =
and
>> then found modification assignments to associated data structures, =
and
>> modified the byte code to set a "modified bit" on the associated =
data.
>> When "end transaction" was encountered, it stopped.
>>=20
>> I think it would be a good idea to focus on the performance of the in
>> memory (messaging only type of application) version. The persistent
>> version is a completely different animal and requires some fairly
>> advanced features for managing all of the appropriate control points.
>> Making one code path do both can be somewhat challenging from an all =
out
>> performance perspective.

I agree, useful.  Very useful.


>>=20
>=20
> Thanks for the useful background information.
>=20
> There is one slim hope I can see for a common code path, but it is a
> very long way off.
>=20
> My prejudice, subject to being convinced that another approach would =
be
> better, would be to try to map a persistent version to a relational
> database through SQL. Relational databases deal with transactions, =
ACID,
> distribution, and performance issues. There are a lot of options for
> users, more than for OO databases, at all price points starting at =
free.


This would not be in the long run something we could use.  We need to do =
ACID transactions with in-memory, persistent, functionality.  And, we =
need to have a common data model that can link to SQL, etc, legacy (or =
new) systems. =20

>=20
> The way outrigger uses its FastList looks rather like a sort of
> simplified relational database, with each FastList instance =
representing
> a table and selects being done by linear scan of the table.
>=20
> If we made a persistent version use a relational database to represent
> the space,

This would not be usable for us.  This is too slow and does not have the =
correct QCC features, especially scalability.

> we could then experiment with performance run-offs between
> our best shot at an ad-hoc in-memory implementation, and what we get
> from the persistent version if we drop in an in-memory database
> implementation. If they come close, we could drop the ad-hoc
> implementation and focus all effort on the relational database =
version.
>=20
> It is a slim hope. Often, a custom tuned data structure will =
out-perform
> a specialization of a general data structure. In any case, I agree =
with
> working first on the in-memory version.

I agree on working on the in-memory version too but making it more =
ubiquitous.

>=20
> Patricia

Michael McGrady
Chief Architect
Topia Technology, Inc.
Cel 1.253.720.3365
Work 1.253.572.9712 extension 2037
mmcgrady@topiatechnology.com