felix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Leangen <o...@leangen.net>
Subject Re: [Converter] Data Tree Structures
Date Tue, 16 Aug 2016 12:29:02 GMT

Hi David D.,

Thanks for your input.

What you write makes sense, but I don’t think it really applies to what I am trying to suggest.

All I am asking for is an introspective view on the DTO data tree structure, nothing more.
Since this is being parsed anyway, this code provides the perfect opportunity to open up this
view of the data. Anything beyond that would be out of scope, but would be able to interact
nicely and easily with the data tree exposed by this new interface.

This is in no way a persistence structure, or any kind of database, especially not relational.
Having a parsable, manipulatable view of the data structure would, however, allow much easier
processing of the data to map to a database, whether it be relational, key/value, graph, etc.

Aa an example, a few years back, I had to write exactly this type of functionality in order
to do a deep storage of a data structure in Berkeley DB. The top-level objects “shared”
other objects, so it was decided that it would be wasteful to have millions of duplicate copies
of those mid-level objects. When storing and retrieving data, I had to introspect the data
structure and dejoin the mid-level objects, store them in their own key/value store, then
join them again when the data was retrieved. Document stores and graph DBs also need instructions
as to how to store the data. Not just RDBs.

I agree with what you write, that the relationships are too complex to deduce from the DTO
structure. Objects that are embedded in the data do not define the relationships; there is
not enough information for that. That type of information would be beyond the scope of what
I am proposing. However, the annotations (for instance) that you would use to mark the relationships
would work very well with this data structure. With that access to the data tree, I could
easily kick off a process against the tree that looks for these annotations so that I can
interface with my persistence layer.

That is for persistence. If you want to work with, for instance, Lucene, for indexing the
data, I think that this type of introspective access to the data tree would also be very useful.

All I am suggesting is that we provide access to the tree, which would _enable_ many types
of processing against it, not that we actually include anything new.

I do understand that it is a step away from a simple “Converter”, but the parsing is essentially
the same. Since the hard work is already being done, why not take advantage of it here? Even
if this tree view ends up being a completely different service, the same code base could easily
serve the two.


> On Aug 16, 2016, at 8:35 PM, David Daniel <david.daniel.1979@gmail.com> wrote:
> In my opinion with the scenarios you described you are no longer describing
> the data but a persistence structure.  Indexes are about duplicating the
> data so it can be retrieved faster given a storage structure and some idea
> of that structure is needed to create them.  If the persisting side could
> determine the structure solely off the presence of the key then I think it
> is not bad to have in but it is impossible for a key to cover every area.
> I think I just don't like the idea of tying the structure into the
> definition without value types in java.  I am also wary about pulling in
> object relationships because that is getting into type/category theory
> which java does not have a standard way of doing.  With relational data if
> you have a key then you can just have a limited number relationships by
> default like 1 to 1 or 1 to many or many to many but I don't like the idea
> of having DTO's as relational defined objects while most things are moving
> away from that.  I also think you cannot accurately define most
> relationships in a small structure.  How do you define multiple elements as
> sum types or product types.  How do you show one DTO has an inherited
> relationship vs a compositional one.  I think there are many functional
> languages that run on the jvm like scala, clojure, kotlin and hazkel that
> do a good job of allowing programmers to create their own relationships
> that can be defined on data and I dont think the spec should be creating
> another.  I think having DTO's as they are and then the programmer can use
> the converter to convert to other objects that have more meaningful
> capabilities is enough for the standard.  I think the things discussed
> above should be in a separate library that adds functionality onto DTO and
> Converter similar to cats in scala.
> My 2 cents,
>  David Daniel
> On Tue, Aug 16, 2016 at 2:00 AM, David Leangen <osgi@leangen.net> wrote:
>> Hi!
>> The Converter service does a lot of introspection and parsing of the DTO
>> data structure. In many cases, a DTO is a very simple object structure.
>> However, it can also be a very complex structure, too.
>> According to my understanding of the objectives of the Converter, one
>> important goal is to be able to persist data. The idea is that the DTO
>> describes the data, the whole data, and nothing but the data, so help me
>> Uncle. Thus, it is the ideal way to ship the state of the system off to
>> PersistenceLand.
>> I can buy into this vision.
>> If we do buy into this vision, then we may be missing out on a few great
>> opportunities here. When data gets persisted, we often need to understand
>> the relationships between the embedded objects. Or, we may want to be able
>> to create an index on the data. These are a few of the reasons why we would
>> want to have some kind of x-ray vision on the data structure. Since we
>> already go through all the trouble of parsing the data structure in order
>> to convert it, and since this is ~95% of the work, it would be really nice
>> to provide access to this information in order to easily link in services
>> that require this intimate knowledge. Otherwise, all the parsing would have
>> to be done over and over again for each service.
>> I believe that it would only take a few methods to be able to leverage all
>> the parsing work done by the Converter. I can think of:
>>  DataTree Converter.toTree(DTO dto); // DataTre gives a tree view of the
>> structure
>>  Object tree.valueAt(DTO dto, String path); // Dot-separated path value
>> within the tree structure
>>  void tree.set(DTO dto, String path, Object value); // Set the value at
>> the given location in the tree structure
>>  void process(DTO dto, Consumer<?> function); // Visit each node for some
>> kind of processing
>> Those are just some examples. Perhaps a new API would be necessary, but my
>> main point here is that since we are going through all this work of
>> implementing a parser, this is the IDEAL time to create this type of view
>> on the data.
>> wdyt?
>> I can explain further the idea if you like. For now, I just wanted to get
>> a quick feedback to see if there is any openness to this kind of thing.
>> =David

View raw message