metamodel-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tomasz Guziałek <Tomasz.Guzia...@HumanInference.com>
Subject RE: [DISCUSS] Documentation improvements?
Date Wed, 17 Dec 2014 11:33:23 GMT
I registered myself at the Wiki, but I believe I do not have the rights for editing. Henry,
is that you who can fix that for me? My username is "tomaszguzialek". Thank you!

Best regards,
Tomasz

-----Original Message-----
From: Kasper Sørensen [mailto:i.am.kasper.sorensen@gmail.com] 
Sent: Tuesday, December 16, 2014 20:04
To: dev@metamodel.apache.org
Subject: Re: [DISCUSS] Documentation improvements?

Hi Tomasz,

Good info too! I like the analogies to existing implementations - I can imagine these are
valueable examples for implementors.

Maybe you could start a page on the MM wiki about this? To begin with it could even be just
a dump of that email. We can adjust and refine it easily when it's in that wiki format I think.

I would call this page something like "GettingStartedNewConnectors".

Best regards,
Kasper

2014-12-15 12:48 GMT+01:00 Tomasz Guziałek <
Tomasz.Guzialek@humaninference.com>:
>
> Hello Alberto and others,
>
> My first thought also was around building new connectors - what is the 
> minimum I need to provide as a developer to query a new datastore and 
> what I will get "for free" from MetaModel. My (quick sketch of) 
> step-by-step guide would look like this:
>
> 1) Create a new XXXDataContext class that extends 
> QueryPostProcessDataContext (to get functionalities for free) that 
> holds a reference to a native API object. Taking existing examples:
>         CouchDbDataContext holds Ektorp' CouchDbInstance
>         HBaseDataContext holds HTablePool
>         CsvDataContext holds a handle to a file (through MetaModel's 
> Resource class, to be precise)
> 2) Implement materializeMainSchemaTable to fetch the data that is 
> going to represent the table, using the native API. Existing examples:
>         CouchDbDataContext fetches a view will all docs through 
> Ektorp's CouchDbInstance
>         HBaseDataContext creates a Scan of a table through HBaseTablePool
>         CsvDataContext reads the whole file using our file handle
> 3) Return a new XXXDataSet instance with the native result passed as a 
> parameter. XXXDataSet class takes the native result object and 
> translates it to the MetaModel's Row objects.
>         CouchDbDataSet will parse the JSON document and instantiate 
> MetaModel's Row object with this data
>         HBaseDataSet extracts the values from the Scan and 
> instantiates MetaModel's Row object with the values
>         CsvDataSet translates a line from a CSV file into MetaModel's 
> Row object
>
> This is the minimum that needs to be implemented. While we have the 
> native result translated to MetaModel's Row objects, selecting 
> specific columns, filtering and so on we get for free from 
> QueryPostProcessDataContext class that we subclassed. Of course, it is 
> not the most performant way of querying, for example doing a full scan 
> to get a single row by its primary key is optimized in many databases. 
> Delegating primary key lookups, count queries, queries with simple 
> WHERE clauses to the native API instead of post-processing it in Java 
> is the next step for a developer of a new MetaModel connector.
>
> I would benefit a lot from such a guide at the beginning of my journey 
> with MetaModel...
>
> Best regards,
> Tomasz
>
>
> -----Original Message-----
> From: Alberto Rodriguez [mailto:ardlema@gmail.com]
> Sent: Friday, December 12, 2014 11:42
> To: dev@metamodel.apache.org
> Cc: juan.rodriguez.hortala@gmail.com
> Subject: Re: [DISCUSS] Documentation improvements?
>
> Hi all,
>
> When I first came across the project I wasn't sure how and where MM "plays"
> with the data. I mean, for the different datasources I didn't know 
> whether MM was fetching the data in memory and then performing joins, 
> selects and so on or was delegating this behaviour to the native 
> drivers. The table that you guys put together in the wiki: 
> QueryExecutionStrategies < 
> https://wiki.apache.org/metamodel/QueryExecutionStrategies> is quite clarifying.
>
> In my particular case, as we needed new connectors that wasn't 
> implemented yet and decided to contribute to the project implementing 
> them, I was "fighting" with the code from day 0 and I must say that 
> the code is pretty self-explanatory. Kasper has been done a great job 
> and if you have some Java experience you might have a new connector 
> ready-to-use within few days. I think we should make an effort and try 
> to share this with the world, MM is great as it is now but I think one 
> of its main powerful features is that you can create your own 
> connectors without much effort. It might be a good idea to create a 
> section on the website like "Create your own connectors" and a guide to implement them.
What do you think?
>
> I would also be more than happy to help in any decision we take 
> regarding the documentation.
>
> Kind regards,
>
> 2014-12-12 9:32 GMT+01:00 Kasper Sørensen 
> <i.am.kasper.sorensen@gmail.com
> >:
> >
> > Hi Juan,
> >
> > Thanks for that feedback. I think it helps a lot to see what 
> > relevant and good information we already have (some of it on third 
> > party sites
> > - we should look to integrate it some more) and also what we should 
> > improve (the javadoc thing is such a simple thing - I'll create a 
> > JIRA issue about doing it ASAP).
> >
> > Kasper
> >
> > 2014-12-11 20:36 GMT+01:00 Juan Rodríguez Hortalá <
> > juan.rodriguez.hortala@gmail.com>:
> > >
> > > Hi all,
> > >
> > > I don't have a lot of experience with Metamodel, I did my first
> > experiments
> > > with it in the capstone project for a MSc, and then I used it in 
> > > my work combined with Apache Sqoop to import data from a 
> > > relational database,
> > where
> > > Metamodel was in charge of obtaining the data schema. But maybe my
> > limited
> > > experience could be of use for answering this question.
> > > I think I discovered Metamodel when searching the internet for a 
> > > way to connect a BI visualization tool with HBase. As the starting 
> > > points I
> > used:
> > >
> > >  * This post
> > > http://www.infoq.com/articles/apache-incubator-metamodel by 
> > > Kasper, which I think is a pretty good quick start, with several code snippets.
> > >  * This other post
> > > http://kasper.eobjects.org/2013/06/introducing-apache-metamodel.ht
> > > ml by Kasper, which is more about concepts and the origin of 
> > > Metamodel.
> > >  * The code examples from the site 
> > > http://metamodel.apache.org/#examples
> > > also help.
> > >  * I also generated the Javadoc documentation, and put it in my 
> > > github
> > for
> > > easy access http://juanrh.github.io/doc/ApacheMetamodel/apidocs/ 
> > > (I hope that is ok for you, by the way!).
> > >
> > > As I said, I haven't done anything much sophisticated, but I do 
> > > think the javadoc documentation and API structure is pretty 
> > > intuitive. What I miss most is documentation about the design and 
> > > architecture so I can
> > understand
> > > the implications in performance of using Metamodel in different 
> > > ways. I think that is important because that could determine 
> > > whether Metamodel could be applied or not for a particular 
> > > problem. But these are just my personal subjective impressions.
> > >
> > > TL;DR: In my opinion the blog post 
> > > http://www.infoq.com/articles/apache-incubator-metamodel is a good 
> > > candidate for the "getting started" guide for Metamodel. Also an 
> > > updated version of the javadoc documentation hosted in the site 
> > > would
> be nice.
> > >
> > > Greetings,
> > >
> > > Juan
> > >
> > >
> > > 2014-12-11 18:47 GMT+01:00 Kasper Sørensen <
> > i.am.kasper.sorensen@gmail.com
> > > >:
> > >
> > > > Hi all,
> > > >
> > > > Discussed with a colleague the other day the perception of 
> > > > MetaModel
> > when
> > > > coming to the project as a newcomer (new in our company and 
> > > > approaching
> > > the
> > > > community). He mentioned that the documentation was too sparse 
> > > > for him
> > to
> > > > get a good insight by just browsing our website and wiki. A 
> > > > proper
> > > getting
> > > > started and better reference documentation would have helped him 
> > > > a lot
> > -
> > > > and improved his overall impression (which has improved as he 
> > > > started
> > > using
> > > > it, but wasn't immediately too good).
> > > >
> > > > I think that's pretty good feedback - documentation is important 
> > > > and we have not spent a lot of time on it I guess.
> > > >
> > > > So are there any "low hanging fruit" we can start by grabbing? 
> > > > And what should be the more dedicated efforts that we go for here?
> > > >
> > > > I was thinking of at least publishing our javadocs and linking 
> > > > to it
> > from
> > > > our website. That should help a little, but is obviously not 
> > > > what he
> > was
> > > > mentioning. A proper "getting started" guide would also be great 
> > > > I
> > think.
> > > > I'm curious which aspects we should stress then. Especially 
> > > > looking at people not from Human Inference / Neopost .... How 
> > > > did you learn about MetaModel and which questions did you have 
> > > > (still
> > > > have?) and how did
> > you
> > > > get started.
> > > >
> > > > Best regards,
> > > > Kasper
> > > >
> > >
> >
>
Mime
View raw message