accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Corey Nolet <cjno...@gmail.com>
Subject Re: Q4A Project
Date Tue, 28 Apr 2015 01:48:13 GMT
I'm always looking for places to help out and integrate/share designs &
ideas. I look forward to chatting with you about Q4A at the hackathon
tomorrow!

Have you, by chance, seen the Spark SQL adapter for the Accumulo Recipes
Event & Entity Stores [1]? At the very least, it's a good example of using
Spark's SQL abstraction over Accumulo. As Mike Drob pointed out, Spark SQL
has a pretty robust query planning / optimization layer. The Event/Entity
stores in Accumulo Recipes also have a pluggable query
planning/optimization layer.


[1]
https://github.com/calrissian/accumulo-recipes/blob/master/thirdparty/spark/src/test/scala/org/calrissian/accumulorecipes/spark/sql/EventStoreCatalystTest.scala

On Mon, Apr 27, 2015 at 9:38 PM, Mike Drob <madrob@cloudera.com> wrote:

> Andrew,
>
> This is a cool thing to work on, I hope you have great success!
>
> A couple of questions about the motivations behind this, if you don't mind
> -
> - There are several SQL implementations already in the Hadoop ecosystem.
> In what ways do you expect this to improve upon
> Hive/Impala/Phoenix/Presto/Spark SQL? I haven't looked at the code, so it
> is quite possible you're already using one of those technologies.
> - In a conversation with some HP engineers earlier this year, they
> mentioned that building a SQL-92 layer is the easy part, and that a mature
> optimization engine is the really hard part. This is where Oracle may still
> be leaps and bounds ahead of its nearest competitors. Do you have plans for
> a query planner? If not, you might be back to writing MapReduce jobs sooner
> than you think.
>
> Look forward to seeing more!
>
> Mike
>
> On Mon, Apr 27, 2015 at 7:37 PM, Andrew Wells <awells@clearedgeit.com>
> wrote:
>
>> I have been working on a project, tentatively called Q4A (Query for
>> Accumulo). Another possible name is ASQ (Accumulo Streaming Query) [discus].
>>
>> This is a streaming query as the query is completed via a stream, should
>> never group data in memory. To batch, intermediate results would be written
>> back to Accumulo temporarily.
>>
>>
>> The *primary goal* is to have a complete SQL implementation native to
>> Accumulo.
>>
>> *Why do this?*
>> I am getting tired of writing bad java code to query a database. I would
>> rather write bad SQL code. Also, people should be able to get queries out
>> faster and it shouldn't take a developer.
>>
>>
>> *Native To Accumulo*:
>>
>>    - There should be no special format to read a database created by Q4A
>>    - There should be no special format for Q4A to query a table
>>    - All tables are tables available to Q4A
>>    - Any special tables, are stored away from the users databases
>>    (indexes, column definitions, etc)
>>
>> *Other Goals*:
>>
>>    - Implement the entire SQL definition (currently all of SQLite)
>>    - Create JDBC Driver/Server
>>    - Push down Expressions to the Tablet Servers
>>    - Install-less queries, use Q4A jar directly against any Accumulo
>>    Cluster ( less push-down expressions)
>>    - documentation :o
>>    - testing ;)
>>
>> *Does it work?*
>> Not yet, the project is still a work in progress. and I will be working
>> on it at the Accumulo Summit this year. Progress is slow as I am getting
>> married in about a month and some change.
>>
>> *Questions:*
>> If you have questions about Q4A as here, I will be at the Accumulo Summit
>> @ ClearEdgeIT Table and Hackathon.
>>
>> *WHERE IS TEH LINK?!1!*
>> Oh.... here: https://github.com/agwells0714/q4a
>>
>> --
>> *Andrew George Wells*
>> *Software Engineer*
>> *awells@clearedgeit.com <awells@clearedgeit.com>*
>>
>>
>

Mime
View raw message