tajo-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyunsik Choi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TAJO-711) Add Avro storage support
Date Wed, 09 Apr 2014 09:01:31 GMT

    [ https://issues.apache.org/jira/browse/TAJO-711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13963946#comment-13963946

Hyunsik Choi commented on TAJO-711:

The patch is very straightforward. Your on-going work looks great to me.

As you aksed in TAJO-710, we firstly need to consider how to specify nested schema in our
DDL statements. I'll also spend some of my own time on the consideration of DDL statement.

The ideas you mentioned look really interesting. Avro schemas usually are too long to be inserted
into DDL statements. {{avro.schema.url}} looks helpful and seems to very cool idea. When it
comes to per-partition properties, currently, Tajo does not store the partition entries in
catalog. For each query on partitioned tables, Tajo traverses matched partition directories
in HDFS according to partition predicates. We should change this part to use partition entries
stored in catalog (i.e., RDBMS), and also should add 'ALTER TABLE ADD/DROP PARTITION' statements.
So, a partition does not have table properties. With the partition improvement work, we also
need to allow each partition to have physical properties. I've created the Jira issue (TAJO-744)
for them.

>From your idea about schema evolving, I also got some rough idea about some different
kind of table to explicitly support the schema evolving for accumulated historical partitions
for long time. If it is necessary, it would be nice to consider this kind of table.

> Add Avro storage support
> ------------------------
>                 Key: TAJO-711
>                 URL: https://issues.apache.org/jira/browse/TAJO-711
>             Project: Tajo
>          Issue Type: New Feature
>            Reporter: David Chen
>            Assignee: David Chen
>         Attachments: TAJO-711.patch
> Add {{FileScanner}} and {{FileAppender}} for reading from and writing to Avro.

This message was sent by Atlassian JIRA

View raw message