hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zoltan Haindrich (JIRA)" <>
Subject [jira] [Created] (HIVE-18051) qfiles: dataset support
Date Mon, 13 Nov 2017 13:07:00 GMT
Zoltan Haindrich created HIVE-18051:

             Summary: qfiles: dataset support
                 Key: HIVE-18051
             Project: Hive
          Issue Type: Improvement
          Components: Testing Infrastructure
            Reporter: Zoltan Haindrich

it would be great to have some kind of test dataset support; currently there is the {{q_test_init.sql}}
which is quite large; and I'm often override it with an invalid string; because I write independent
qtests most of the time - and the load of {{src}} and other tables are just a waste of time
for me ; not to mention that the loading of those tables may also trigger breakpoints - which
is a bit annoying.

Most of the tests are "only" using the {{src}} table and possibly 2 others; however the main
init script contains a bunch of tables - meanwhile there are quite few other tests which could
possibly also benefit from a more general feature; for example the creation of {{bucket_small}}
is present in 20 q files.

the proposal would be to enable the qfiles to be annotated with metadata like datasets:
--! qt:dataset:src,bucket_small

proposal for storing a dataset:

* the loader script would be at: {{data/datasets/__NAME__/load.hive.sql}}
* the table data could be stored under that location

a draft about this; and other qfiles related ideas:

This message was sent by Atlassian JIRA

View raw message