datafu-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eyal Allweil (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DATAFU-127) New macro - samply by keys
Date Thu, 22 Mar 2018 12:43:00 GMT

    [ https://issues.apache.org/jira/browse/DATAFU-127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409458#comment-16409458
] 

Eyal Allweil commented on DATAFU-127:
-------------------------------------

The reason I used (and prefer) _sample_ over _filter_ is that this macro really is meant for
making a sample. After all, filtering by keys it's basically just doing an inner join, you
don't need a macro for that. Also, the macro preserves all the fields that exist in the original,
which is usually a waste. We use it for preparing samples of large tables to use afterwards
in development or CI - which fields will be used isn't necessarily known or going to remain
the same over time.

> New macro - samply by keys
> --------------------------
>
>                 Key: DATAFU-127
>                 URL: https://issues.apache.org/jira/browse/DATAFU-127
>             Project: DataFu
>          Issue Type: New Feature
>            Reporter: Eyal Allweil
>            Assignee: Eyal Allweil
>            Priority: Major
>              Labels: macro
>         Attachments: DATAFU-127.patch
>
>
> Two macros that return a sample of a larger table based on a list of keys, with the schema
of the larger table. One of the macros filters by dates, the other doesn't.
> If there are multiple rows with a key that appears in the key list, all of them will
be returned (no deduplication is done). The results are returned ordered by the key field
in a single file.
> The implementation uses a replicated join for efficiency, but this means the key list
shouldn't be too large as to not fit in memory.
> The first macro's definition looks as follows:
> DEFINE sample_by_keys(table, sample_set, join_key_table, join_key_sample) returns out
{
> - table_name 				- table name to sample
> - sample_set 				- a set of keys
> - join_key_table 			- join column name in the table
> - join_key_sample 			- join column name in the sample



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message