hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sohan Jain" <sohanj...@fb.com>
Subject Re: Review Request: HIVE-2261: Add API to metastore for table filtering based on table properties
Date Fri, 17 Jun 2011 00:39:44 GMT


> On 2011-06-17 00:13:20, Paul Yang wrote:
> > trunk/metastore/if/hive_metastore.thrift, line 46
> > <https://reviews.apache.org/r/910/diff/1/?file=21192#file21192line46>
> >
> >     Can we rename this to TableQueryFilterType so that it's clear that it's only
used for tables?

Sure thing.


> On 2011-06-17 00:13:20, Paul Yang wrote:
> > trunk/metastore/if/hive_metastore.thrift, line 53
> > <https://reviews.apache.org/r/910/diff/1/?file=21192#file21192line53>
> >
> >     Where is this used?

Ah, that's not supposed to be there; I will remove it.


> On 2011-06-17 00:13:20, Paul Yang wrote:
> > trunk/metastore/if/hive_metastore.thrift, lines 267-270
> > <https://reviews.apache.org/r/910/diff/1/?file=21192#file21192line267>
> >
> >     Hive doesn't really use the retention field. Can you remove operations on this
field from the rest of the diff?

Yep.


> On 2011-06-17 00:13:20, Paul Yang wrote:
> > trunk/metastore/if/hive_metastore.thrift, line 277
> > <https://reviews.apache.org/r/910/diff/1/?file=21192#file21192line277>
> >
> >     The interface is a little odd because we have to use names like 'owner' or 'retention'
in addition to specifying the QueryFilterType. Maybe we should make the field that the QueryFilterType
references be called 'field', so you'd have a filter like 'field = ".*test_user.*"' (for owner)
or 'field > 90' (for retention)

Actually, with the current implementation, the field in the filter string can be arbitrarily
named; the field name is parsed out by the antlr grammar and renamed according to TableQueryFilterType.
 So if TableQueryFilterType is OWNER, then the following filters are equivalent: 
"owner like \".*test.\"", 
"key like \".*test.*\"", 
"field like \".*test.*\"",
etc.


> On 2011-06-17 00:13:20, Paul Yang wrote:
> > trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line
1203
> > <https://reviews.apache.org/r/910/diff/1/?file=21193#file21193line1203>
> >
> >     Style issue, { should be on same line as if

Will fix.


> On 2011-06-17 00:13:20, Paul Yang wrote:
> > trunk/metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java,
lines 189-197
> > <https://reviews.apache.org/r/910/diff/1/?file=21198#file21198line189>
> >
> >     JDO-174 looks like it was fixed a while back - is this still an issue? >
and < may be useful operators for the the parameters field. (e.g. if retention were stored
there instead of the member field)

I agree that the > and < could be pretty useful operators here.  The issue was said
to be resolved over 5 years ago, but surprisingly I think it just closed a few days ago 
https://issues.apache.org/jira/browse/JDO-174?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel#issue-tabs.
 

In any case, comparing the result of a map.get() with anything other than equality throws
the same error as the one described in JDO-174.  Trying to cast the result of map.get() to
a String with ((String) this.parameters.get("keyName") > "value" throws the following seemingly
unresolvable error:

org.datanucleus.store.exceptions.NoTableManagedException: Persistent class "java.lang.String"
has no table in the database, but the operation requires it. Please check the specification
of the MetaData for this class.


- Sohan


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/910/#review855
-----------------------------------------------------------


On 2011-06-16 03:13:24, Sohan Jain wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/910/
> -----------------------------------------------------------
> 
> (Updated 2011-06-16 03:13:24)
> 
> 
> Review request for hive and Paul Yang.
> 
> 
> Summary
> -------
> 
> Create a function listTableNamesByFilter that returns a list of names for tables in a
database that match a certain filter. The syntax of the filter is similar to the one created
by HIVE-1609. You can filter the table list based on owner, retention, or table parameter
key/values. The filtering takes place at the JDO level for efficiency/speed.  Added a QueryFilterType
enum to easily add new filters and separate logic for filtering.  
> 
> Example filter statements include: 
> filterType = QueryFilterType.OWNER; filter = "owner like ".*test_user.*"
> filterType = QueryFilterType.RETENTION; filter = "retention < 90 and retention >
30"
> filterType = QueryFilterType.PARAMS; filter = "numPartitions = \"2\" and  retention_days
= \"30\""
> 
> The filter can currently parse string or integer values, where values interpreted as
strings must be in quotes.
> 
> 
> This addresses bug HIVE-2226.
>     https://issues.apache.org/jira/browse/HIVE-2226
> 
> 
> Diffs
> -----
> 
>   trunk/metastore/if/hive_metastore.thrift 1135227 
>   trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1135227

>   trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
1135227 
>   trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1135227

>   trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1135227

>   trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1135227 
>   trunk/metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java
1135227 
>   trunk/metastore/src/java/org/apache/hadoop/hive/metastore/parser/Filter.g 1135227 
>   trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1135227

> 
> Diff: https://reviews.apache.org/r/910/diff
> 
> 
> Testing
> -------
> 
> Added test cases to TestHiveMetaStore
> 
> 
> Thanks,
> 
> Sohan
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message