hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "HBase Review Board (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-1518) context_ngrams() UDAF for estimating top-k contextual n-grams
Date Fri, 13 Aug 2010 21:34:16 GMT

    [ https://issues.apache.org/jira/browse/HIVE-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898425#action_12898425
] 

HBase Review Board commented on HIVE-1518:
------------------------------------------

Message from: "John Sichi" <jsichi@facebook.com>

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/644/#review902
-----------------------------------------------------------



http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java
<http://review.cloudera.org/r/644/#comment2936>

    Shouldn't name="context_ngrams"?
    



http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java
<http://review.cloudera.org/r/644/#comment2937>

    Here and elsewhere, why do you cast to ArrayList?  Can't you just use the List interface?
 I don't think Hive makes any guarantee that an ArrayList will always be returned, even if
that is currently the case.
    



http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NGramEstimator.java
<http://review.cloudera.org/r/644/#comment2938>

    (I missed this in an earlier review):  can you use generics here to specify the HashMap<K,V>
types and avoid so much casting?



http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NGramEstimator.java
<http://review.cloudera.org/r/644/#comment2939>

    specify Comparator<Double> to avoid casting


- John





> context_ngrams() UDAF for estimating top-k contextual n-grams
> -------------------------------------------------------------
>
>                 Key: HIVE-1518
>                 URL: https://issues.apache.org/jira/browse/HIVE-1518
>             Project: Hadoop Hive
>          Issue Type: New Feature
>          Components: Query Processor
>    Affects Versions: 0.6.0
>            Reporter: Mayank Lahiri
>            Assignee: Mayank Lahiri
>             Fix For: 0.7.0
>
>         Attachments: HIVE-1518.1.patch, HIVE-1518.2.patch
>
>
> Create a new context_ngrams() function that generalizes the ngrams() UDAF to allow the
user to specify context around n-grams. The analogy is "fill-in-the-blanks", and is best illustrated
with an example:
> SELECT context_ngrams(sentences(tweets), array("i", "love", null), 300) FROM twitter;
> will estimate the top-300 words that follow the phrase "i love" in a database of tweets.
The position of the null(s) specifies where to generate the n-gram from, and can be placed
anywhere. For example:
> SELECT context_ngrams(sentences(tweets), array("i", "love", null, "but", "hate", null),
300) FROM twitter;
> will estimate the top-300 word-pairs that fill in the blanks specified by null.
> POSSIBLE USES:
> 1. Pre-computing search lookaheads
> 2. Sentiment analysis for products or entities -- e.g., querying with context = array("twitter",
"is", null)
> 3. Navigation path analysis in URL databases

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message