Mailing-List: contact hive-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hive-dev@hadoop.apache.org
Message-ID: <23100325.340291281733937279.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 17:12:17 -0400 (EDT)
From: "John Sichi (JIRA)" <jira@apache.org>
To: hive-dev@hadoop.apache.org
Subject: [jira] Updated: (HIVE-1518) context_ngrams() UDAF for estimating
 top-k contextual n-grams
In-Reply-To: <2625867.207331281137296189.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HIVE-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

John Sichi updated HIVE-1518:
-----------------------------

    Status: Open  (was: Patch Available)

Submitted a review here:

https://review.cloudera.org/r/644/

Some of my comments are on existing code which is being moved as part of this patch; consider them retroactive since I should have made them on the original ngrams patch.

In general, see if you can use generics for collections wherever possible.


> context_ngrams() UDAF for estimating top-k contextual n-grams
> -------------------------------------------------------------
>
>                 Key: HIVE-1518
>                 URL: https://issues.apache.org/jira/browse/HIVE-1518
>             Project: Hadoop Hive
>          Issue Type: New Feature
>          Components: Query Processor
>    Affects Versions: 0.6.0
>            Reporter: Mayank Lahiri
>            Assignee: Mayank Lahiri
>             Fix For: 0.7.0
>
>         Attachments: HIVE-1518.1.patch, HIVE-1518.2.patch
>
>
> Create a new context_ngrams() function that generalizes the ngrams() UDAF to allow the user to specify context around n-grams. The analogy is "fill-in-the-blanks", and is best illustrated with an example:
> SELECT context_ngrams(sentences(tweets), array("i", "love", null), 300) FROM twitter;
> will estimate the top-300 words that follow the phrase "i love" in a database of tweets. The position of the null(s) specifies where to generate the n-gram from, and can be placed anywhere. For example:
> SELECT context_ngrams(sentences(tweets), array("i", "love", null, "but", "hate", null), 300) FROM twitter;
> will estimate the top-300 word-pairs that fill in the blanks specified by null.
> POSSIBLE USES:
> 1. Pre-computing search lookaheads
> 2. Sentiment analysis for products or entities -- e.g., querying with context = array("twitter", "is", null)
> 3. Navigation path analysis in URL databases

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.