hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward Capriolo (JIRA)" <>
Subject [jira] Commented: (HIVE-645) A UDF that can export data to JDBC databases.
Date Thu, 06 Aug 2009 23:33:14 GMT


Edward Capriolo commented on HIVE-645:

My jiras got a lot of attention today. I feel like the most popular guy on the internet. :)
There may be some syntax inaccuracy here but I this is my use case.

As input I have a hive table with weblogs. It may be partitioned or not.
hive_table web_logs
2009-06-04,05:49:00,/index.jsp, 200
2009-06-04,05:50:00,/index.jsp, 200
2009-06-04,05:49:00,/indexsfg.jsp, 404

I wish to produce a report of status codes for day
2009-06-04, 200,2 
2009-06-04, 404,1 

I create a mysql table for this data
create table status_count(
string date,
int status,
int count,
primary key (date,status)

select dboutput(
'insert into status_count (date,status,count) values (?,?,?)', 
date,httpstatus, count(1) ) from web_logs group by date,httpstatus;

In my case mysql primary key constraints deal with failed mappers/reducers. After all, if
a mapper runs again it will try to insert again and fail. Not a problem. In most cases the
data hive is dumping to mysql is relational and should have a primary key. 

If the data does not have a primary key then yes dboutput() will not function correctly with
failed mappers/reducers, but its intented use case would be to push a summary at a mysql database
possibly 100 - 100,000 rows. 

> A UDF that can export data to JDBC databases.
> ---------------------------------------------
>                 Key: HIVE-645
>                 URL:
>             Project: Hadoop Hive
>          Issue Type: New Feature
>          Components: Query Processor
>            Reporter: Edward Capriolo
>            Assignee: Edward Capriolo
>            Priority: Minor
>         Attachments: hive-645-2.patch, hive-645.patch
> A UDF that can export data to JDBC databases.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message