hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luis Ramos (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-645) A UDF that can export data to JDBC databases.
Date Mon, 24 Aug 2009 22:29:59 GMT

    [ https://issues.apache.org/jira/browse/HIVE-645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747110#action_12747110
] 

Luis Ramos commented on HIVE-645:
---------------------------------

Edward,

Adding the jar got rid of the Failed for ExecDriver, thanks, HOWEVER, I still don't see any
difference on mysql (it doesn't insert or anything).

I did these steps:
1. add jar build/contrib/hive_contrib.jar;
2. CREATE TEMPORARY FUNCTION dboutput AS 'org.apache.hadoop.hive.contrib.genericudf.example.GenericUDFDBOutput';
3. SELECT dboutput('jdbc:mysql://localhost:3306/analytics','root','password','INSERT INTO
status_count(date,status,count) VALUES (?,?,?)',day,httpstatus,count(1)) FROM web_logs GROUP
BY day, httpstatus;

On my hive.log I see:
WARN  mapred.JobClient (JobClient.java:configureCommandLineOptions(510)) - Use GenericOptionsParser
for parsing the arguments. Applications should implement Tool for the same.

Output to console is:
$ hive -hiveconf hive.root.logger=INFO,console
hive> SELECT dboutput('jdbc:mysql://localhost:3306/analytics','root','password','INSERT
INTO status_count(date,status,count) VALUES (?,?,?)',day,httpstatus,count(1)) FROM web_logs
GROUP BY day, httpstatus;
09/08/24 22:21:00 INFO parse.ParseDriver: Parsing command: SELECT dboutput('jdbc:mysql://localhost:3306/analytics','root','password','INSERT
INTO status_count(date,status,count) VALUES (?,?,?)',day,httpstatus,count(1)) FROM web_logs
GROUP BY day, httpstatus
09/08/24 22:21:00 INFO parse.ParseDriver: Parse Completed
09/08/24 22:21:00 INFO parse.SemanticAnalyzer: Starting Semantic Analysis
09/08/24 22:21:00 INFO parse.SemanticAnalyzer: Completed phase 1 of Semantic Analysis
09/08/24 22:21:00 INFO parse.SemanticAnalyzer: Get metadata for source tables
09/08/24 22:21:00 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=web_logs
09/08/24 22:21:00 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
09/08/24 22:21:00 INFO metastore.ObjectStore: ObjectStore, initialize called
09/08/24 22:21:00 INFO metastore.ObjectStore: Initialized ObjectStore
09/08/24 22:21:00 INFO hive.log: DDL: struct web_logs { string day, string time, string url,
string httpstatus}
09/08/24 22:21:00 INFO parse.SemanticAnalyzer: Get metadata for subqueries
09/08/24 22:21:00 INFO parse.SemanticAnalyzer: Get metadata for destination tables
09/08/24 22:21:00 INFO parse.SemanticAnalyzer: Completed getting MetaData in Semantic Analysis
09/08/24 22:21:00 INFO ppd.OpProcFactory: Processing for FS(34)
09/08/24 22:21:00 INFO ppd.OpProcFactory: Processing for SEL(33)
09/08/24 22:21:00 INFO ppd.OpProcFactory: Processing for GBY(32)
09/08/24 22:21:00 INFO ppd.OpProcFactory: Processing for RS(31)
09/08/24 22:21:00 INFO ppd.OpProcFactory: Processing for GBY(30)
09/08/24 22:21:00 INFO ppd.OpProcFactory: Processing for SEL(29)
09/08/24 22:21:00 INFO ppd.OpProcFactory: Processing for TS(28)
09/08/24 22:21:00 INFO parse.SemanticAnalyzer: Completed partition pruning
09/08/24 22:21:00 INFO parse.SemanticAnalyzer: Completed sample pruning
09/08/24 22:21:00 INFO parse.SemanticAnalyzer: Completed plan generation
09/08/24 22:21:00 INFO ql.Driver: Semantic Analysis Completed
09/08/24 22:21:00 INFO ql.Driver: Starting command: SELECT dboutput('jdbc:mysql://localhost:3306/analytics','root','password','INSERT
INTO status_count(date,status,count) VALUES (?,?,?)',day,httpstatus,count(1)) FROM web_logs
GROUP BY day, httpstatus
Total MapReduce jobs = 1
09/08/24 22:21:00 INFO ql.Driver: Total MapReduce jobs = 1
09/08/24 22:21:00 INFO exec.ExecDriver: BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=108
Number of reduce tasks not specified. Estimated from input data size: 1
09/08/24 22:21:00 INFO exec.ExecDriver: Number of reduce tasks not specified. Estimated from
input data size: 1
In order to change the average load for a reducer (in bytes):
09/08/24 22:21:00 INFO exec.ExecDriver: In order to change the average load for a reducer
(in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
09/08/24 22:21:00 INFO exec.ExecDriver:   set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
09/08/24 22:21:00 INFO exec.ExecDriver: In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
09/08/24 22:21:00 INFO exec.ExecDriver:   set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
09/08/24 22:21:00 INFO exec.ExecDriver: In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
09/08/24 22:21:00 INFO exec.ExecDriver:   set mapred.reduce.tasks=<number>
09/08/24 22:21:00 INFO exec.ExecDriver: adding libjars: file:///home/hadoop/hive/build/contrib/hive_contrib.jar
09/08/24 22:21:00 INFO exec.ExecDriver: Processing alias web_logs
09/08/24 22:21:00 INFO exec.ExecDriver: Adding input file hdfs://verizonmaster:50120/user/hive/warehouse/web_logs
09/08/24 22:21:00 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments.
Applications should implement Tool for the same.
09/08/24 22:21:00 INFO mapred.FileInputFormat: Total input paths to process : 1
09/08/24 22:21:00 INFO mapred.FileInputFormat: Total input paths to process : 1
Starting Job = job_200908201702_0102, Tracking URL = http://verizonmaster:50030/jobdetails.jsp?jobid=job_200908201702_0102
09/08/24 22:21:01 INFO exec.ExecDriver: Starting Job = job_200908201702_0102, Tracking URL
= http://verizonmaster:50030/jobdetails.jsp?jobid=job_200908201702_0102
Kill Command = /home/hadoop/hadoop-0.18.3/bin/../bin/hadoop job  -Dmapred.job.tracker=verizonmaster:50121
-kill job_200908201702_0102
09/08/24 22:21:01 INFO exec.ExecDriver: Kill Command = /home/hadoop/hadoop-0.18.3/bin/../bin/hadoop
job  -Dmapred.job.tracker=verizonmaster:50121 -kill job_200908201702_0102
2009-08-24 10:21:02,237 map = 0%,  reduce = 0%
09/08/24 22:21:02 INFO exec.ExecDriver: 2009-08-24 10:21:02,237 map = 0%,  reduce = 0%
2009-08-24 10:21:08,253 map = 50%,  reduce = 50%
09/08/24 22:21:08 INFO exec.ExecDriver: 2009-08-24 10:21:08,253 map = 50%,  reduce = 50%
2009-08-24 10:21:10,265 map = 100%,  reduce = 100%
09/08/24 22:21:10 INFO exec.ExecDriver: 2009-08-24 10:21:10,265 map = 100%,  reduce = 100%
Ended Job = job_200908201702_0102
09/08/24 22:21:21 INFO exec.ExecDriver: Ended Job = job_200908201702_0102
09/08/24 22:21:21 INFO exec.FileSinkOperator: Moving tmp dir: hdfs://verizonmaster:50120/tmp/hive-hadoop/942129758/_tmp.10001
to: hdfs://verizonmaster:50120/tmp/hive-hadoop/942129758/_tmp.10001.intermediate
09/08/24 22:21:21 INFO exec.FileSinkOperator: Moving tmp dir: hdfs://verizonmaster:50120/tmp/hive-hadoop/942129758/_tmp.10001.intermediate
to: hdfs://verizonmaster:50120/tmp/hive-hadoop/942129758/10001
OK
09/08/24 22:21:21 INFO ql.Driver: OK
09/08/24 22:21:21 INFO mapred.FileInputFormat: Total input paths to process : 1
09/08/24 22:21:21 INFO mapred.FileInputFormat: Total input paths to process : 1
1
1
Time taken: 21.03 seconds

I'm going to try other test cases and different data, I tried sending constants into status_count
but that didn't work either. Is there other jar's I might be missing.

Thank You.

> A UDF that can export data to JDBC databases.
> ---------------------------------------------
>
>                 Key: HIVE-645
>                 URL: https://issues.apache.org/jira/browse/HIVE-645
>             Project: Hadoop Hive
>          Issue Type: New Feature
>          Components: Contrib
>            Reporter: Edward Capriolo
>            Assignee: Edward Capriolo
>            Priority: Minor
>         Attachments: hive-645-2.patch, hive-645-3.patch, hive-645-4.patch, hive-645-5.patch,
hive-645.patch
>
>
> A UDF that can export data to JDBC databases.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message