hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raimon Bosch <raimon.bo...@gmail.com>
Subject Creating a hive table for a custom log
Date Thu, 01 Sep 2011 14:08:13 GMT

Hi,

I'm trying to create a table similar to apache_log but I'm trying to avoid
to write my own map-reduce task because I don't want to have my HDFS files
twice.

So if you're working with log lines like this:

186.92.134.151 [31/Aug/2011:00:10:41 +0000] "GET
/client/action1/?transaction_id=8002&user_id=871793100001248&ts=1314749223525&item1=271&item2=6045&environment=2
HTTP/1.1"

112.201.65.238 [31/Aug/2011:00:10:41 +0000] "GET
/client/action1/?transaction_id=9002&ts=1314749223525&user_id=9048871793100&item2=6045&item1=271&environment=2
HTTP/1.1"

90.45.198.251 [31/Aug/2011:00:10:41 +0000] "GET
/client/action2/?transaction_id=9022&ts=1314749223525&user_id=9048871793100&item2=6045&item1=271&environment=2
HTTP/1.1"

And having in mind that the parameters could be in different orders. Which
will be the best strategy to create this table? Write my own
org.apache.hadoop.hive.contrib.serde2? Is there any resource already
implemented that I could use to perform this task?

In the end the objective is convert all the parameters in fields and use as
type the "action". With this big table I will be able to perform my queries,
my joins or my views.

Any ideas?

Thanks in Advance,
Raimon Bosch.
-- 
View this message in context: http://old.nabble.com/Creating-a-hive-table-for-a-custom-log-tp32379849p32379849.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.


Mime
View raw message