hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bertrand Dechoux <>
Subject Continuous log analysis requires 'dynamic' partitions, is that possible?
Date Tue, 24 Jul 2012 15:45:18 GMT

Let's say logs are stored inside hdfs using the following file tree
So for apache, that would be :

I would like to know how to define a table for this information. I found
out that the table should be external and should be using partitions.
However, I did not found any way to dynamically create the partitions. Is
there no automatic way to define them?
In that case, the partition 'template' would be <month>/<day> with the root
being apache.

I know how to 'hack a fix' : create a script which would generate all the
"add partition statement" and run the resulting statements without caring
about the results because partitions may not exist or may already have been
added. Better, I could parse the result of 'show partition' for the table
and run only the relevant statement but it still feels like a hack.

Is there any clean way to do it?


Bertrand Dechoux

View raw message