hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ruslan Al-fakikh" <ruslan.al-fak...@jalent.ru>
Subject RE: Continuous log analysis requires 'dynamic' partitions, is that possible?
Date Tue, 24 Jul 2012 16:35:59 GMT
If you are not using Amazon take a look at this:

https://issues.apache.org/jira/browse/HIVE-874

 

Ruslan

 

From: Puneet Khatod [mailto:puneet.khatod@tavant.com] 
Sent: Tuesday, July 24, 2012 8:32 PM
To: user@hive.apache.org
Subject: RE: Continuous log analysis requires 'dynamic' partitions, is that
possible?

 

If you are using Amazon (AWS), you can use 'recover partitions' to enable
all top level partitions.

This will add required dynamicity.

 

Regards,

Puneet Khatod

 

From: Bertrand Dechoux [mailto:dechouxb@gmail.com] 
Sent: 24 July 2012 21:15
To: user@hive.apache.org
Subject: Continuous log analysis requires 'dynamic' partitions, is that
possible?

 

Hi,

Let's say logs are stored inside hdfs using the following file tree
/<logtype>/<month>/<day>.
So for apache, that would be :
/apache/01/01
/apache/01/02
...
/apache/02/01
...

I would like to know how to define a table for this information. I found out
that the table should be external and should be using partitions.
However, I did not found any way to dynamically create the partitions. Is
there no automatic way to define them?
In that case, the partition 'template' would be <month>/<day> with the root
being apache.

I know how to 'hack a fix' : create a script which would generate all the
"add partition statement" and run the resulting statements without caring
about the results because partitions may not exist or may already have been
added. Better, I could parse the result of 'show partition' for the table
and run only the relevant statement but it still feels like a hack.

Is there any clean way to do it?

Regards,

Bertrand Dechoux

Any comments or statements made in this email are not necessarily those of
Tavant Technologies.
The information transmitted is intended only for the person or entity to
which it is addressed and may 
contain confidential and/or privileged material. If you have received this
in error, please contact the 
sender and delete the material from any computer. All e-mails sent from or
to Tavant Technologies 
may be subject to our monitoring procedures.


Mime
View raw message