falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shwetha GS <shwetha...@inmobi.com>
Subject Re: Partition-import to HCatalog/Hive
Date Wed, 11 Jun 2014 04:59:14 GMT
Hi Mithun,

1. Table export is done using hive export command which is part of hive
action in oozie replication workflow:
https://github.com/apache/incubator-falcon/blob/master/feed/src/main/resources/config/workflow/falcon-table-export.hql

2. Yes, falcon assumes that hive at source and target are compatible. Do
you see any issues?

-Shwetha


On Wed, Jun 11, 2014 at 1:59 AM, Mithun Radhakrishnan <
mithun.radhakrishnan@yahoo.com.invalid> wrote:

> Greetings, Falcon-dev.
>
> I've a n00b question about Falcon's support for HCatalog partition-import.
> My (incomplete) understanding is that the implementation copies data
> alongside the serialized metadata, and resolves the partition-schema on the
> target cluster.
>
> 1. I couldn't find the Falcon code that exports the HCat/Hive metadata to
> HDFS. I expected that org.apache.hadoop.hive.ql.parse.EximUtil might be
> used for this, but there's no reference to this class in the Falcon master
> branch. Might I please enquire where/how that's done? A pointer to code
> would be ideal, thanks.
>
> 2. The table/partition metadata might currently be serialized to HDFS in
> thrift (I'll have to check). Does Falcon currently assume that the Hive
> versions running on the source and target clusters are compatible? (i.e.
> That the metadata can be imported on the target?)
>
> Thanks,
> Mithun

-- 
_____________________________________________________________
The information contained in this communication is intended solely for the 
use of the individual or entity to whom it is addressed and others 
authorized to receive it. It may contain confidential or legally privileged 
information. If you are not the intended recipient you are hereby notified 
that any disclosure, copying, distribution or taking any action in reliance 
on the contents of this information is strictly prohibited and may be 
unlawful. If you have received this communication in error, please notify 
us immediately by responding to this email and then delete it from your 
system. The firm is neither liable for the proper and complete transmission 
of the information contained in this communication nor for any delay in its 
receipt.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message