crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Ortiz <dpo5...@gmail.com>
Subject Re: Reading Hive Tables into PCollection
Date Fri, 29 Jan 2016 15:19:32 GMT
http://hortonworks.com/blog/using-orcfile-cascading-apache-crunch/

Here's the java excerpt from that article to read into Avro class (I'm
assuming).

[code language=”Java”]
// Read an ORCFile using reflection-based serialization (slowest):
OrcFileSource<Person> source = new OrcFileSource<Person>(new
Path(inputPath), \
Orcs.reflection(Person.class));
PCollection<Person> persons = pipeline.read(source);

On Fri, Jan 29, 2016 at 10:17 AM Robinson, Landon - Landon <
landon.t.robinson@lowes.com> wrote:

> Orc format.
> ---------------------------------------------------------------------------
> Landon Robinson
> Big Data & Hadoop Engineer
> IT Business Intelligence, Lowe’s Companies Inc.
> ---------------------------------------------------------------------------
>
> From: David Ortiz <dpo5003@gmail.com>
> Reply-To: Apache Crunch Mailing List <user@crunch.apache.org>
> Date: Thursday, January 28, 2016 at 1:22 PM
> To: Apache Crunch Mailing List <user@crunch.apache.org>
> Subject: Re: Reading Hive Tables into PCollection
>
> What format are they stored as?
>
> On Thu, Jan 28, 2016 at 1:20 PM Robinson, Landon - Landon <
> landon.t.robinson@lowes.com> wrote:
>
>> Crunch Gurus,
>>
>> What is the Crunch-convenient or recommended way to read the contents of
>> a Hive table into a Pcollection?
>> Thanks!
>> Best,
>> Landon
>>
>> ---------------------------------------------------------------------------
>> Landon Robinson
>> Big Data & Hadoop Engineer
>>
>> ---------------------------------------------------------------------------
>> NOTICE: All information in and attached to the e-mails below may be
>> proprietary, confidential, privileged and otherwise protected from improper
>> or erroneous disclosure. If you are not the sender's intended recipient,
>> you are not authorized to intercept, read, print, retain, copy, forward, or
>> disseminate this message. If you have erroneously received this
>> communication, please notify the sender immediately by phone (704-758-1000)
>> or by e-mail and destroy all copies of this message electronic, paper, or
>> otherwise.
>>
>> *By transmitting documents via this email: Users, Customers, Suppliers
>> and Vendors collectively acknowledge and agree the transmittal of
>> information via email is voluntary, is offered as a convenience, and is not
>> a secured method of communication; Not to transmit any payment information
>> E.G. credit card, debit card, checking account, wire transfer information,
>> passwords, or sensitive and personal information E.G. Driver's license,
>> DOB, social security, or any other information the user wishes to remain
>> confidential; To transmit only non-confidential information such as plans,
>> pictures and drawings and to assume all risk and liability for and
>> indemnify Lowe's from any claims, losses or damages that may arise from the
>> transmittal of documents or including non-confidential information in the
>> body of an email transmittal. Thank you. *
>>
> NOTICE: All information in and attached to the e-mails below may be
> proprietary, confidential, privileged and otherwise protected from improper
> or erroneous disclosure. If you are not the sender's intended recipient,
> you are not authorized to intercept, read, print, retain, copy, forward, or
> disseminate this message. If you have erroneously received this
> communication, please notify the sender immediately by phone (704-758-1000)
> or by e-mail and destroy all copies of this message electronic, paper, or
> otherwise.
>
> *By transmitting documents via this email: Users, Customers, Suppliers and
> Vendors collectively acknowledge and agree the transmittal of information
> via email is voluntary, is offered as a convenience, and is not a secured
> method of communication; Not to transmit any payment information E.G.
> credit card, debit card, checking account, wire transfer information,
> passwords, or sensitive and personal information E.G. Driver's license,
> DOB, social security, or any other information the user wishes to remain
> confidential; To transmit only non-confidential information such as plans,
> pictures and drawings and to assume all risk and liability for and
> indemnify Lowe's from any claims, losses or damages that may arise from the
> transmittal of documents or including non-confidential information in the
> body of an email transmittal. Thank you. *
>

Mime
View raw message