crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Krishna, Sivaram - Sivaram" <vc.Sivaram.Kris...@lowes.com>
Subject Re: Exception handling within the Run method
Date Fri, 17 Jun 2016 19:53:57 GMT
Thank you very much Josh, getSize works perfect for my use case.

Siva

On 6/17/16, 2:46 PM, "Josh Wills" <josh.wills@gmail.com> wrote:

>The exception is getting thrown in Pipeline.run(), which is called by
>Pipeline.done(), during the actual planning/execution of the job (so not
>during the read() block, which only defines the source-- we don't actually
>check that it exists until the runtime starts.) If you want to catch that
>exception before the job starts (and then possibly substitute an empty
>PCollection in its place), you can call getSize(Configuration) on the
>Source object (
>https://crunch.apache.org/apidocs/0.14.0/org/apache/crunch/Source.html ),
>which is what throws that Exception during the runtime.
>
>J
>
>On Fri, Jun 17, 2016 at 11:22 AM, Krishna, Sivaram - Sivaram <
>vc.Sivaram.Krishna@lowes.com> wrote:
>
>> Crunch Gurus,
>>
>> I¹m trying to read an Orc file and get the following error when the
>> directory containing the file is not present in HDFS.
>>
>> Failing Oozie Launcher, Main class
>> [org.apache.oozie.action.hadoop.JavaMain], main() threw exception,
>> java.lang.IllegalStateException: Input source Orc(hdfs://dir/path/) does
>> not exist!
>>
>> So I added a try catch for this specific exception,
>>
>> public class Foo extends Configured implements Tool {
>>     public static void main(String[] args) throws Exception {
>>         ToolRunner.run(new Configuration(), new Foo(), args);
>>     }
>>
>>     @Override
>>     public int run(String[] args) throws Exception {
>>
>> Configuration crunchConf = getConf();
>>         Pipeline pipeline = new MRPipeline(Foo.class, crunchConf);
>>
>>         PCollection<myRecord> myRecordPCollection =
>> pipeline.emptyPCollection(Orcs.reflects(myRecord.class));
>>         try{
>>             myRecordPCollection = pipeline.read(myRecordOrcFileSource);
>>         }catch(IllegalStateException e) {
>>             e.printStackTrace();
>>         }
>>
>>         PCollection<myNextRecord> myNextRecordPCollection =
>> pipeline.read(myNextRecordOrcFileSource);
>>
>>         PipelineResult result = pipeline.done();
>>         System.out.println("--> Pipeline ending.");
>>
>>         return result.succeeded() ? 0 : 1;
>>     }
>> }
>>
>> But I¹m unable to catch the exception within the run() and continue with
>> the remaining code. How could I handle this scenario? Any help is
>> appreciated.
>>
>> Thanks,
>> Siva
>>
>>
>>
>> ________________________________
>>
>> ***** Please note: The Sender of this email is either a Contractor or
>> Vendor of Lowe's Companies, Inc. and is not an employee or agent of
>>Lowe's
>> Companies Inc. *****
>>
>>
>> NOTICE: All information in and attached to the e-mails below may be
>> proprietary, confidential, privileged and otherwise protected from
>>improper
>> or erroneous disclosure. If you are not the sender's intended recipient,
>> you are not authorized to intercept, read, print, retain, copy,
>>forward, or
>> disseminate this message. If you have erroneously received this
>> communication, please notify the sender immediately by phone
>>(704-758-1000)
>> or by e-mail and destroy all copies of this message electronic, paper,
>>or
>> otherwise.
>>
>> By transmitting documents via this email: Users, Customers, Suppliers
>>and
>> Vendors collectively acknowledge and agree the transmittal of
>>information
>> via email is voluntary, is offered as a convenience, and is not a
>>secured
>> method of communication; Not to transmit any payment information E.G.
>> credit card, debit card, checking account, wire transfer information,
>> passwords, or sensitive and personal information E.G. Driver's license,
>> DOB, social security, or any other information the user wishes to remain
>> confidential; To transmit only non-confidential information such as
>>plans,
>> pictures and drawings and to assume all risk and liability for and
>> indemnify Lowe's from any claims, losses or damages that may arise from
>>the
>> transmittal of documents or including non-confidential information in
>>the
>> body of an email transmittal. Thank you.
>>


________________________________

***** Please note: The Sender of this email is either a Contractor or Vendor of Lowe's Companies,
Inc. and is not an employee or agent of Lowe's Companies Inc. *****


NOTICE: All information in and attached to the e-mails below may be proprietary, confidential,
privileged and otherwise protected from improper or erroneous disclosure. If you are not the
sender's intended recipient, you are not authorized to intercept, read, print, retain, copy,
forward, or disseminate this message. If you have erroneously received this communication,
please notify the sender immediately by phone (704-758-1000) or by e-mail and destroy all
copies of this message electronic, paper, or otherwise. 

By transmitting documents via this email: Users, Customers, Suppliers and Vendors collectively
acknowledge and agree the transmittal of information via email is voluntary, is offered as
a convenience, and is not a secured method of communication; Not to transmit any payment information
E.G. credit card, debit card, checking account, wire transfer information, passwords, or sensitive
and personal information E.G. Driver's license, DOB, social security, or any other information
the user wishes to remain confidential; To transmit only non-confidential information such
as plans, pictures and drawings and to assume all risk and liability for and indemnify Lowe's
from any claims, losses or damages that may arise from the transmittal of documents or including
non-confidential information in the body of an email transmittal. Thank you.

Mime
View raw message