Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 88C7F110C2 for ; Sat, 23 Aug 2014 05:17:55 +0000 (UTC) Received: (qmail 50703 invoked by uid 500); 23 Aug 2014 05:17:55 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 50645 invoked by uid 500); 23 Aug 2014 05:17:55 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 50633 invoked by uid 99); 23 Aug 2014 05:17:55 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Aug 2014 05:17:55 +0000 X-ASF-Spam-Status: No, hits=2.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,URI_HEX,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of cjnolet@gmail.com designates 209.85.213.171 as permitted sender) Received: from [209.85.213.171] (HELO mail-ig0-f171.google.com) (209.85.213.171) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Aug 2014 05:17:50 +0000 Received: by mail-ig0-f171.google.com with SMTP id l13so655859iga.4 for ; Fri, 22 Aug 2014 22:17:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=ZEYgm69ko+/Fz43ex5Zv/bedKH6GzBrfRWB9BgQMG1g=; b=JxhePK+vxhfl8NWlXD/OAaYsjlJjHwFKghvZwXcfEzdZCSpkq4AVZV6wC7nVp3/+sp sEUaUQtKMWBTD4tqJWPD84ha2hXExU88juvw0qensohrK9mKMpO41N462ekkVMwGilyk JRwvH2eVPoxiSn6LeeQKl5S1qQj6NHHj2KtegtAxEA3FMUBomS6gVXFn0fU72ZScD85r uVjibTALtQNzakdAKXiQiOL+X0171uDP9xa5hrGzYGpeiPIMSw3LlkKpZnufHXs/RwFu +ABaLEJHFlrVK2CX70TxrNwDcrw6QiT3+Hyi83M1HLUvmGgDkvZdhhuWQjJifvZue94x OIAA== MIME-Version: 1.0 X-Received: by 10.42.114.130 with SMTP id g2mr4488697icq.46.1408771049438; Fri, 22 Aug 2014 22:17:29 -0700 (PDT) Received: by 10.64.167.229 with HTTP; Fri, 22 Aug 2014 22:17:29 -0700 (PDT) In-Reply-To: References: <1408767338901-11189.post@n5.nabble.com> <1408768282747-11191.post@n5.nabble.com> <1408770004607-11193.post@n5.nabble.com> Date: Sat, 23 Aug 2014 01:17:29 -0400 Message-ID: Subject: Re: AccumuloMultiTableInputFormat IllegalStatementException From: Corey Nolet To: user@accumulo.apache.org Content-Type: multipart/alternative; boundary=20cf303bf576f8c4bc05014511ea X-Virus-Checked: Checked by ClamAV on apache.org --20cf303bf576f8c4bc05014511ea Content-Type: text/plain; charset=UTF-8 Also, if you don't mind me asking, why isn't your job setup class extending Configured? That was you are picking up configurations injected from the environment. You would do "MyJobSetUpClass extends Configured" Then use getConf() instead of newing up a new configuration. On Sat, Aug 23, 2014 at 1:11 AM, Corey Nolet wrote: > Job.getInstance(configuration) copies the configuration and makes its own. > Try doing your debug statement from earlier on job.getConfiguration() and > let's see what the base64 string looks like. > > > > On Sat, Aug 23, 2014 at 1:00 AM, JavaHokie > wrote: > >> Sure thing, here's my run method implementation: >> >> Configuration configuration = new Configuration(); >> >> configuration.set("fs.defaultFS", "hdfs://127.0.0.1:8020"); >> configuration.set("mapreduce.job.tracker", "localhost:54311"); >> configuration.set("mapreduce.framework.name", "yarn"); >> configuration.set("yarn.resourcemanager.address", >> "localhost:8032"); >> >> Job job = Job.getInstance(configuration); >> >> /* >> * Set the basic stuff >> */ >> job.setJobName("TwitterJoin Query"); >> job.setJarByClass(TwitterJoin.class); >> >> /* >> * Set Mapper and Reducer Classes >> */ >> job.setMapperClass(TwitterJoinMapper.class); >> job.setReducerClass(TwitterJoinReducer.class); >> >> /* >> * Set the Mapper MapOutputKeyClass and MapOutputValueClass >> */ >> job.setMapOutputKeyClass(Text.class); >> job.setMapOutputValueClass(Text.class); >> >> /* >> * Set the Reducer OutputKeyClass and OutputValueClass >> */ >> job.setOutputKeyClass(Text.class); >> job.setOutputValueClass(Mutation.class); >> >> /* >> * Set InputFormat and OutputFormat classes >> */ >> >> job.setInputFormatClass(AccumuloMultiTableInputFormat.class); >> job.setOutputFormatClass(AccumuloOutputFormat.class); >> >> /* >> * Configure InputFormat and OutputFormat Classes >> */ >> Map configs = new >> HashMap(); >> >> List ranges = Lists.newArrayList(new >> Range("104587"),new >> Range("105255")); >> >> InputTableConfig edgeConfig = new InputTableConfig(); >> edgeConfig.setRanges(ranges); >> edgeConfig.setAutoAdjustRanges(true); >> >> InputTableConfig followerConfig = new InputTableConfig(); >> followerConfig.setRanges(ranges); >> followerConfig.setAutoAdjustRanges(true); >> >> configs.put("following",followerConfig); >> configs.put("twitteredges",edgeConfig); >> >> >> AccumuloMultiTableInputFormat.setConnectorInfo(job,"root",new >> PasswordToken("********".getBytes())); >> >> >> AccumuloMultiTableInputFormat.setZooKeeperInstance(job,"localhost","localhost"); >> >> AccumuloMultiTableInputFormat.setScanAuthorizations(job,new >> Authorizations("private")); >> AccumuloMultiTableInputFormat.setInputTableConfigs(job, >> configs); >> >> >> AccumuloOutputFormat.setZooKeeperInstance(job,"localhost","localhost"); >> AccumuloOutputFormat.setConnectorInfo(job,"root",new >> PasswordToken("********".getBytes())); >> AccumuloOutputFormat.setCreateTables(job,true); >> >> AccumuloOutputFormat.setDefaultTableName(job,"twitteredgerollup"); >> >> /* >> * Kick off the job, wait for completion, and return >> applicable code >> */ >> boolean success = job.waitForCompletion(true); >> >> if (success) { >> return 0; >> } >> >> return 1; >> } >> >> >> >> -- >> View this message in context: >> http://apache-accumulo.1065345.n5.nabble.com/AccumuloMultiTableInputFormat-IllegalStateException-tp11186p11193.html >> Sent from the Users mailing list archive at Nabble.com. >> > > --20cf303bf576f8c4bc05014511ea Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Also, if you don't mind me asking, why isn't your = job setup class extending Configured? That was you are picking up configura= tions injected from the environment.

You would do "= MyJobSetUpClass extends Configured" Then use getConf() instead of newi= ng up a new configuration.


On Sat,= Aug 23, 2014 at 1:11 AM, Corey Nolet <cjnolet@gmail.com> wr= ote:
Job.getInstance(configurati= on) copies the configuration and makes its own. Try doing your debug statem= ent from earlier on job.getConfiguration() and let's see what the base6= 4 string looks like.


On Sat, Aug 23, 2014 at 1:00 AM, JavaHokie= <soozandjohnyost@gmail.com> wrote:
Sure thing, here's my run method impleme= ntation:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Configuration configuration =3D n= ew Configuration();

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 configuration.set("fs.defaul= tFS", "hdfs://127.0.0.1:8020");
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 configuration.set("mapreduce= .job.tracker", "localhost:54311");
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 configuration.set("mapreduce.framework.= name", "yarn");
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 configuration.set("yarn.reso= urcemanager.address",
"localhost:8032");

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Job job =3D Job.get= Instance(configuration);

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /*
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* Set the bas= ic stuff
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0*/
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 job.setJobName(&quo= t;TwitterJoin Query");
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 job.setJarByClass(T= witterJoin.class);

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /*
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* Set Mapper = and Reducer Classes
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0*/
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 job.setMapperClass(= TwitterJoinMapper.class);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 job.setReducerClass= (TwitterJoinReducer.class);

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /*
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* Set the Mapper MapOutputK= eyClass and MapOutputValueClass
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0*/
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 job.setMapOutputKeyClass(Text.cla= ss);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 job.setMapOutputValueClass(Text.c= lass);

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /*
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* Set the Reducer OutputKey= Class and OutputValueClass
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0*/
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 job.setOutputKeyClass(Text.class)= ;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 job.setOutputValueClass(Mutation.= class);

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /*
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* Set InputFo= rmat and OutputFormat classes
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0*/
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 job.setInputFormatC= lass(AccumuloMultiTableInputFormat.class);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 job.setOutputFormat= Class(AccumuloOutputFormat.class);

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /*
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* Configure I= nputFormat and OutputFormat Classes
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0*/
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Map<String,= InputTableConfig> configs =3D new
HashMap<String,InputTableConfig>();

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 List<Range> r= anges =3D Lists.newArrayList(new Range("104587"),new
Range("105255"));

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 InputTableConfig ed= geConfig =3D new InputTableConfig();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 edgeConfig.setRange= s(ranges);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 edgeConfig.se= tAutoAdjustRanges(true);

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 InputTableConfig fo= llowerConfig =3D new InputTableConfig();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 followerConfig.setR= anges(ranges);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 followerConfi= g.setAutoAdjustRanges(true);

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 configs.put("f= ollowing",followerConfig);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 configs.put("t= witteredges",edgeConfig);

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 AccumuloMulti= TableInputFormat.setConnectorInfo(job,"root",new
PasswordToken("********".getBytes()));

AccumuloMultiTableInputFormat.setZooKeeperInstance(job,"localhost"= ;,"localhost");
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 AccumuloMultiTableI= nputFormat.setScanAuthorizations(job,new
Authorizations("private"));
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 AccumuloMultiTableI= nputFormat.setInputTableConfigs(job, configs);

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 AccumuloOutputForma= t.setZooKeeperInstance(job,"localhost","localhost"); =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 AccumuloOutputForma= t.setConnectorInfo(job,"root",new
PasswordToken("********".getBytes()));
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 AccumuloOutputForma= t.setCreateTables(job,true);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 AccumuloOutputForma= t.setDefaultTableName(job,"twitteredgerollup");

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /*
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* Kick off th= e job, wait for completion, and return applicable code
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0*/
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 boolean success =3D job.waitForCo= mpletion(true);

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (success) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return 0;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 }

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return 1;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }



--
View this message in context: http://apache-accumulo.1065345.n5.nabble.com/Accu= muloMultiTableInputFormat-IllegalStateException-tp11186p11193.html
Sent from the Users mailing list archive at Nabble.com.


--20cf303bf576f8c4bc05014511ea--