Return-Path: 
 <dev-return-7511-apmail-accumulo-dev-archive=accumulo.apache.org@accumulo.apache.org>
X-Original-To: apmail-accumulo-dev-archive@www.apache.org
Delivered-To: apmail-accumulo-dev-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2D30C1032C
	for <apmail-accumulo-dev-archive@www.apache.org>;
 Tue, 19 Nov 2013 23:05:40 +0000 (UTC)
Received: (qmail 27679 invoked by uid 500); 19 Nov 2013 23:05:40 -0000
Delivered-To: apmail-accumulo-dev-archive@accumulo.apache.org
Received: (qmail 27619 invoked by uid 500); 19 Nov 2013 23:05:39 -0000
Mailing-List: contact dev-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@accumulo.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@accumulo.apache.org>
List-Post: <mailto:dev@accumulo.apache.org>
List-Id: <dev.accumulo.apache.org>
Reply-To: dev@accumulo.apache.org
Delivered-To: mailing list dev@accumulo.apache.org
Received: (qmail 27601 invoked by uid 99); 19 Nov 2013 23:05:39 -0000
Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Nov 2013 23:05:39 +0000
Received: from reviews.apache.org (localhost [127.0.0.1])
	by reviews.apache.org (Postfix) with ESMTP id 0964D1D3BBA;
	Tue, 19 Nov 2013 23:05:36 +0000 (UTC)
Content-Type: multipart/alternative;
 boundary="===============7105018339957201047=="
MIME-Version: 1.0
Subject: Re: Review Request 8559: Add Accumulo support to Sqoop
From: "Philip Grim" <phil@insufficient-light.com>
To: "Jarek Cecho" <jarcec@apache.org>
Cc: "accumulo" <dev@accumulo.apache.org>,
 "Philip Grim" <phil@insufficient-light.com>,
 "Sqoop" <sqoop-dev@incubator.apache.org>
Date: Tue, 19 Nov 2013 23:05:36 -0000
Message-ID: <20131119230536.6018.71553@reviews.apache.org>
X-ReviewBoard-URL: https://reviews.apache.org
Auto-Submitted: auto-generated
Sender: "Philip Grim" <noreply@reviews.apache.org>
X-ReviewGroup: accumulo, Sqoop
X-ReviewRequest-URL: https://reviews.apache.org/r/8559/
X-Sender: "Philip Grim" <noreply@reviews.apache.org>
References: <20131119210243.27204.58495@reviews.apache.org>
In-Reply-To: <20131119210243.27204.58495@reviews.apache.org>
Reply-To: "Philip Grim" <phil@insufficient-light.com>
X-ReviewRequest-Repository: sqoop-trunk

--===============7105018339957201047==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit



> On Nov. 19, 2013, 9:02 p.m., Jarek Cecho wrote:
> > Thank you for following up Philip!
> > 
> > I run tests on all supported Hadoop profiles and it seems to be working! I'm however having difficulties with real cluster tests, where I'm getting:
> > 
> > Error: java.lang.ClassNotFoundException: org.apache.accumulo.core.client.MutationsRejectedException
> >         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >         at java.security.AccessController.doPrivileged(Native Method)
> >         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >         at java.lang.Class.forName0(Native Method)
> >         at java.lang.Class.forName(Class.java:247)
> >         at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1526)
> >         at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1491)
> >         at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1585)
> >         at org.apache.sqoop.mapreduce.DelegatingOutputFormat$DelegatingRecordWriter.<init>(DelegatingOutputFormat.java:104)
> >         at org.apache.sqoop.mapreduce.DelegatingOutputFormat.getRecordWriter(DelegatingOutputFormat.java:82)
> >         at org
> > 
> > Sqoop do not propagate it's entire classpath to the mapreduce job, instead it's the connector/tool/job that specifies which jars needs to be propagated. For example HCatalog integration is having a util method [1], HBase integration is using special HBase API [2]. I think that the accumulo job have to do something similar.
> > 
> > Links:
> > 1: https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java#L727
> > 2: https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/HBaseImportJob.java#L228
> >

Yes, this isn't something I would notice, because in all of my Accumulo installations, I add the Accumulo and Zookeeper jars to the Hadoop classpath so that I don't have to specify all those jars every time I want to run a Map/Reduce job.

I'm posting the patch with the code to add the libjars, however I still need to do some testing.


- Philip


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8559/#review29129
-----------------------------------------------------------


On Nov. 18, 2013, 10:16 p.m., Philip Grim wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8559/
> -----------------------------------------------------------
> 
> (Updated Nov. 18, 2013, 10:16 p.m.)
> 
> 
> Review request for accumulo, Sqoop and Jarek Cecho.
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> Adds the ability to import to an Accumulo table in much the same manner as the current HBase import capability.  Reported in JIRA as ACCUMULO-141 and SQOOP-767.
> 
> 
> Diffs
> -----
> 
>   bin/configure-sqoop e604197 
>   bin/configure-sqoop.cmd ec57e37 
>   ivy.xml c5130ae 
>   src/docs/user/accumulo-args.txt PRE-CREATION 
>   src/docs/user/accumulo.txt PRE-CREATION 
>   src/docs/user/import.txt dfc9b39 
>   src/docs/user/validation.txt 282cfd6 
>   src/java/org/apache/sqoop/SqoopOptions.java 5c7a56a 
>   src/java/org/apache/sqoop/accumulo/AccumuloConstants.java PRE-CREATION 
>   src/java/org/apache/sqoop/accumulo/AccumuloMutationProcessor.java PRE-CREATION 
>   src/java/org/apache/sqoop/accumulo/AccumuloUtil.java PRE-CREATION 
>   src/java/org/apache/sqoop/accumulo/MutationTransformer.java PRE-CREATION 
>   src/java/org/apache/sqoop/accumulo/ToStringMutationTransformer.java PRE-CREATION 
>   src/java/org/apache/sqoop/manager/SqlManager.java 1ffa40f 
>   src/java/org/apache/sqoop/mapreduce/AccumuloImportJob.java PRE-CREATION 
>   src/java/org/apache/sqoop/mapreduce/AccumuloImportMapper.java PRE-CREATION 
>   src/java/org/apache/sqoop/tool/BaseSqoopTool.java 018d11f 
>   src/java/org/apache/sqoop/tool/ImportTool.java fbbde1d 
>   src/test/org/apache/sqoop/accumulo/AccumuloTestCase.java PRE-CREATION 
>   src/test/org/apache/sqoop/accumulo/TestAccumuloImport.java PRE-CREATION 
>   src/test/org/apache/sqoop/accumulo/TestAccumuloQueryImport.java PRE-CREATION 
>   src/test/org/apache/sqoop/accumulo/TestAccumuloUtil.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/8559/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Philip Grim
> 
>


--===============7105018339957201047==--