accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dlmarion <>
Subject Re: Unable to load Iterator with setscaniter and setshelliter
Date Sun, 15 Jun 2014 15:03:22 GMT
general.vfs.classpaths is not set?
I saw in an earlier email that you are setting the context classloader. There are some places
where the context manager was not being used when it should. This could be one of those cases.
I would submit a jira ticket with all the information and I will try to look at it in the
next day or so. Be sure to include classpath settings, list of files in HDFS for those locations,
and the commands that are failing.

Sent via the Samsung GALAXY S®4, an AT&T 4G LTE smartphone

<div>-------- Original message --------</div><div>From: David Medinets <>
</div><div>Date:06/15/2014  10:53 AM  (GMT-05:00) </div><div>To: accumulo-user
<> </div><div>Subject: Re: Unable to load Iterator
with setscaniter and setshelliter </div><div>
</div>The classpath settings in accumulo-site.xml are the following (which I think is
the default):

      <description>Classpaths that accumulo checks for updates and class files.
      When using the Security Manager, please remove the ".../target/classes/" values.

On Sun, Jun 15, 2014 at 10:49 AM, dlmarion <> wrote:
What does your classpath settings look like in accumulo-site.xml. I recently made some fixes
in 1.6.1-Snapshot where the context classloader was not being used in all cases. I dont think
this case was affected though.

Sent via the Samsung GALAXY S®4, an AT&T 4G LTE smartphone

-------- Original message --------
From: Josh Elser
Date:06/15/2014 10:31 AM (GMT-05:00)
Subject: Re: Unable to load Iterator with setscaniter and setshelliter

Naw, the commons-vfs loader should be loading those resources using a second classloader.

Maybe it's just a problem with the HDFS code? Does it work if you have the jar with your iterator
in lib/ or lib/ext? Or, maybe something is wrong like you defined a private constructor which
is throwing that Exception?

On Jun 15, 2014 8:50 AM, "William Slacum" <> wrote:
Wouldn't the iterator have to be on the classpath for the JVM that launches the shell command?

On Sun, Jun 15, 2014 at 9:02 AM, Vicky Kak <> wrote:

>>setiter -n MyIterator -p 10 -scan -minc -majc -class com.codebits.d4m.iterator.MyIterator

The above line fails for me with the similar kind of error i.e ClassNotFoundException

root@accumulo atest> setiter -n MyIterator -p 10 -scan -minc -majc -class org.dallaybatta.MyIterator
2014-06-15 18:20:18,061 [shell.Shell] ERROR:
Command could not be initialized (Unable to load org.dallaybatta.MyIterator; class not found.)

My hdfs contains the corresponding jars but it yet fails.
After digging a code for a while I figured that the error is coming from

    try {
      clazz = classloader.loadClass(className).asSubclass(SortedKeyValueIterator.class);
      untypedInstance = clazz.newInstance();
    } catch (ClassNotFoundException e) {
      StringBuilder msg = new StringBuilder("Unable to load ").append(className);
      if (className.indexOf('.') < 0) {
        msg.append("; did you use a fully qualified package name?");
      } else {
        msg.append("; class not found.");
      throw new ShellCommandException(ErrorCode.INITIALIZATION_FAILURE, msg.toString());
    } catch (InstantiationException e) {


Typically the ClassNotFoundException can appear also if the dependent classes are not present,
here SortedKeyValueIterator could be the use case, I moved the accumulo core to the classpath
folder in the hdfs but still could not get it working. May be some other dependent classes
are required, needs more time to analyse in this direction.

The document states the following so it should ideally work,
"although the VFS classloader allows for classpath manipulation using a variety of schemes
including URLs and HDFS URIs."

I find it strange that your test and my tests results differ as you are able to set Iterator
for a single row but I am not.


On Sun, Jun 15, 2014 at 8:25 AM, David Medinets <> wrote:
I'm sure that I'm overlooking something simple. I can load my iterator using setiter but not
with setscaniter or setshelliter.

Here is my do-nothing iterator:

public class MyIterator extends WrappingIterator implements OptionDescriber {

    public IteratorOptions describeOptions() {
        String name = "dummy";
        String description = "Dummy Description";
        Map<String, String> namedOptions = new HashMap<String, String>();
        List<String> unnamedOptionDescriptions = null;
        return new IteratorOptions(name, description, namedOptions, unnamedOptionDescriptions);

    public boolean validateOptions(Map<String, String> options) {
        return true;

I copy the jar file out to HDFS:

hadoop fs -mkdir /user/vagrant/d4m/classpath
hadoop fs -put /vagrant/schema/target/d4m_schema-0.0.1-SNAPSHOT.jar /user/vagrant/classpath

I set the table-specific classpath context:

createtable atest
table atest
insert row cf cq value
config -s general.vfs.context.classpath.d4m=hdfs://affy-master:9000/user/vagrant/classpath
config -t atest -s table.classpath.context=d4m

Now I can configure the iterator and scan over the single row without a problem:

setiter -n MyIterator -p 10 -scan -minc -majc -class com.codebits.d4m.iterator.MyIterator
deleteiter -n MyIterator -scan -minc -majc 

However, the setscaniter commands fails:

root@instance atest> setscaniter -n MyIterator -p 10 -class com.codebits.d4m.iterator.MyIterator
2014-06-15 02:54:14,098 [shell.Shell] WARN : Deprecated, use setshelliter
Dummy Description
2014-06-15 02:54:14,126 [shell.Shell] ERROR:
Command could not be initialized (Unable to load com.codebits.d4m.iterator.MyIterator)

As does the setshelliter:

root@instance atest> setshelliter -pn d4m -n MyIterator -p 10 -class com.codebits.d4m.iterator.MyIterator
Dummy Description
2014-06-15 02:55:07,025 [shell.Shell] ERROR:
Command could not be initialized (Unable to load com.codebits.d4m.iterator.MyIterator)

I don't see any messages in the log files. 

Any suggestions to resolve these issues?

View raw message