accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dlmarion <dlmar...@comcast.net>
Subject Re: Unable to load Iterator with setscaniter and setshelliter
Date Sun, 15 Jun 2014 14:49:46 GMT
What does your classpath settings look like in accumulo-site.xml. I recently made some fixes
in 1.6.1-Snapshot where the context classloader was not being used in all cases. I dont think
this case was affected though.


Sent via the Samsung GALAXY S®4, an AT&T 4G LTE smartphone

<div>-------- Original message --------</div><div>From: Josh Elser <josh.elser@gmail.com>
</div><div>Date:06/15/2014  10:31 AM  (GMT-05:00) </div><div>To: user@accumulo.apache.org
</div><div>Subject: Re: Unable to load Iterator with setscaniter and setshelliter
</div><div>
</div>Naw, the commons-vfs loader should be loading those resources using a second classloader.

Maybe it's just a problem with the HDFS code? Does it work if you have the jar with your iterator
in lib/ or lib/ext? Or, maybe something is wrong like you defined a private constructor which
is throwing that Exception?

On Jun 15, 2014 8:50 AM, "William Slacum" <wilhelm.von.cloud@accumulo.net> wrote:
Wouldn't the iterator have to be on the classpath for the JVM that launches the shell command?


On Sun, Jun 15, 2014 at 9:02 AM, Vicky Kak <vicky.kak@gmail.com> wrote:

>>setiter -n MyIterator -p 10 -scan -minc -majc -class com.codebits.d4m.iterator.MyIterator
>>scan

The above line fails for me with the similar kind of error i.e ClassNotFoundException

root@accumulo atest> setiter -n MyIterator -p 10 -scan -minc -majc -class org.dallaybatta.MyIterator
2014-06-15 18:20:18,061 [shell.Shell] ERROR: org.apache.accumulo.shell.ShellCommandException:
Command could not be initialized (Unable to load org.dallaybatta.MyIterator; class not found.)


My hdfs contains the corresponding jars but it yet fails.
After digging a code for a while I figured that the error is coming from org.apache.accumulo.shell.commands.SetIterCommand::execute

*********************************************************
    try {
      clazz = classloader.loadClass(className).asSubclass(SortedKeyValueIterator.class);
      untypedInstance = clazz.newInstance();
    } catch (ClassNotFoundException e) {
      StringBuilder msg = new StringBuilder("Unable to load ").append(className);
      if (className.indexOf('.') < 0) {
        msg.append("; did you use a fully qualified package name?");
      } else {
        msg.append("; class not found.");
      }
      throw new ShellCommandException(ErrorCode.INITIALIZATION_FAILURE, msg.toString());
    } catch (InstantiationException e) {

*********************************************************

Typically the ClassNotFoundException can appear also if the dependent classes are not present,
here SortedKeyValueIterator could be the use case, I moved the accumulo core to the classpath
folder in the hdfs but still could not get it working. May be some other dependent classes
are required, needs more time to analyse in this direction.

The document states the following so it should ideally work,
"although the VFS classloader allows for classpath manipulation using a variety of schemes
including URLs and HDFS URIs."

I find it strange that your test and my tests results differ as you are able to set Iterator
for a single row but I am not.

Thanks,
Vicky




On Sun, Jun 15, 2014 at 8:25 AM, David Medinets <david.medinets@gmail.com> wrote:
I'm sure that I'm overlooking something simple. I can load my iterator using setiter but not
with setscaniter or setshelliter.

Here is my do-nothing iterator:

public class MyIterator extends WrappingIterator implements OptionDescriber {

    @Override
    public IteratorOptions describeOptions() {
        String name = "dummy";
        String description = "Dummy Description";
        Map<String, String> namedOptions = new HashMap<String, String>();
        List<String> unnamedOptionDescriptions = null;
        return new IteratorOptions(name, description, namedOptions, unnamedOptionDescriptions);
    }

    @Override
    public boolean validateOptions(Map<String, String> options) {
        return true;
    }
    
}

I copy the jar file out to HDFS:

hadoop fs -mkdir /user/vagrant/d4m/classpath
hadoop fs -put /vagrant/schema/target/d4m_schema-0.0.1-SNAPSHOT.jar /user/vagrant/classpath

I set the table-specific classpath context:

createtable atest
table atest
insert row cf cq value
config -s general.vfs.context.classpath.d4m=hdfs://affy-master:9000/user/vagrant/classpath
config -t atest -s table.classpath.context=d4m

Now I can configure the iterator and scan over the single row without a problem:

setiter -n MyIterator -p 10 -scan -minc -majc -class com.codebits.d4m.iterator.MyIterator
scan
deleteiter -n MyIterator -scan -minc -majc 

However, the setscaniter commands fails:

root@instance atest> setscaniter -n MyIterator -p 10 -class com.codebits.d4m.iterator.MyIterator
2014-06-15 02:54:14,098 [shell.Shell] WARN : Deprecated, use setshelliter
Dummy Description
2014-06-15 02:54:14,126 [shell.Shell] ERROR: org.apache.accumulo.core.util.shell.ShellCommandException:
Command could not be initialized (Unable to load com.codebits.d4m.iterator.MyIterator)

As does the setshelliter:

root@instance atest> setshelliter -pn d4m -n MyIterator -p 10 -class com.codebits.d4m.iterator.MyIterator
Dummy Description
2014-06-15 02:55:07,025 [shell.Shell] ERROR: org.apache.accumulo.core.util.shell.ShellCommandException:
Command could not be initialized (Unable to load com.codebits.d4m.iterator.MyIterator)

I don't see any messages in the log files. 

Any suggestions to resolve these issues?



Mime
View raw message