avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jesse Jaggars <jhjagg...@gmail.com>
Subject java.lang.NoSuchMethodError: org.apache.avro.Schema.access$1400()Ljava/lang/ThreadLocal
Date Mon, 19 Aug 2013 01:29:37 GMT
I'm trying to build a simple mapreduce job that reads avro files and outputs plain text.

I pulled data from a mysql database with sqoop and wrote the files out as snappy compressed
avro files. I've operated on the files using AvroStorage in pig, but the current task I'm
trying to accomplish would be better suited with a plain MR job I think. I'm using avro 1.7.4
but I think the version used to generate the files is 1.5.3 (whatever ships with HDP 1.2).
I've tried depending on avro 1.5.3 in my project but I get the same error (but a different
line).

When I try to execute my job the following exception is printed:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.avro.Schema.access$1400()Ljava/lang/ThreadLocal;
        at org.apache.avro.Schema$Parser.parse(Schema.java:924)
        at org.apache.avro.Schema$Parser.parse(Schema.java:917)
        at my.classpath.jobs.CandidatesFor.run(CandidatesFor.java:44)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at my.classpath.jobs.CandidatesFor.main(CandidatesFor.java:72)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

Below is the setup portion of my mapreduce job. The failure occurs when I call parse on the
Schema object.

public class CandidatesFor extends Configured implements Tool {

    @Override
    public int run(String[] args) throws Exception {
        JobConf conf = new JobConf(getConf(), getClass());
        conf.setJobName("CandidatesFor");

        InputStream is = getClass().getClassLoader().getResourceAsStream("avro/data.avsc");
        assert null != is;
        String schemaString = IOUtils.toString(is);
        System.out.println(schemaString);
        Schema.Parser parser = new Schema.Parser();
        Schema schema = parser.parse(schemaString); // exception is thrown here
	…

Looking in the code for Schema the line is:

	boolean saved = validateNames.get();

I wrote a unit test to help me understand how things hang together. The test passes:

    @Test
    public void canReadSchema() throws Exception {
        ClassLoader loader = getClass().getClassLoader();
        InputStream schema_is = loader.getResourceAsStream("avro/data.avsc");
        File data = new File(loader.getResource("avro/part-m-00000.avro").toURI());
        assertNotNull(data);
        Schema schema = new Schema.Parser().parse(schema_is);
        DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>(schema);
        DataFileReader<GenericRecord> dataFileReader = new DataFileReader<GenericRecord>(data,
datumReader);
        GenericRecord case_ = null;
        while (dataFileReader.hasNext()) {
            case_ = dataFileReader.next(case_);
            assertNotNull(case_.get("subject"));
            assertNotNull(case_.get("description"));
        }
    }

I'm sure I'm missing something obvious but I don't know enough to recognize it. Any help would
be greatly appreciated. 
Thanks
Mime
View raw message