avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lurga" <lurga....@gmail.com>
Subject Re: Re: Question on writing/reading file with different schema
Date Tue, 13 Apr 2010 07:04:33 GMT
Hi,
27
20
31
Dante Hicks	
Exception in thread "main" org.apache.avro.AvroRuntimeException: java.io.EOFException
	at org.apache.avro.file.DataFileStream.next(DataFileStream.java:184)
	at cn.znest.test.avro.AddressBook.browseName(AddressBook.java:91)
	at cn.znest.test.avro.AddressBook.main(AddressBook.java:43)
Caused by: java.io.EOFException
	at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:163)
	at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:262)
	at org.apache.avro.io.ValidatingDecoder.readString(ValidatingDecoder.java:93)
	at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:277)
	at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:271)
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:83)
	at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:105)
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:77)
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:70)
	at org.apache.avro.file.DataFileStream.next(DataFileStream.java:195)
	at org.apache.avro.file.DataFileStream.next(DataFileStream.java:182)
	... 2 more

My code is below. In this example, I create three record: person(3 fileds: First Last Age),
age(Age), extract(First Last). The record "age" has the last filed of "Person", so AddressBook.browseAge()
will be executed successfully. But the record "extract" does not have the last filed of "Person",
so executing AddressBook.browseName() will cause an exception.  
In avro/c, read_record (datum_read.c) loops every write_schema fileds. 
In avro/java, GenericDatumReader.readRecord  loops every read_schema fileds. I think that's
the point.

import java.io.File;
import java.io.IOException;

import org.apache.avro.Schema;
import org.apache.avro.file.DataFileReader;
import org.apache.avro.file.DataFileWriter;
import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericDatumReader;
import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericData.Record;
import org.apache.avro.util.Utf8;

public class AddressBook {
	String fileName = "AddressBook.db";
	String prefix = "{\"type\":\"record\",\"name\": \"Person\",\"fields\":[";
	String suffix = "]}";
	String fieldFirst = "{\"name\":\"First\",\"type\":\"string\"}";
	String fieldLast = "{\"name\":\"Last\",\"type\":\"string\"}";
	String fieldAge = "{\"name\":\"Age\",\"type\":\"int\"}";
	Schema personSchema = Schema.parse(prefix + fieldFirst + "," + fieldLast + ","  + fieldAge
+ suffix);
	Schema ageSchema = Schema.parse(prefix + fieldAge + suffix);
	Schema extractSchema = Schema.parse(prefix + fieldFirst + "," + fieldLast + suffix);
	/**
	 * @param args
	 * @throws IOException
	 */
	public static void main(String[] args) throws IOException {
		AddressBook ab = new AddressBook();
		ab.init();
		ab.browseAge();
		ab.browseName();
	}

	public void init() throws IOException {		
		DataFileWriter<Record> writer = new DataFileWriter<Record>(
				new GenericDatumWriter<Record>(personSchema)).create(
						personSchema, new File(fileName));
		try {
			writer.append(createPerson("Dante", "Hicks", 27));
			writer.append(createPerson("Randal", "Graves", 20));
			writer.append(createPerson("Steve", "Jobs", 31));
		} finally {
			writer.close();
		}
	}
	
	private Record createPerson(String first, String last, int age) {
		Record person = new GenericData.Record(personSchema);
		person.put("First", new Utf8(first));
		person.put("Last", new Utf8(last));
		person.put("Age", age);
		return person;
	}
	
	public void browseAge() throws IOException {		
		GenericDatumReader<Record> dr = new GenericDatumReader<Record>();
		dr.setExpected(ageSchema);
		DataFileReader<Record> reader = new DataFileReader<Record>(new File(
		  fileName), dr);
		
		try {
			while (reader.hasNext()) {
				Record person = reader.next();
				System.out.println(person.get("Age").toString());
			}
		} finally {
			reader.close();
		}
	}
	
	public void browseName() throws IOException {		
		GenericDatumReader<Record> dr = new GenericDatumReader<Record>();
		dr.setExpected(extractSchema);
		DataFileReader<Record> reader = new DataFileReader<Record>(new File(
		  fileName), dr);
		
		try {
			while (reader.hasNext()) {
				Record person = reader.next();
				System.out.println(person.get("First").toString() + " " + person.get("Last").toString()
+ "\t");
			}
		} finally {
			reader.close();
		}
	}
}


2010-04-13 
Lurga            

发件人: Scott Carey 
发送时间: 2010-04-13  10:55:41 
收件人: avro-user@hadoop.apache.org 
抄送: 
主题: Re: Question on writing/reading file with different schema 
 
On Apr 12, 2010, at 7:42 PM, Scott Carey wrote:
> Try something like:
> 
> DatumReader dr = new GenericDatumReader();
> dr.setExpected(extractSchema);
> DataFileReader<Record> reader = new DataFileReader<Record>(new File(
>  fileName), dr);
> 
If the above still fails, please provide the full stack trace that results.
> 
>> Regards,
>> 
>> 2010-04-13 
>> 
>> 
>> 
>> Lurga 
>> 
> 
Mime
View raw message