nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <>
Subject Re: Record path API chooses CHOICE[STRING, RECORD] when the field is missing
Date Mon, 29 Jul 2019 19:36:40 GMT

What Record Reader is being used here? The problem appears to be due to the Record Reader
itself assigning that as the field type.

I created a dummy unit test to verify the RecordPath stuff is correct:

public void testFromEmail() {
    final List<RecordField> fields = new ArrayList<>();
    fields.add(new RecordField("firstName", RecordFieldType.STRING.getDataType()));
    fields.add(new RecordField("lastName", RecordFieldType.STRING.getDataType()));
    fields.add(new RecordField("creationDateTime", RecordFieldType.TIMESTAMP.getDataType(),
    final RecordSchema schema = new SimpleRecordSchema(fields);

    final Map<String, Object> values = new HashMap<>();
    values.put("firstName", "John");
    values.put("lastName", "Doe");
    values.put("creationDateTime", new Timestamp(System.currentTimeMillis()));
    final Record record = new MapRecord(schema, values);

    final Optional<FieldValue> optionalFieldValue = RecordPath.compile("/creationDateTime").evaluate(record).getSelectedFields().findFirst();
    final FieldValue fieldValue = optionalFieldValue.get();

Which prints out the correct field type:

RecordField[name=creationDateTime, dataType=TIMESTAMP:yyyy-MM-dd HH:mm:ss, nullable=true]

So I presume the Record Reader may not be properly applying the schema to the Record that
it returns.


On Jul 28, 2019, at 10:19 AM, Mike Thomsen <<>>

I have a simple avro schema in a test case that looks like this:

   "type": "record",
   "name": "PersonRecord",
   "fields": [
       { "name": "firstName", "type": "string" },
       { "name": "lastName", "type": "string" },
       { "name": "creationDateTime", "type": [ "null", "type": "long",
"logicalType": "timestamp-millis" }]

Then I try something like this...

RecordPath path = recordPathCache.getCompiled("/creationDateTime");
RecordPathResult rp = path.evaluate(targetRecord);
Optional<FieldValue> nodeField = rp.getSelectedFields().findFirst();

if (!nodeField.isPresent()) {
   throw new ProcessException("...");

FieldValue fieldValue = nodeField.get();
//fieldValue.getField() is a Choice of String, Record

Is there a way to get the correct field type here? I assume that
Choice[String, Record] default here was done to facilitate schema inference.



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message