nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject Re: Record path API chooses CHOICE[STRING, RECORD] when the field is missing
Date Mon, 29 Jul 2019 19:36:40 GMT
Mike,

What Record Reader is being used here? The problem appears to be due to the Record Reader
itself assigning that as the field type.

I created a dummy unit test to verify the RecordPath stuff is correct:


@Test
public void testFromEmail() {
    final List<RecordField> fields = new ArrayList<>();
    fields.add(new RecordField("firstName", RecordFieldType.STRING.getDataType()));
    fields.add(new RecordField("lastName", RecordFieldType.STRING.getDataType()));
    fields.add(new RecordField("creationDateTime", RecordFieldType.TIMESTAMP.getDataType(),
true));
    final RecordSchema schema = new SimpleRecordSchema(fields);

    final Map<String, Object> values = new HashMap<>();
    values.put("firstName", "John");
    values.put("lastName", "Doe");
    values.put("creationDateTime", new Timestamp(System.currentTimeMillis()));
    final Record record = new MapRecord(schema, values);

    final Optional<FieldValue> optionalFieldValue = RecordPath.compile("/creationDateTime").evaluate(record).getSelectedFields().findFirst();
    final FieldValue fieldValue = optionalFieldValue.get();
    System.out.println(fieldValue.getField());
}

Which prints out the correct field type:

RecordField[name=creationDateTime, dataType=TIMESTAMP:yyyy-MM-dd HH:mm:ss, nullable=true]

So I presume the Record Reader may not be properly applying the schema to the Record that
it returns.

Thanks
-Mark

On Jul 28, 2019, at 10:19 AM, Mike Thomsen <mikerthomsen@gmail.com<mailto:mikerthomsen@gmail.com>>
wrote:

I have a simple avro schema in a test case that looks like this:

{
   "type": "record",
   "name": "PersonRecord",
   "fields": [
       { "name": "firstName", "type": "string" },
       { "name": "lastName", "type": "string" },
       { "name": "creationDateTime", "type": [ "null", "type": "long",
"logicalType": "timestamp-millis" }]
   ]
}

Then I try something like this...

RecordPath path = recordPathCache.getCompiled("/creationDateTime");
RecordPathResult rp = path.evaluate(targetRecord);
Optional<FieldValue> nodeField = rp.getSelectedFields().findFirst();

if (!nodeField.isPresent()) {
   throw new ProcessException("...");
}

FieldValue fieldValue = nodeField.get();
//fieldValue.getField() is a Choice of String, Record

Is there a way to get the correct field type here? I assume that
Choice[String, Record] default here was done to facilitate schema inference.

Thanks,

Mike


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message