nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: Infer Avro Schema [CONFIDENTIAL]
Date Tue, 14 Apr 2020 11:04:48 GMT
Yep definitely agree Chris.  This is a good time to switch to using record
readers and writers and the built in schema handling.

Thanks

On Tue, Apr 14, 2020 at 3:06 AM Chris Sampson
<chris.sampson@naimuri.com.invalid> wrote:

> An alternative (without adding the kite nar back into your installation),
> is to use one of the "standard" Record processors (e.g. ConvertRecord,
> UpdateRecord, ValidateRecord or CalculateRecordStats) with an appropriate
> RecordReader with schema access set to "Infer Schema" and schema output to
> "Write as Attribute" (I'm paraphrasing the settings from memory, but they
> should be fairly evident). The RecordWriter can then output in the same
> format as the reader and "Inherit Schema", so no need to define an explicit
> schema anywhere.
>
> This has the added advantage of adding in the opportunity to conduct other
> business logic at the same time, e.g. confirming the data you're processing
> is indeed json or counting the number of records in a flowfile, etc.
>
> I couldn't give much of a statement about the comparative
> accuracy/efficiency of the different approaches though in afraid.
>
>
> Cheers,
>
> Chris Sampson
>
> On Mon, 13 Apr 2020, 18:34 Joe Witt, <joe.witt@gmail.com> wrote:
>
> > Sue,
> >
> > Due to space issues we had to remove certain nars from the convenience
> > binary we ship.  They're listed here
> > https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance. We
> > still do publish the nars and you can use them but you have to add them
> to
> > your install.
> >
> > Thanks
> > Joe
> >
> > On Mon, Apr 13, 2020 at 1:26 PM Tholen, Sue (Chief Data Office) <
> > Susan.Tholen@thehartford.com> wrote:
> >
> > > I thought that version management ensured backward compatibility but
> > > cannot find an equivalent function in v1.10 for "infer avro schema".
> > > Nor could I find anything in the release notes about it going away...
> > > Does a similar capability exist?
> > > ************************************************************
> > > This communication, including attachments, is for the exclusive use of
> > > addressee and may contain proprietary, confidential and/or privileged
> > > information.  If you are not the intended recipient, any use, copying,
> > > disclosure, dissemination or distribution is strictly prohibited.  If
> you
> > > are not the intended recipient, please notify the sender immediately by
> > > return e-mail, delete this communication and destroy all copies.
> > > ************************************************************
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message