From dev-return-18188-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Tue Oct 2 03:17:11 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 06274180660 for ; Tue, 2 Oct 2018 03:17:10 +0200 (CEST) Received: (qmail 79370 invoked by uid 500); 2 Oct 2018 01:17:09 -0000 Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list dev@nifi.apache.org Delivered-To: moderator for dev@nifi.apache.org Received: (qmail 70525 invoked by uid 99); 2 Oct 2018 01:12:50 -0000 X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.812 X-Spam-Level: X-Spam-Status: No, score=-0.812 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=protonmail.com Date: Tue, 02 Oct 2018 01:12:37 +0000 To: Matt Burgess From: Ed J Cc: "dev@nifi.apache.org" Reply-To: Ed J Subject: Re: POJO to Record Message-ID: In-Reply-To: References: Feedback-ID: elo0wzhlwS1qNaa_edIVGHVxNYGDf2tCbSsMoqqUsD4mI780XMNAeO-t4zZnonpp8d6ZkV_sCHnyePRbeRaAuA==:Ext:ProtonMail MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Matt, Thanks for the tips. Will look into those options this week. Many of the fields are dynamic and our XML schema doesn't really qualify fo= r XMLReader based on what I can glean from the docs, for example: "Records = are expected in the second level of XML data, embedded in an enclosing root= tag." The other rationale for building an external custom parser to transform the= XML is that it provides the flexibility to parse from other tools/apps in = addition to NiFi. -Ed =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Me= ssage =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 On Friday, September 28, 2018 11:49 AM, Matt Burgess = wrote: > Moving users list to BCC and adding devs list, as this question is > better suited for the devs. > > Ed, > > Does XMLReader not parse your XML? I guess you'd need to know the > schema so if you don't (especially if it is dynamic) then you might be > faced with a custom processor as you described. > > Take a look at DataTypeUtils, it has methods toRecord() and > convertRecordFieldtoObject() which helps you convert from POJO > <->Record. The JoltTransformRecord processor uses these, so you can > have a look at that code for an example. > > Regards, > Matt > > On Fri, Sep 28, 2018 at 11:28 AM Ed J edja@protonmail.com wrote: > > > Hi, > > I'm ingesting some XML with an overly complicated structure and I've bu= ild a custom java DOM parser to wrangle it into a saner form. (Is that frow= ned upon - should I just use the built-in processors to wrangle it?) > > So my question is, I've parsed the XML into a simple POJO, how do I get= that pojo into the next processor as a 'Record'. My custom nifi processor'= s onTrigger looks like this: > > MyParser parser =3D new MyParser(); > > MyPojo pojo =3D parser.parse(flowFileContents); > > // TODO: convert pojo to record > > Record myrecord =3D pojo.toRecord(); // how to do this? > > session.transfer(myrecord, REL_SUCCESS) > > I'm guessing I could convert the pojo to JSON, write that out to the ne= xt stage, and then use a JsonRecordReader to convert the JSON into a record= . However, if I can go straight from pojo to record, that seems more effici= ent - just not sure how to go about it. Thanks.