nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aldrin Piri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NIFI-853) Create Processors to put JSON data to a Relational Database
Date Wed, 19 Aug 2015 01:20:46 GMT

    [ https://issues.apache.org/jira/browse/NIFI-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14702268#comment-14702268
] 

Aldrin Piri commented on NIFI-853:
----------------------------------

I think things are close, but playing around with some things a bit, there is one thing I
think needs to be addressed and another that is questionable in terms of necessity.  Both
of these are centered around multiple inserts.

Consider the following json adapted from the persons.json provided in tests:

{code}
[
  {
    "id": 1,
    "name": "Mark",
    "code": 48
  },
  {
    "id": 2,
    "code": 48,
    "name": "George"
  },
  {
    "code": 21,
    "id": 3,
    "name": "Harry"
  },
  {
    "id": 4,
    "name": "Julie",
    "code": 48
  },
  {
    "puppy": 12,
    "code": 16,
    "name": "Frank Henry",
    "id": 82
  }
]
{code}

# JSON objects have no strict ordering of their elements to be considered equivalent.  Accordingly,
while all items have the requisite fields needed, items are currently just iterated across
in order.  I think the fix is to create a map of anticipated fields.
# Coincides with the concept of the prior item and would make use of the map, but when presented
with an array of objects, how do we remedy items that are in excess of the representative
item?  It seems like this should likely just be an error condition and the associated flow
file routed off.  We can't be sure of the intent of the user.  From the segment above, if
the object with id 82 was first, all other elements would appear to be short a field whereas,
in its current configuration, it appears to have an excess field. 

> Create Processors to put JSON data to a Relational Database
> -----------------------------------------------------------
>
>                 Key: NIFI-853
>                 URL: https://issues.apache.org/jira/browse/NIFI-853
>             Project: Apache NiFi
>          Issue Type: Task
>          Components: Extensions
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>             Fix For: 0.3.0
>
>         Attachments: 0001-NIFI-853-Added-processors-ConvertFlatJSONToSQL-PutSQ.patch,
0002-NIFI-853-Made-updates-to-processors.patch
>
>
> Most of the discussion/design for these processors happened in the comments of NIFI-293,
which was the initial ticket for implementing JDBC functionality in NiFi, but was closed in
a previous version, so this ticket was created to do the work.
> The idea is to have a processor that will take in FlowFiles whose contents are arbitrary
SQL INSERT/UPDATE commands. The commands can be parameterized with the parameters' values
and types in FlowFile attributes.
> We then should have a processor that converts a JSON document into a SQL command to either
update or insert data into a database table. We will also want some other processors in the
future probably to handle other data types, such as converting XML, CSV, Avro, etc. into SQL
commands.
> This breakout gives us a nice coherence to the "do only one thing and do it well" principle
by separating the logic of handling all of the incoming formats from the logic of updating
the database.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message