I generally use Play Framework Api's for comple json structures.

https://www.playframework.com/documentation/2.5.x/ScalaJson#Json

On Wed, Oct 12, 2016 at 11:34 AM, Kappaganthu, Sivaram (ES) <Sivaram.Kappaganthu@adp.com> wrote:

Hi,

 

Does this mean that handling any Json with kind of below schema  with spark is not a good fit?? I have requirement to parse the below Json that spans across multiple lines. Whats the best way to parse the jsns of this kind?? Please suggest.

 

root

|-- maindate: struct (nullable = true)

|    |-- mainidnId: string (nullable = true)

|-- Entity: array (nullable = true)

|    |-- element: struct (containsNull = true)

|    |    |-- Profile: struct (nullable = true)

|    |    |    |-- Kind: string (nullable = true)

|    |    |-- Identifier: string (nullable = true)

|    |    |-- Group: array (nullable = true)

|    |    |    |-- element: struct (containsNull = true)

|    |    |    |    |-- Period: struct (nullable = true)

|    |    |    |    |    |-- pid: string (nullable = true)

|    |    |    |    |    |-- pDate: string (nullable = true)

|    |    |    |    |    |-- quarter: long (nullable = true)

|    |    |    |    |    |-- labour: array (nullable = true)

|    |    |    |    |    |    |-- element: struct (containsNull = true)

|    |    |    |    |    |    |    |-- category: string (nullable = true)

|    |    |    |    |    |    |    |-- id: string (nullable = true)

|    |    |    |    |    |    |    |-- person: struct (nullable = true)

|    |    |    |    |    |    |    |    |-- address: array (nullable = true)

|    |    |    |    |    |    |    |    |    |-- element: struct (containsNull = true)

|    |    |    |    |    |    |    |    |    |    |-- city: string (nullable = true)

|    |    |    |    |    |    |    |    |    |    |-- line1: string (nullable = true)

|    |    |    |    |    |    |    |    |    |    |-- line2: string (nullable = true)

|    |    |    |    |    |    |    |    |    |    |-- postalCode: string (nullable = true)

|    |    |    |    |    |    |    |    |    |    |-- state: string (nullable = true)

|    |    |    |    |    |    |    |    |    |    |-- type: string (nullable = true)

|    |    |    |    |    |    |    |    |-- familyName: string (nullable = true)

|    |    |    |    |    |    |    |-- tax: array (nullable = true)

|    |    |    |    |    |    |    |    |-- element: struct (containsNull = true)

|    |    |    |    |    |    |    |    |    |-- code: string (nullable = true)

|    |    |    |    |    |    |    |    |    |-- qwage: double (nullable = true)

|    |    |    |    |    |    |    |    |    |-- qvalue: double (nullable = true)

|    |    |    |    |    |    |    |    |    |-- qSubjectvalue: double (nullable = true)

|    |    |    |    |    |    |    |    |    |-- qfinalvalue: double (nullable = true)

|    |    |    |    |    |    |    |    |    |-- ywage: double (nullable = true)

|    |    |    |    |    |    |    |    |    |-- yalue: double (nullable = true)

|    |    |    |    |    |    |    |    |    |-- ySubjectvalue: double (nullable = true)

|    |    |    |    |    |    |    |    |    |-- yfinalvalue: double (nullable = true)

|    |    |    |    |    |    |    |-- tProfile: array (nullable = true)

|    |    |    |    |    |    |    |    |-- element: struct (containsNull = true)

|    |    |    |    |    |    |    |    |    |-- isExempt: boolean (nullable = true)

|    |    |    |    |    |    |    |    |    |-- jurisdiction: struct (nullable = true)

|    |    |    |    |    |    |    |    |    |    |-- code: string (nullable = true)

|    |    |    |    |    |    |    |    |    |-- maritalStatus: string (nullable = true)

|    |    |    |    |    |    |    |    |    |-- numberOfDeductions: long (nullable = true)

|    |    |    |    |    |    |    |-- wDate: struct (nullable = true)

|    |    |    |    |    |    |    |    |-- originalHireDate: string (nullable = true)

|    |    |    |    |    |-- year: long (nullable = true)

 

 

From: Luciano Resende [mailto:luckbr1975@gmail.com]
Sent: Monday, October 10, 2016 11:39 PM
To: Jean Georges Perrin
Cc: user @spark
Subject: Re: JSON Arrays and Spark

 

Particularly the note at the required format :

Note that the file that is offered as a json file is not a typical JSON file. Each line must contain a separate, self-contained valid JSON object. As a consequence, a regular multi-line JSON file will most often fail.

 

On Mon, Oct 10, 2016 at 9:57 AM, Jean Georges Perrin <jgp@jgp.net> wrote:

Hi folks,

 

I am trying to parse JSON arrays and it’s getting a little crazy (for me at least)…

 

1)

If my JSON is:

{"vals":[100,500,600,700,800,200,900,300]}

 

I get:

+--------------------+

|                vals|

+--------------------+

|[100, 500, 600, 7...|

+--------------------+

 

root

 |-- vals: array (nullable = true)

 |    |-- element: long (containsNull = true)

 

and I am :)

 

2)

If my JSON is:

[100,500,600,700,800,200,900,300]

 

I get:

+--------------------+

|     _corrupt_record|

+--------------------+

|[100,500,600,700,...|

+--------------------+

 

root

 |-- _corrupt_record: string (nullable = true)

 

Both are legit JSON structures… Do you think that #2 is a bug?

 

jg

 

 

 

 

 




--


This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.