spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean Georges Perrin <...@jgp.net>
Subject Re: JSON Arrays and Spark
Date Mon, 10 Oct 2016 18:15:42 GMT
Thanks!

I am ok with strict rules (despite being French), but even:
[{
	"red": "#f00", 
	"green": "#0f0"
},{
	"red": "#f01", 
	"green": "#0f1"
}]

is not going through…

Is there a way to see what he does not like?

the JSON parser has been pretty good to me until recently.


> On Oct 10, 2016, at 12:59 PM, Sudhanshu Janghel <> wrote:
> 
> As far as my experience goes spark can parse only certain types of Json correctly not
all and has strict Parsing rules unlike python
> 
> 
> On Oct 10, 2016 6:57 PM, "Jean Georges Perrin" <jgp@jgp.net <mailto:jgp@jgp.net>>
wrote:
> Hi folks,
> 
> I am trying to parse JSON arrays and it’s getting a little crazy (for me at least)…
> 
> 1)
> If my JSON is:
> {"vals":[100,500,600,700,800,200,900,300]}
> 
> I get:
> +--------------------+
> |                vals|
> +--------------------+
> |[100, 500, 600, 7...|
> +--------------------+
> 
> root
>  |-- vals: array (nullable = true)
>  |    |-- element: long (containsNull = true)
> 
> and I am :)
> 
> 2)
> If my JSON is:
> [100,500,600,700,800,200,900,300]
> 
> I get:
> +--------------------+
> |     _corrupt_record|
> +--------------------+
> |[100,500,600,700,...|
> +--------------------+
> 
> root
>  |-- _corrupt_record: string (nullable = true)
> 
> Both are legit JSON structures… Do you think that #2 is a bug?
> 
> jg
> 
> 
> 
> 
> 
> 
> Disclaimer: The information in this email is confidential and may be legally privileged.
Access to this email by anyone other than the intended addressee is unauthorized. If you are
not the intended recipient of this message, any review, disclosure, copying, distribution,
retention, or any action taken or omitted to be taken in reliance on it is prohibited and
may be unlawful.


Mime
View raw message