incubator-drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Phillips <sphill...@maprtech.com>
Subject Re: Validate your JSON files
Date Mon, 24 Nov 2014 23:09:35 GMT
No, Drill uses jackson to parse the json as a stream. It's fine if the json
record has newline characters.

Your validation tool is still useful, in the case where each json record is
contained in a single line, which is common. Just be aware that it won't
work in all cases.

On Mon, Nov 24, 2014 at 3:04 PM, Christopher Matta <cmatta@mapr.com> wrote:

> Steven,
> Yes it does, doesn't Drill  also require that the entire JSON record be on
> a single line?
>
> I wrote this for situations when the data set is too large to paste into a
> web-based validator.
>
> Chris Matta
> cmatta@mapr.com
> 215-701-3146
>
> On Mon, Nov 24, 2014 at 6:01 PM, Steven Phillips <sphillips@maprtech.com>
> wrote:
>
> > Christopher,
> >
> > Does your validator require that the entire json record be on a single
> > line?
> >
> > On Mon, Nov 24, 2014 at 2:57 PM, Aman Sinha <asinha@maprtech.com> wrote:
> >
> > > BTW, there's a web based validator called jsonlint.com whose source is
> > > available at :  https://github.com/arc90/jsonlintdotcom
> > >
> > > On Mon, Nov 24, 2014 at 2:07 PM, Christopher Matta <cmatta@mapr.com>
> > > wrote:
> > >
> > > > I’ve been running across errors in Drill when a JSON record is
> invalid.
> > > To
> > > > reduce the number of these errors, I wrote this small, simple
> > application
> > > > that will open a specified file, check if each line is a valid JSON
> > > record,
> > > > and error if it’s not:
> > > >
> > > > https://github.com/cjmatta/jsonr
> > > >
> > > > Usage:
> > > >
> > > > [cmatta@ip-172-16-1-173 jsonar]$ ./jsonar -f
> > > > ../tweets/2014/11/24/21/tweets.json
> > > > Checking for valid JSON in ../tweets/2014/11/24/21/tweets.json
> > > > CWARNING:root:JSON load error on line 16640 of
> > > > ../tweets/2014/11/24/21/tweets.json
> > > > WARNING:root:JSON load error on line 16641 of
> > > > ../tweets/2014/11/24/21/tweets.json
> > > > WARNING:root:JSON load error on line 16642 of
> > > > ../tweets/2014/11/24/21/tweets.json
> > > > Checking line 17000
> > > > Done.
> > > >
> > > > Please check it out, use it, contribute back if there’s something
> > broken
> > > or
> > > > missing.
> > > >
> > > > Chris Matta
> > > > cmatta@mapr.com
> > > > 215-701-3146
> > > > ​
> > > >
> > >
> >
> >
> >
> > --
> >  Steven Phillips
> >  Software Engineer
> >
> >  mapr.com
> >
>



-- 
 Steven Phillips
 Software Engineer

 mapr.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message