incubator-drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Matta <cma...@mapr.com>
Subject Re: Validate your JSON files
Date Tue, 25 Nov 2014 12:31:29 GMT
Ted, I'll take a look! Thanks.

Chris Matta
cmatta@mapr.com
215-701-3146

On Tue, Nov 25, 2014 at 6:27 AM, Ted Dunning <ted.dunning@gmail.com> wrote:

> Chris,
>
> Your tool could be updated to use Jackson and would then have the exact
> same semantics as Drill.
>
> It is still great as it is.... just could be slightly greater.
>
>
> On Mon, Nov 24, 2014 at 11:09 PM, Steven Phillips <sphillips@maprtech.com>
> wrote:
>
> > No, Drill uses jackson to parse the json as a stream. It's fine if the
> json
> > record has newline characters.
> >
> > Your validation tool is still useful, in the case where each json record
> is
> > contained in a single line, which is common. Just be aware that it won't
> > work in all cases.
> >
> > On Mon, Nov 24, 2014 at 3:04 PM, Christopher Matta <cmatta@mapr.com>
> > wrote:
> >
> > > Steven,
> > > Yes it does, doesn't Drill  also require that the entire JSON record be
> > on
> > > a single line?
> > >
> > > I wrote this for situations when the data set is too large to paste
> into
> > a
> > > web-based validator.
> > >
> > > Chris Matta
> > > cmatta@mapr.com
> > > 215-701-3146
> > >
> > > On Mon, Nov 24, 2014 at 6:01 PM, Steven Phillips <
> sphillips@maprtech.com
> > >
> > > wrote:
> > >
> > > > Christopher,
> > > >
> > > > Does your validator require that the entire json record be on a
> single
> > > > line?
> > > >
> > > > On Mon, Nov 24, 2014 at 2:57 PM, Aman Sinha <asinha@maprtech.com>
> > wrote:
> > > >
> > > > > BTW, there's a web based validator called jsonlint.com whose
> source
> > is
> > > > > available at :  https://github.com/arc90/jsonlintdotcom
> > > > >
> > > > > On Mon, Nov 24, 2014 at 2:07 PM, Christopher Matta <
> cmatta@mapr.com>
> > > > > wrote:
> > > > >
> > > > > > I’ve been running across errors in Drill when a JSON record
is
> > > invalid.
> > > > > To
> > > > > > reduce the number of these errors, I wrote this small, simple
> > > > application
> > > > > > that will open a specified file, check if each line is a valid
> JSON
> > > > > record,
> > > > > > and error if it’s not:
> > > > > >
> > > > > > https://github.com/cjmatta/jsonr
> > > > > >
> > > > > > Usage:
> > > > > >
> > > > > > [cmatta@ip-172-16-1-173 jsonar]$ ./jsonar -f
> > > > > > ../tweets/2014/11/24/21/tweets.json
> > > > > > Checking for valid JSON in ../tweets/2014/11/24/21/tweets.json
> > > > > > CWARNING:root:JSON load error on line 16640 of
> > > > > > ../tweets/2014/11/24/21/tweets.json
> > > > > > WARNING:root:JSON load error on line 16641 of
> > > > > > ../tweets/2014/11/24/21/tweets.json
> > > > > > WARNING:root:JSON load error on line 16642 of
> > > > > > ../tweets/2014/11/24/21/tweets.json
> > > > > > Checking line 17000
> > > > > > Done.
> > > > > >
> > > > > > Please check it out, use it, contribute back if there’s something
> > > > broken
> > > > > or
> > > > > > missing.
> > > > > >
> > > > > > Chris Matta
> > > > > > cmatta@mapr.com
> > > > > > 215-701-3146
> > > > > > ​
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >  Steven Phillips
> > > >  Software Engineer
> > > >
> > > >  mapr.com
> > > >
> > >
> >
> >
> >
> > --
> >  Steven Phillips
> >  Software Engineer
> >
> >  mapr.com
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message