Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@crunch.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAKFfB9=up+gmPFrmsjCsOQUfBjdV0iEenjpdBBsexOVveqQ9Gw@mail.gmail.com>
References: <3035AB48-DE57-47FA-BBF1-9054D149AF00@gmail.com>
 <62963EB9-2FE3-4450-B33C-7900D5096A50@gmail.com>
 <CAKFfB9=up+gmPFrmsjCsOQUfBjdV0iEenjpdBBsexOVveqQ9Gw@mail.gmail.com>
From: Josh Wills <josh.wills@gmail.com>
Date: Fri, 26 Feb 2016 15:43:14 -0800
Message-ID: 
 <CANb5z2+NOVKZmO0KO0cMte9iPiy1twa9-y2jKY=izSfvON7avw@mail.gmail.com>
Subject: Re: Processing splittable inputs
To: dev <dev@crunch.apache.org>
Content-Type: multipart/alternative; boundary=001a11c304baffe399052cb4dd2c

--001a11c304baffe399052cb4dd2c
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Yeah, I suspect the Source-property approach is the right thing here.

On Fri, Feb 26, 2016 at 3:37 PM, Micah Whitacre <mkwhit@gmail.com> wrote:

> Where are you trying to specify them?  Inside a DoFn?  Prior to
> constructing the MRPipeline?
>
> I'd suggest trying either:
> 1. Setting those values on the initial Configuration object you pass to t=
he
> MRPipeline
> 2. Setting them as Source specific properties[1] on the source itself.
>
> The latter approach might be better if you are reading a lot of different
> sources into your pipeline and don't want to affect them all.
>
> [1] -
>
> http://crunch.apache.org/apidocs/0.12.0/org/apache/crunch/Source.html#inp=
utConf(java.lang.String,%20java.lang.String)
>
> On Fri, Feb 26, 2016 at 5:17 PM, Ben Juhn <benjijuhn@gmail.com> wrote:
>
> > The data isn=E2=80=99t compressed.  The parameters aren=E2=80=99t showi=
ng up in the job
> > configuration either.
> >
> >
> > > On Feb 25, 2016, at 5:15 PM, Ben Juhn <benjijuhn@gmail.com> wrote:
> > >
> > > Hello there,
> > >
> > > I haven=E2=80=99t been able to get crunch to split inputs into multip=
le
> > mappers.  Currently it=E2=80=99s giving me one mapper per text file, ev=
en though
> > they=E2=80=99re 1GB each.  I=E2=80=99ve tried supplying split.maxsize o=
n the command line
> > and in the DoFn implementation:
> > >
> > > @Override
> > > public void configure(Configuration conf) {
> > > conf.set("crunch.combine.file.size", "67108864");
> > > conf.set("mapreduce.input.fileinputformat.split.maxsize", "67108864")=
;
> > > conf.set("mapreduce.input.fileinputformat.split.minsize", "67108864")=
;
> > > }
> > >
> > > Any suggestions?
> > >
> > > Thanks,
> > > Ben
> > >
> >
> >
>

--001a11c304baffe399052cb4dd2c--