Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of himanshuvj@gmail.com
 designates 209.85.160.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <1786E485-FE5A-44A1-9B60-8F55B38C3014@89clouds.com>
References: 
 <CAKG3H40shQCNK3CTAR-5myphLG5Bp0yy8D0=OjSfKcRXcNvN7A@mail.gmail.com>
 <1786E485-FE5A-44A1-9B60-8F55B38C3014@89clouds.com>
From: Himanshu Vijay <himanshuvj@gmail.com>
Date: Thu, 3 May 2012 15:47:11 -0700
Message-ID: 
 <CAKG3H40QNBCseL2riMt0Rj5-WTZOE8Sm0tRfDiGTFuRp9g-Hvg@mail.gmail.com>
Subject: Re: Splitting data input to Distcp
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=e89a8f923aa4d77d6b04bf2996f6

--e89a8f923aa4d77d6b04bf2996f6
Content-Type: text/plain; charset=ISO-8859-1

Pedro,

Thanks for the response. Unfortunately I am running it on in-house cluster
and from there I need to upload to S3.

-Himanshu

On Wed, May 2, 2012 at 2:03 PM, Pedro Figueiredo <pfig@89clouds.com> wrote:

>
> On 2 May 2012, at 18:29, Himanshu Vijay wrote:
>
> > Hi,
> >
> > I have 100 files each of ~3 GB. I need to distcp them to S3 but copying
> > fails because of large size of files. The files are not gzipped so they
> are
> > splittable. Is there a way or property to tell Distcp to first split the
> > input files into let's say 200 MB or N lines each before copying to
> > destination.
> >
>
> Assuming you're using EMR, use s3distcp:
>
>
> http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/UsingEMR_s3distcp.html
>
> In any case, that's strange because S3's limit is 5GB per PUT request;
> again if you're running on EMR, try starting your cluster with
>
> --bootstrap-action
> s3://elasticmapreduce/bootstrap-actions/configure-hadoop \
>  --args
> "-c,fs.s3n.multipart.uploads.enabled=true,-c,fs.s3n.multipart.uploads.split.size=524288000"
>
> (or add those to whatever parameters you currently use).
>
> Going back to plain distcp, I'm not sure about what the -sizelimit option
> does, as I've never used it.
>
> If push comes to shove, seeing as you have a Hadoop cluster, running a job
> to write the files to S3 with compression enabled is always an option :)
>
> Cheers,
>
> Pedro Figueiredo
> Skype: pfig.89clouds
> http://89clouds.com/ - Big Data Consulting
>
>
>
>
>


-- 
-Himanshu Vijay

--e89a8f923aa4d77d6b04bf2996f6--