Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of dontariq@gmail.com designates
 209.85.212.49 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAGmGr9Q59+aCY9iwdA_q6x0wYtqeOVS0t0DJfgyVw-LjotWoQ@mail.gmail.com>
References: 
 <CAAGmGr9Q59+aCY9iwdA_q6x0wYtqeOVS0t0DJfgyVw-LjotWoQ@mail.gmail.com>
From: Mohammad Tariq <dontariq@gmail.com>
Date: Wed, 16 Jan 2013 21:08:26 +0530
Message-ID: 
 <CAMVC6RPsCxj8G-0MGjy4NVV4NzgVpfzzKpBtfPvOq0N6mOEJMg@mail.gmail.com>
Subject: Re: Loading file to HDFS with custom chunk structure
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=f46d04339ce4bb718d04d369adb7

--f46d04339ce4bb718d04d369adb7
Content-Type: text/plain; charset=ISO-8859-1

Hello there,

    You don't have to split the file. When you push anything into
HDFS, it automatically gets splitted into small chunks of uniform
size (64MB or 128MB usually). And the MapReduce frameworks
ensures that each block is processed locally on the node where it
is located.

Do you have any specific requirement??

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com


On Wed, Jan 16, 2013 at 9:01 PM, Kaliyug Antagonist <
kaliyugantagonist@gmail.com> wrote:

> I want to load a SegY <http://en.wikipedia.org/wiki/SEG_Y> file onto HDFS
> of a 3-node Apache Hadoop cluster.
>
> To summarize, the SegY file consists of :
>
>    1. 3200 bytes *textual header*
>    2. 400 bytes *binary header*
>    3. Variable bytes *data*
>
> The 99.99% size of the file is due to the variable bytes data which is
> collection of thousands of contiguous traces. For any SegY file to make
> sense, it must have the textual header+binary header+at least one trace of
> data. What I want to achieve is to split a large SegY file onto the Hadoop
> cluster so that a smaller SegY file is available on each node for local
> processing.
>
> The scenario is as follows:
>
>    1. The SegY file is large in size(above 10GB) and is resting on the
>    local file system of the NameNode machine
>    2. The file is to be split on the nodes in such a way each node has a
>    small SegY file with a strict structure - 3200 bytes *textual header*+ 400 bytes
>    *binary header* + variable bytes *data*As obvious, I can't blindly use
>    FSDataOutputStream or hadoop fs -copyFromLocal as this may not ensure the
>    format in which the chunks of the larger file are required
>
> Please guide me as to how I must proceed.
>
> Thanks and regards !
>

--f46d04339ce4bb718d04d369adb7
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hello there,<div><br></div><div style>=A0 =A0 You don&#39;=
t have to split the file. When you push anything into=A0</div><div style>HD=
FS,=A0it automatically gets splitted into small chunks of uniform=A0</div><=
div style>

size=A0(64MB or 128MB usually). And the MapReduce frameworks</div><div styl=
e>ensures that each block is processed locally on the node where it</div><d=
iv style>is located.</div><div style><br></div><div style>Do you have any s=
pecific requirement??</div>

</div><div class=3D"gmail_extra"><br clear=3D"all"><div><div dir=3D"ltr">Wa=
rm Regards,<div>Tariq</div><div><a href=3D"https://mtariq.jux.com/" target=
=3D"_blank">https://mtariq.jux.com/</a><br></div><div><a href=3D"http://clo=
udfront.blogspot.com" target=3D"_blank">cloudfront.blogspot.com</a><br>

</div></div></div>
<br><br><div class=3D"gmail_quote">On Wed, Jan 16, 2013 at 9:01 PM, Kaliyug=
 Antagonist <span dir=3D"ltr">&lt;<a href=3D"mailto:kaliyugantagonist@gmail=
.com" target=3D"_blank">kaliyugantagonist@gmail.com</a>&gt;</span> wrote:<b=
r><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:=
1px #ccc solid;padding-left:1ex">

<div dir=3D"ltr"><p>I want to load a <a href=3D"http://en.wikipedia.org/wik=
i/SEG_Y" rel=3D"nofollow" target=3D"_blank"><font color=3D"#0066cc">SegY</f=
ont></a> file onto HDFS of a 3-node Apache Hadoop cluster.</p><p>To summari=
ze, the SegY file consists of :</p>


<ol><li>3200 bytes <strong>textual header</strong></li><li>400 bytes <stron=
g>binary header</strong></li><li>Variable bytes <strong>data</strong></li><=
/ol><p>The 99.99% size of the file is due to the variable bytes data which =
is collection of thousands of contiguous traces. For any SegY file to make =
sense, it must have the textual header+binary header+at least one trace of =
data. What I want to achieve is to split a large SegY file onto the Hadoop =
cluster so that a smaller SegY file is available on each node for local pro=
cessing.</p>


<p>The scenario is as follows:</p><ol><li>The SegY file is large in size(ab=
ove 10GB) and is resting on the local file system of the NameNode machine</=
li><li>The file is to be split on the nodes in such a way each node has a s=
mall SegY file with a strict structure - 3200 bytes <strong>textual header<=
/strong> + 400 bytes <strong>binary header</strong> + variable bytes <stron=
g>data</strong>As obvious, I can&#39;t blindly use FSDataOutputStream or ha=
doop fs -copyFromLocal as this may not ensure the format in which the chunk=
s of the larger file are required</li>


</ol><p>Please guide me as to how I must proceed.</p><p>Thanks and regards =
!</p></div>
</blockquote></div><br></div>

--f46d04339ce4bb718d04d369adb7--