Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
Received-SPF: pass (athena.apache.org: domain of junxian.yan@gmail.com
 designates 209.85.210.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=mpvVxx1wnI3lAIunr4JORUBUdSiJo6v5gNbfT42AnkcY2kinqYWd1igRypn1yqNRNW
         SwbtkcIVlPDn0UBgq0dbw5G0us9u2Q6xjsh8GhdkxPW8rsvznmTKZ8ubKSLgyYo+JHap
         ke8SNaq+ZwWRFfkoo2aR7cQxNz2dQdsgk4eow=
MIME-Version: 1.0
Date: Tue, 31 May 2011 02:55:33 -0700
Message-ID: <BANLkTimxGz3J4AOOGkc7bcT1U8zdjiR68g@mail.gmail.com>
Subject: question about number of map tasks for small file
From: Junxian Yan <junxian.yan@gmail.com>
To: user@hive.apache.org
Content-Type: multipart/alternative; boundary=bcaec53963beb3bf7e04a48f677b

--bcaec53963beb3bf7e04a48f677b
Content-Type: text/plain; charset=ISO-8859-1

Hi Guys

I use flume to store log file , and use hive to query.

Flume always store the small file with suffix .seq Now I have over 35
thousand seq files. Every time when I launch query script, 35 thousand map
tasks will be created and it's so long time to wait for completing.

I also try to set CombineHiveInputFormat, but if I set this option, it seems
the task will be executed slowly. Because total size of the data folder over
700M.  Now in my testing env, I only have 3 data nodes. I also tried to add
mapred.map.tasks=5 after the CombineHiveInputFormat setting, seems doesn't
work. There's alway only one map task if set CombineHiveInputFormat.

Can you plz show me a solution in which I can set map task number freely

BTW: version for hadoop is 20 and hive is 0.5

Richard

--bcaec53963beb3bf7e04a48f677b
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Guys<div><br></div><div>I use flume to store log file , and use hive to =
query.=A0</div><div><br></div><div>Flume always store the small file with s=
uffix .seq Now I have over 35 thousand seq files. Every time when I launch =
query script, 35 thousand map tasks will be created and it&#39;s so long ti=
me to wait for completing.=A0</div>
<div><br></div><div>I also try to set=A0CombineHiveInputFormat, but if I se=
t this option, it seems the task will be executed slowly. Because total siz=
e of the data folder over 700M. =A0Now in my testing env, I only have 3 dat=
a nodes. I also tried to add mapred.map.tasks=3D5 after the=A0CombineHiveIn=
putFormat setting, seems doesn&#39;t work. There&#39;s alway only one map t=
ask if set=A0CombineHiveInputFormat.</div>
<div><br></div><div>Can you plz show me a solution in which I can set map t=
ask number freely</div><div><br></div><div>BTW: version for hadoop is 20 an=
d hive is 0.5</div><div><br></div><div>Richard</div>

--bcaec53963beb3bf7e04a48f677b--