Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <5cb2471f-0529-af9a-3312-68371a256dc9@imixs.com>
References: <5cb2471f-0529-af9a-3312-68371a256dc9@imixs.com>
From: Alexey Eremihin <a.eremihin@corp.badoo.com.INVALID>
Date: Mon, 4 Sep 2017 22:32:27 +0300
Message-ID: <CAGP94Wnr-CnZKhPjcMpk8THkUh8KoA=xoXta2BXLq9_JTqhmcQ@mail.gmail.com>
Subject: Re: Is Hadoop basically not suitable for a photo archive?
To: Ralph Soika <ralph.soika@imixs.com>
Cc: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary="f403045fa370c4b7810558622b47"
archived-at: Mon, 04 Sep 2017 19:32:39 -0000

--f403045fa370c4b7810558622b47
Content-Type: text/plain; charset="UTF-8"

Hi Ralph,
In general Hadoop is able to store such data. And even Har archives can be
used with conjunction with WebHDFS (by passing offset and limit
attributes). What are your reading requirements? FS meta data are not
distributed and reading the data is limited by the HDFS NameNode server
performance. So if you would like to download files with high RPS that
would not work well.

On Monday, September 4, 2017, Ralph Soika <ralph.soika@imixs.com> wrote:

> Hi,
>
> I know that the issue around the small-file problem was asked frequently,
> not only in this mailing list.
> I also have read already some books about Haddoop and I also started to
> work with Hadoop. But still I did not really understand if Hadoop is the
> right choice for my goals.
>
> To simplify my problem domain I would like to use the use case of a photo
> archive:
>
> - An external application produces about 10 million photos in one year.
> The files contain important business critical data.
> - A single photo file has a size between 1 and 10 MB.
> - The photos need to be stored over several years (10-30 years).
> - The data store should support replication over several servers.
> - A checksum-concept is needed to guarantee the data integrity of all
> files over a long period of time.
> - To write and read the files a Rest API is preferred.
>
> So far Hadoop seems to be absolutely the perfect solution. But my last
> requirement seems to throw Hadoop out of the race.
>
> - The photos need to be readable with very short latency from an external
> enterprise application
>
> With Hadoop HDFS and the Web Proxy everything seems perfect. But it seems
> that most of the Hadoop experts advise against this usage if the size of my
> data files (1-10 MB) are well below the Hadoop block size of 64 or 128 MB.
>
> I think I understood the concepts of HAR or sequential files.
> But if I pack, for example, my files together in a large file of many
> Gigabytes it is impossible to access one single photo from the Hadoop
> repository in a reasonable time. It makes no sense in my eyes to pack
> thousands of files into a large file just so that Hadoop jobs can handle it
> better. To simply access a single file from a web interface - as in my case
> - it seems to be all counterproductive.
>
> So my question is: Is Hadoop only feasible to archive large Web-server log
> files and not designed to handle big archives of small files with also
> business critical data?
>
>
> Thanks for your advice in advance.
>
> Ralph
> --
>
>

--f403045fa370c4b7810558622b47
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Ralph,=C2=A0<div>In general Hadoop is able to store such data. And even =
Har archives can be used with conjunction with WebHDFS (by passing offset a=
nd limit attributes). What are your reading requirements? FS meta data are =
not distributed and reading the data is limited by the HDFS NameNode server=
 performance. So if you would like to download files with high RPS that wou=
ld not work well.<br><br>On Monday, September 4, 2017, Ralph Soika &lt;<a h=
ref=3D"mailto:ralph.soika@imixs.com">ralph.soika@imixs.com</a>&gt; wrote:<b=
r><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:=
1px #ccc solid;padding-left:1ex">
 =20

   =20
 =20
  <div text=3D"#333333" bgcolor=3D"#FFFFFF">
    <p>Hi,</p>
    <p>I know that the issue around the small-file problem was asked
      frequently, not only in this mailing list.<br>
      I also have read already some books about Haddoop and I also
      started to work with Hadoop. But still I did not really understand
      if Hadoop is the right choice for my goals.<br>
      <br>
      To simplify my problem domain I would like to use the use case of
      a photo archive:<br>
      <br>
      - An external application produces about 10 million photos in one
      year. The files contain important business critical data.<br>
      - A single photo file has a size between 1 and 10 MB. <br>
      - The photos need to be stored over several years (10-30 years).<br>
      - The data store should support replication over several servers.
      <br>
      - A checksum-concept is needed to guarantee the data integrity of
      all files over a long period of time.<br>
      - To write and read the files a Rest API is preferred. <br>
      <br>
      So far Hadoop seems to be absolutely the perfect solution. But my
      last requirement seems to throw Hadoop out of the race.<br>
      <br>
      - The photos need to be readable with very short latency from an
      external enterprise application<br>
      <br>
      With Hadoop HDFS and the Web Proxy everything seems perfect. But
      it seems that most of the Hadoop experts advise against this usage
      if the size of my data files (1-10 MB) are well below the Hadoop
      block size of 64 or 128 MB.<br>
      <br>
      I think I understood the concepts of HAR or sequential files. <br>
      But if I pack, for example, my files together in a large file of
      many Gigabytes it is impossible to access one single photo from
      the Hadoop repository in a reasonable time. It makes no sense in
      my eyes to pack thousands of files into a large file just so that
      Hadoop jobs can handle it better. To simply access a single file
      from a web interface - as in my case - it seems to be all
      counterproductive. <br>
      <br>
      So my question is: Is Hadoop only feasible to archive large
      Web-server log files and not designed to handle big archives of
      small files with also business critical data?<br>
      <br>
      <br>
      Thanks for your advice in advance.<br>
      <br>
      Ralph<br>
    </p>
    <div>-- <br>
      <br>
    </div>
  </div>

</blockquote></div>

--f403045fa370c4b7810558622b47--