Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAGiZwvNadWwY0SaxaHwy3+5-AiM=c2aL6cVJsosrGdRW1tsoWw@mail.gmail.com>
References: 
 <CAGiZwvNadWwY0SaxaHwy3+5-AiM=c2aL6cVJsosrGdRW1tsoWw@mail.gmail.com>
Date: Sat, 16 Apr 2016 18:40:18 -0700
Message-ID: 
 <CALte62zb7poFQfhiGaZjN6xoY0QURHQO8MaGDLO11QPPpxGiOw@mail.gmail.com>
Subject: Re: To Store Large Number of Video and Image files
From: Ted Yu <yuzhihong@gmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a113f5022907c890530a45377

--001a113f5022907c890530a45377
Content-Type: text/plain; charset=UTF-8

Have you taken a look at HBASE-11339 (HBase MOB) ?

Note: this feature does not handle 10GB objects well. Consider store GB
image on hdfs.

Cheers

On Sat, Apr 16, 2016 at 6:21 PM, Ascot Moss <ascot.moss@gmail.com> wrote:

> Hi,
>
> I have a project that needs to store large number of image and video files,
> the file size varies from 10MB to 10GB, the initial number of files will be
> 0.1 billion and would grow over 1 billion, what will be the practical
> recommendations to store and view these files?
>
>
>
> #1 One cluster, store the HDFS URL in HBase and store the actual file in
> HDFS? (block_size as 128MB and replication factor as 3)
>
>
> #2 One cluster, Store small files in HBase directly and use #1 for large
> files? (block_size as 128MB and replication factor as 3)
>
>
> #3 Multiple Hadoop/HBase clusters, each with different block_size settings?
>
>
>      e.g. cluster 1 (small): block_size as 128MB and replication factor as
> 3, store all files in HBase if their file size is smaller 128MB
>
>             cluster 2 (large): bigger block_size, say 4GB, replication
> factor as 3, store the HDFS URL in HBase and store the actual file in HDFS
>
>
>
> #4 Use Hadoop Federation for large number of files?
>
>
> About Fault Tolerance, need to consider four types of failures: driver,
> host, rack, and  datacenter failures.
>
>
> Regards
>

--001a113f5022907c890530a45377--