hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Udbhav Agarwal <udbhav.agar...@syncoms.com>
Subject RE: Hbase on docker container with persistent storage
Date Fri, 14 Jul 2017 13:06:53 GMT
Additionally, can you please provide me some links which can guide me to setup up such system
with volumes ? Thank you.

Udbhav
-----Original Message-----
From: Udbhav Agarwal [mailto:udbhav.agarwal@syncoms.com] 
Sent: Friday, July 14, 2017 6:31 PM
To: user@hbase.apache.org
Cc: dimaspivak@apache.org
Subject: RE: Hbase on docker container with persistent storage

Thank you Dima for the response.
	Let me reiterate what I want to achieve in my case. I am using hbase to persist my bigdata(Terabytes
and petabytes) coming from various sources through spark streaming and kafka.  Spark streaming
and kafka are running as separate microservices inside different and excusive containers.
These containers are communicating with http service protocol. Currently I am using hbase
setup on 4 VMs on a single host machine. I have a microservice inside a container to connect
to this hbase. This whole setup is functional and I am able to persist data into as well as
get data from hbase into spark streaming. My use case is of real time ingestion into hbase
as well as real time query from hbase.
	Now I am planning to deploy hbase itself inside container. I want to know what are the options
for this. In how many possible ways I can achieve this ? If I use volumes of container, will
they be able to hold such amount of data (TBs & PBs) ? How will I setup up hdfs inside
volumes ? how can I use the power of distributed file system there? Is this the best way ?



Thanks,
Udbhav
-----Original Message-----
From: Dima Spivak [mailto:dimaspivak@apache.org]
Sent: Friday, July 14, 2017 3:44 AM
To: hbase-user <user@hbase.apache.org>
Subject: Re: Hbase on docker container with persistent storage

Udbhav,

Volumes are Docker's way of having folders or files from the host machine bypass the union
filesystem used within a Docker container. As such, if a container with a volume is killed,
the data from that volume should remain there. That said, if whatever caused the container
to die affects the filesystem within the container, it would also affect the data on the host.

Running HBase in the manner you've described is not typical in anything resembling a production
environment, but if you explain more about your use case, we could provide more advice. That
said, how you'd handle data locality and, in particular, multi-host deployments of HBase in
this manner is more of a concern for me than volume data corruption. What kind of scale do
you need to support? What kind of performance do you expect?

-Dima

On Thu, Jul 13, 2017 at 12:18 AM, Samir Ahmic <ahmic.samir@gmail.com> wrote:

> Hi Udbhav,
> Great work on hbase docker deployment was done in
> https://issues.apache.org/jira/browse/HBASE-12721 you may start your 
> journey from there.  As for rest of your questions maybe there are 
> some folks here that were doing similar testing and may give you more info.
>
> Regards
> Samir
>
> On Thu, Jul 13, 2017 at 7:57 AM, Udbhav Agarwal < 
> udbhav.agarwal@syncoms.com>
> wrote:
>
> > Hi All,
> > I need to run hbase 0.98 backed by hdfs on docker container and want 
> > to stop the data lost if the container restarts.
> >                As per my understanding of docker containers, they 
> > work in a way that if any of the container is stopped/killed , every 
> > information related to it gets killed. It implies if I am running 
> > hbase in a
> container
> > and I have stored some data in some tables and consequently if the 
> > container is stopped then the data will be lost. I need a way in 
> > which I can stop this data loss.
> >                I have gone through concept of volume in docker. Is 
> > it possible to stop this data loss with this approach? What if 
> > volume gets corrupted? Is there any instance of volume running there 
> > which can be stopped and can cause data loss ?
> >                Is there a possibility that I can use hdfs running at 
> > some external host outside the docker and my hbase running inside 
> > docker ? Is such scenario possible ? If yes, How ?
> >                Thank you in advance.
> >
> >
> > Thanks,
> > Udbhav Agarwal
> >
> >
>
Mime
View raw message