Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of vboylin1987@gmail.com
 designates 209.85.160.50 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CADoiZqq_uWDnpAYYhrAu2vaXYHrFFidhMquidu+8Lh3HQgz8ig@mail.gmail.com>
References: 
 <CAKxYK2EemQW1Zi4_YZU8QvMa7JQGqtVc5_usjkY+41gDZiGncw@mail.gmail.com>
	<CADoiZqq_uWDnpAYYhrAu2vaXYHrFFidhMquidu+8Lh3HQgz8ig@mail.gmail.com>
Date: Thu, 8 Aug 2013 00:28:08 +0800
Message-ID: 
 <CAKxYK2HhHfrQpqsNJ_7c5afNNfYVbG-98SSNxb9RZP40R194ag@mail.gmail.com>
Subject: Re: Is there any way to use a hdfs file as a Circular buffer?
From: Wukang Lin <vboylin1987@gmail.com>
To: hadoop-user <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=047d7b624c4adc784704e35e06d1

--047d7b624c4adc784704e35e06d1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Niels and Bertrand,
    Thank you for you great advices.
    In our scenario, we need to store a steady stream of binary data into a
circular storage,throughput and concurrency are the most important
indicators.The first way seems work, but as  hdfs is not friendly for small
files, this approche may be not smooth enough.HBase is good, but  not
appropriate for us, both for throughput and storage.mongodb is quite good
for web applications, but not suitable the scenario we meet all the same.
    we need a distributed storage system,with Highe throughput, HA,LB and
secure. Maybe It act much like hbase, manager a lot of small file(hfile) as
a large region. we manager a lot of small file as a large one. Perhaps we
should develop it by ourselives.

Thank you.
Lin Wukang


2013/7/25 Niels Basjes <Niels@basjes.nl>

> A circular file on hdfs is not possible.
>
> Some of the ways around this limitation:
> - Create a series of files and delete the oldest file when you have too
> much.
> - Put the data into an hbase table and do something similar.
> - Use completely different technology like mongodb which has built in
> support for a circular buffer (capped collection).
>
> Niels
>
> Hi all,
>    Is there any way to use a hdfs file as a Circular buffer? I mean, if I=
 set a quotas to a directory on hdfs, and writting data to a file in that d=
irectory continuously. Once the quotas exceeded, I can redirect the writter=
 and write the data from the beginning of the file automatically .
>
>

--047d7b624c4adc784704e35e06d1
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Niels and Bertrand,<div>=A0 =A0 Thank you for you great=
 advices.</div><div>=A0 =A0 In our scenario, we need to store=A0a steady st=
ream of binary data into a circular=A0storage,throughput and concurrency ar=
e the most important indicators.The first way seems work, but as =A0hdfs is=
 not friendly for small files, this=A0approche may be not smooth enough.HBa=
se is good, but =A0not appropriate for us, both for throughput and storage.=
<span style=3D"font-family:arial,sans-serif">mongodb</span><span style=3D"f=
ont-family:arial,sans-serif">=A0is quite good for web applications, but=A0<=
/span><font face=3D"arial, sans-serif">not suitable the scenario we meet al=
l the same.</font></div>
<div><div><font face=3D"arial, sans-serif">=A0 =A0 we need a distributed st=
orage system,with Highe=A0</font>throughput,=A0<span style=3D"font-family:a=
rial,sans-serif">HA,LB and secure. Maybe It act much like hbase, manager a =
lot of small file(hfile) as a large region. we manager a lot of small file =
as a large one.=A0</span><font face=3D"arial, sans-serif">Perhaps we should=
 develop it by ourselives.</font></div>
</div><div><font face=3D"arial, sans-serif"><br></font></div><div><font fac=
e=3D"arial, sans-serif">Thank you.</font></div><div><font face=3D"arial, sa=
ns-serif">Lin Wukang</font></div>
<div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">2013/7/25 Nie=
ls Basjes <span dir=3D"ltr">&lt;<a href=3D"mailto:Niels@basjes.nl" target=
=3D"_blank">Niels@basjes.nl</a>&gt;</span><br><blockquote class=3D"gmail_qu=
ote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-co=
lor:rgb(204,204,204);border-left-style:solid;padding-left:1ex">


<p dir=3D"ltr">A circular file on hdfs is not possible.</p>
<p dir=3D"ltr">Some of the ways around this limitation:<br>
- Create a series of files and delete the oldest file when you have too muc=
h.<br>
- Put the data into an hbase table and do something similar.<br>
- Use completely different technology like mongodb which has built in suppo=
rt for a circular buffer (capped collection).</p><span><font color=3D"#8888=
88">
<p dir=3D"ltr">Niels</p></font></span><div><div>
<div style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-co=
lor:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir=3D"=
ltr"><pre style=3D"white-space:pre-wrap">Hi all,
   Is there any way to use a hdfs file as a Circular buffer? I mean, if I s=
et a quotas to a directory on hdfs, and writting data to a file in that dir=
ectory continuously. Once the quotas exceeded, I can redirect the writter a=
nd write the data from the beginning of the file automatically .</pre>


</div>
</div>
</div></div></blockquote></div><br></div></div>

--047d7b624c4adc784704e35e06d1--