hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bertrand Dechoux <decho...@gmail.com>
Subject Re: what exactly does data in HDFS look like?
Date Sat, 19 Jul 2014 11:34:56 GMT
But basically you are right : it is the same concept as with a classical
file system. A file is seen as a sequence of bytes. For various efficiency
reasons, the whole sequence is not stored like that but first splits into
blocks (subsequences). With a local file system, these blocks will be
within the local drives. With HDFS, they are somewhere within the cluster
(and replicated, of course).

So really, the filesystem doesn't care about what is inside the file and
the format is something it is really oblivious to.


On Sat, Jul 19, 2014 at 7:02 AM, Shahab Yunus <shahab.yunus@gmail.com>

> The data itself is eventually store in a form of file. Each blocks of the
> file and it replicas are stored in files and directories on different
> nodes. The Namenode that keep the information and maintains it about each
> file and where its blocks (and replicated blocks exist in the cluster.)
> As for the format, it is stored as bytes. In the normal cases you use the
> DFS or FileOutputStream classes to  write data and in those instances it is
> written in byte form (conversion to bytes i.e. serialize data.) When you
> read the data, you use the same counterpart classes like InputStream and
> those convert the data from byte to text (i.e. deserialization). Point
> being, HDFS is oblivious to the fact whether it was JSON of XML.
> This would be more evident if you see the code to read/write from HDFS
> (writing example below):
> https://sites.google.com/site/hadoopandhive/home/how-to-write-a-file-in-hdfs-using-hadoop
> Now on the other hand, if you were using compression or other storage
> formats like Avro or Parquet then those formats come with their own classes
> which take care of serialization and deserialization.
> For basic cases, this should be helpful:
> https://www.inkling.com/read/hadoop-definitive-guide-tom-white-3rd/chapter-3/data-flow
> More here on data storage:
> http://stackoverflow.com/questions/2358402/where-hdfs-stores-files-locally-by-default
> http://hadoop.apache.org/docs/r1.2.1/hdfs_design.html#Data+Organization
> https://developer.yahoo.com/hadoop/tutorial/module1.html#data
> Regards,
> Shahab
> On Sat, Jul 19, 2014 at 12:12 AM, Adaryl "Bob" Wakefield, MBA <
> adaryl.wakefield@hotmail.com> wrote:
>>   And by that I mean is there an HDFS file type? I feel like I’m missing
>> something. Let’s say I have a HUGE json file that I import into HDFS. Does
>> it retain it’s JSON format in HDFS? What if it’s just random tweets I’m
>> streaming. Is it kind of like a normal disk where there are all kinds of
>> files sitting on disk in their own format it’s just that in HDFS they are
>> spread out over nodes?
>> B.

View raw message