hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chathuri Wimalasena <kamalas...@gmail.com>
Subject Re: Upgrading production hadoop system from 2.5.1 to 2.7.2
Date Thu, 24 Mar 2016 08:16:08 GMT
Hi Ravi,

Thank you for all the information, Our application is indexing twitter data
to HBase and then do some data analytics on top of that. That's why HDFS
data is very important to us. We cannot tolerate any data loss with the
update. Do you remember how long it took for you to upgrade it from 2.4.1
to 2.7.1 ?

Thanks,
Chathuri

On Wed, Mar 23, 2016 at 7:09 PM, Ravi Prakash <ravihadoop@gmail.com> wrote:

> Hi Chathuri!
>
> Technically there is a rollback option during upgrade. I don't know how
> well it has been tested, but the idea is that old metadata is not deleted
> until the cluster administrator says $ hdfs dfsadmin -finalizeUpgrade . I'm
> fairly confident that the HDFS upgrade will work smoothly. We have upgraded
> quite a few Hadoop-2.4.1 clusters to Hadoop-2.7.1 successfully (never
> having to roll back). Its your applications that work on top of HDFS and
> YARN that I'd be concerned about.
>
> HTH
> Ravi
>
> On Wed, Mar 23, 2016 at 2:22 PM, Chathuri Wimalasena <kamalasini@gmail.com
> > wrote:
>
>> Thanks for information Ravi. Is there a way that I can back up data
>> before the  update ? I was thinking about this approach..
>>
>> Copy the current hadoop directories to a new set of directories.
>> Point hadoop to this new set
>> Start the migration with the backup set
>>
>> Please let me know if people have done this upgrade successfully. I
>> believe many things can go wrong in a lengthy upgrade like this. The data
>> in the cluster is very important.
>> Thanks,
>> Chathuri
>>
>> On Wed, Mar 23, 2016 at 4:37 PM, Ravi Prakash <ravihadoop@gmail.com>
>> wrote:
>>
>>> Hi Chathuri!
>>>
>>>    - When we upgrade, does it change the namenode data structures and
>>>    data nodes? I assume it only changes the name node...
>>>
>>> It changes the NN as well as DN layout. As a matter of fact, this
>>> upgrade will take a long time on Datanodes as well because of
>>> https://issues.apache.org/jira/browse/HDFS-6482
>>>
>>>    - What are the risks with this upgrade ?
>>>
>>> What Hadoop applications do you run on top of your cluster? The hope is
>>> that everything continues working smoothly for the most part, but
>>> inevitably some backward incompatible changes creep in.
>>>
>>>    - Is there a place where I can review the changes made to file
>>>    system from 2.5.1 to 2.7.2?
>>>
>>> The release notes. http://hadoop.apache.org/releases.html .You'd have
>>> to accumulate all the changes in the versions.
>>>
>>> Practically, I'd try to run my application on your upgraded test cluster.
>>>
>>> HTH
>>>
>>> Ravi
>>>
>>> On Wed, Mar 23, 2016 at 12:17 PM, Chathuri Wimalasena <
>>> kamalasini@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We have a hadoop production deployment with 1 name node and 10 data
>>>> nodes which has more than 20TB of data in HDFS. We are currently using
>>>> Hadoop 2.5.1 and we want to update it to latest Hadoop version, 2.7.2.
>>>>
>>>> I followed the following link (
>>>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html)
>>>> and updated a single node system running in pseudo distributed mode and it
>>>> went without any issues. But this system did not have that much data as the
>>>> production system.
>>>>
>>>> Since this is a production system, I'm reluctant to do this update. I
>>>> would like to see what other people have done in these cases and their
>>>> experiences... Here are few questions I have..
>>>>
>>>>    - When we upgrade, does it change the namenode data structures and
>>>>    data nodes? I assume it only changes the name node...
>>>>    - What are the risks with this upgrade ?
>>>>    - Is there a place where I can review the changes made to file
>>>>    system from 2.5.1 to 2.7.2?
>>>>
>>>> I would really appreciate if you can share your experiences.
>>>>
>>>> Thanks in advance,
>>>> Chathuri
>>>>
>>>
>>>
>>
>

Mime
View raw message