hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sanjay Radia <sanjayo...@gmail.com>
Subject Re: LinkedIn Dynamometer Tool (was About 2.7.4 Release)
Date Fri, 21 Jul 2017 18:01:54 GMT
Erik
  Great stuff. 
BTW did you build on top of the “simulated data nodes” in HDFS which has a way to storing
only the length of data (but not real data)? That work allowed supplementing  with a matching
editsLog for the NN. Your approach of using a real image has the advantage of being able to
replay traces from audit logs.
(Ref https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DataNodeCluster.java)

thanks

sanjay
> On Jul 20, 2017, at 10:42 AM, Erik Krogen <ekrogen@linkedin.com.INVALID> wrote:
> 
> forking off of the 2.7.4 release thread to answer this question about
> Dynamometer
> 
> Dynamometer is a tool developed at LinkedIn for scale testing HDFS,
> specifically the NameNode. We have been using it for some time now and have
> recently been making some enhancements to ease of use and reproducibility.
> We hope to post a blog post sometime in the not-too-distant future, and
> also to open source it. I can provide some details here given that we have
> been leveraging it as part of our 2.7.4 release / upgrade process (in
> addition to previous upgrades).
> 
> The basic idea is to get full-scale black-box testing of the HDFS NN while
> using significantly less (~10%) hardware than a real cluster of that size
> would require. We use real NN images from our at-scale clusters paired with
> some logic to fake out DNs into thinking they are storing data when they
> are not, allowing us to stuff more DNs onto each machine. Since we use a
> real image, we can replay real traces (collected from audit logs) to
> compare actual production performance vs. performance on this simulated
> cluster (with additional tuning, different version, etc.). We leverage YARN
> to manage setting up this cluster and to replay the traces.
> 
> Happy to answer questions.
> 
> Erik
> 
> On Wed, Jul 19, 2017 at 5:05 PM, Konstantin Shvachko <shv.hadoop@gmail.com>
> wrote:
> 
>> Hi Tianyi,
>> 
>> Glad you are interested in Dynamometer. Erik (CC-ed) is actively working
>> on this project right now, I'll let him elaborate.
>> Erik, you should probably respond on Apache dev list, as I think it could
>> be interesting for other people as well, asince we planned to open source
>> it. You can fork the "About 2.7.4 Release" thread with a new subject and
>> give some details about Dynamometer there.
>> 
>> Thanks,
>> --Konstantin
>> 
>> On Wed, Jul 19, 2017 at 1:40 AM, 何天一 <hetianyi@bytedance.com> wrote:
>> 
>>> Hi, Shavachko.
>>> 
>>> You mentioned an internal tool called Dynamometer to test NameNode
>>> performance earlier in the 2.7.4 release thread.
>>> I wonder if you could share some ideas behind the tool. Or is there a
>>> plan to bring Dynamometer to open source community?
>>> 
>>> Thanks.
>>> 
>>> BR,
>>> Tianyi
>>> 
>>> On Fri, Jul 14, 2017 at 8:45 AM Konstantin Shvachko <shv.hadoop@gmail.com>
>>> wrote:
>>> 
>>>> Hi everybody.
>>>> 
>>>> We have been doing some internal testing of Hadoop 2.7.4. The testing is
>>>> going well.
>>>> Did not find any major issues on our workloads.
>>>> Used an internal tool called Dynamometer to check NameNode performance on
>>>> real cluster traces. Good.
>>>> Overall test cluster performance looks good.
>>>> Some more testing is still going on.
>>>> 
>>>> I plan to build an RC next week. If there are no objection.
>>>> 
>>>> Thanks,
>>>> --Konst
>>>> 
>>>> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <
>>>> shv.hadoop@gmail.com>
>>>> wrote:
>>>> 
>>>>> Hey guys.
>>>>> 
>>>>> An update on 2.7.4 progress.
>>>>> We are down to 4 blockers. There is some work remaining on those.
>>>>> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
>>>>> Would be good if people could follow up on review comments.
>>>>> 
>>>>> I looked through nightly Jenkins build results for 2.7.4 both on Apache
>>>>> Jenkins and internal.
>>>>> Some test fail intermittently, but there no consistent failures. I
>>>> filed
>>>>> HDFS-11985 to track some of them.
>>>>> https://issues.apache.org/jira/browse/HDFS-11985
>>>>> I do not currently consider these failures as blockers. LMK if some of
>>>>> them are.
>>>>> 
>>>>> We started internal testing of branch-2.7 on one of our smallish (100+
>>>>> nodes) test clusters.
>>>>> Will update on the results.
>>>>> 
>>>>> There is a plan to enable BigTop for 2.7.4 testing.
>>>>> 
>>>>> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
>>>>> Thank you everybody for contributing to this effort.
>>>>> 
>>>>> Regards,
>>>>> --Konstantin
>>>>> 
>>>>> 
>>>>> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aajisaka@apache.org>
>>>>> wrote:
>>>>> 
>>>>>> Sure.
>>>>>> If you want to edit the wiki, please tell me your ASF confluence
>>>> account.
>>>>>> 
>>>>>> -Akira
>>>>>> 
>>>>>> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>>>>>> 
>>>>>>> Couple of more JIRAs need to be back ported for 2.7.4 release.
These
>>>> will
>>>>>>> solve RM HA unstability issues.
>>>>>>> https://issues.apache.org/jira/browse/YARN-5333
>>>>>>> https://issues.apache.org/jira/browse/YARN-5988
>>>>>>> https://issues.apache.org/jira/browse/YARN-6304
>>>>>>> 
>>>>>>> I will raise a JIRAs to back port it.
>>>>>>> 
>>>>>>> @Akira , could  you help to add these JIRAs into wiki?
>>>>>>> 
>>>>>>> Thanks & Regards
>>>>>>> Rohith Sharma K S
>>>>>>> 
>>>>>>> On 29 May 2017 at 12:19, Akira Ajisaka <aajisaka@apache.org>
wrote:
>>>>>>> 
>>>>>>> Created a page for 2.7.4 release.
>>>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>>>>>>>> 
>>>>>>>> If you want to edit this wiki, please ping me.
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Akira
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>>>>>>>> 
>>>>>>>> Hi Konstantin Shvachko
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> how about creating a wiki page for 2.7.4 release status
like 2.8
>>>> and
>>>>>>>>> trunk in following link.??
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> https://cwiki.apache.org/confluence/display/HADOOP
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> ________________________________
>>>>>>>>> From: Konstantin Shvachko <shv.hadoop@gmail.com>
>>>>>>>>> Sent: Saturday, May 13, 2017 3:58 AM
>>>>>>>>> To: Akira Ajisaka
>>>>>>>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org;
>>>>>>>>> yarn-dev@hadoop.apache.org
>>>>>>>>> Subject: Re: About 2.7.4 Release
>>>>>>>>> 
>>>>>>>>> Latest update on the links and filters. Here is the correct
link
>>>> for
>>>>>>>>> the
>>>>>>>>> filter:
>>>>>>>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
>>>>>>>>> requestId=12340814
>>>>>>>>> 
>>>>>>>>> Also updated: https://s.apache.org/Dzg4
>>>>>>>>> 
>>>>>>>>> Had to do some Jira debugging. Sorry for confusion.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> --Konstantin
>>>>>>>>> 
>>>>>>>>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko
<
>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Hey Akira,
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> I didn't have private filters. Most probably Jira
caches
>>>> something.
>>>>>>>>>> Your filter is in the right direction, but for some
reason it
>>>> lists
>>>>>>>>>> only
>>>>>>>>>> 22 issues, while mine has 29.
>>>>>>>>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir
>>>>>>>>>> a/browse/YARN-5543>
>>>>>>>>>> .
>>>>>>>>>> 
>>>>>>>>>> Anyways, I created a Jira filter now "Hadoop 2.7.4
release
>>>> blockers",
>>>>>>>>>> shared it with "everybody", and updated my link to
point to that
>>>>>>>>>> filter.
>>>>>>>>>> So
>>>>>>>>>> you can use any of the three methods below to get
the correct
>>>> list:
>>>>>>>>>> 1. Go to https://s.apache.org/Dzg4
>>>>>>>>>> 2. Go to the filter via
>>>>>>>>>>    https://issues.apache.org/jira/issues?filter=12340814
>>>>>>>>>>   or by finding "Hadoop 2.7.4 release blockers" filter
in the
>>>> jira
>>>>>>>>>> 3. On Advanced issues search page paste this:
>>>>>>>>>> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels
=
>>>>>>>>>> release-blocker
>>>>>>>>>> AND "Target Version/s" = 2.7.4
>>>>>>>>>> 
>>>>>>>>>> Hope this solves the confusion for which issues are
included.
>>>>>>>>>> Please LMK if it doesn't, as it is important.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> --Konstantin
>>>>>>>>>> 
>>>>>>>>>> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <
>>>> aajisaka@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi Konstantin,
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Thank you for volunteering as release manager!
>>>>>>>>>>> 
>>>>>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> I couldn't see the link. Maybe is it private
filter?
>>>>>>>>>>> 
>>>>>>>>>>> Here is a link I generated: https://s.apache.org/ehKy
>>>>>>>>>>> This filter includes resolved issue and excludes
fixversion ==
>>>> 2.7.4
>>>>>>>>>>> 
>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>> Akira
>>>>>>>>>>> 
>>>>>>>>>>> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hi Brahma Reddy Battula,
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Actually the original link works fine:
>>>> https://s.apache.org/Dzg4
>>>>>>>>>>>> Your link excludes closed and resolved issues,
which needs
>>>>>>>>>>>> backporting,
>>>>>>>>>>>> and
>>>>>>>>>>>> which we cannot reopen, as discussed in this
thread earlier.
>>>>>>>>>>>> 
>>>>>>>>>>>> Looked through the issues you proposed:
>>>>>>>>>>>> 
>>>>>>>>>>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
>>>>>>>>>>>> Seems like a new feature. It helps failover
to standby node when
>>>>>>>>>>>> primary
>>>>>>>>>>>> is
>>>>>>>>>>>> under heavy load, but it introduces new APIs,
addresses, config
>>>>>>>>>>>> parameters.
>>>>>>>>>>>> And needs at least one follow up jira.
>>>>>>>>>>>> Looks like a backward compatible change,
though.
>>>>>>>>>>>> Did you have a chance to run it in production?
>>>>>>>>>>>> 
>>>>>>>>>>>> +1 on
>>>>>>>>>>>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>>>>>>>>>>>> 
>>>>>>>>>>>> [HDFS-10987] Make Decommission less expensive
when lot of ...<
>>>>>>>>>>> 
>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987>
>>>>>>>>> issues.apache.org
>>>>>>>>> When user want to decommission a node which having 50M
blocks ,it
>>>> could
>>>>>>>>> hold the namesystem lock for long time.We've seen it
is taking 36
>>>> sec.
>>>>>>>>> As
>>>>>>>>> we knew during this ...
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>>> [HDFS-9902] Support different values of dfs.datanode.du
...<
>>>>>>>>>>> 
>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902>
>>>>>>>>> issues.apache.org
>>>>>>>>> Now Hadoop support different storage type for DISK, SSD,
ARCHIVE
>>>> and
>>>>>>>>> RAM_DISK, but they share one configuration
>>>> dfs.datanode.du.reserved.
>>>>>>>>> The
>>>>>>>>> DISK size may be several ...
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>>> Trash does not descent into child directories
to check for ...<
>>>>>>>>>>> 
>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312>
>>>>>>>>> issues.apache.org
>>>>>>>>> HDFS trash does not descent into child directory to check
if user
>>>> has
>>>>>>>>> permission to delete files. For example: Run the following
command
>>>> to
>>>>>>>>> initialize directory ...
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>>> Upgrade Jsch jar to latest version to fix
vulnerability in ...<
>>>>>>>>>>> 
>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>>>>>> issues.apache.org
>>>>>>>>> Recently there was on vulnerability reported on jsch
library. Its
>>>> fixed
>>>>>>>>> in latest 0.1.54 version before CVE was made public.
>>>>>>>>> https://cve.mitre.org/cgi-bin/cvename.cgi ...
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Added them to 2.7.4 release. You should see them via
the above link
>>>>>>>>>>>> now.
>>>>>>>>>>>> Would be good if you could attach backport
patches for some of
>>>> them?
>>>>>>>>>>>> 
>>>>>>>>>>>> Appreciate your help,
>>>>>>>>>>>> --Konstantin
>>>>>>>>>>>> 
>>>>>>>>>>>> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy
Battula <
>>>>>>>>>>>> brahmareddy.battula@huawei.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Looks following link is not correct..
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> https://s.apache.org/Dzg4
>>>>>>>>>>>>> 
>>>>>>>>>>>>> It should be like following..?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> https://s.apache.org/wi3U
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Apart from Konstantin mentioned,Following
also good to go..?
>>>> let me
>>>>>>>>>>>>> know
>>>>>>>>>>>>> your thoughts on this.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> For Large Cluster:
>>>>>>>>>>>>> =============
>>>>>>>>>>>>> 
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9311===Life
Line
>>>>>>>>>>>>> Protocol
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987===Deecommis
>>>> sion
>>>>>>>>>>>>> Expensive when lot's of blocks are present
>>>>>>>>>>>>> 
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902===
>>>>>>>>>>>>> "dfs.datanode.du.reserved"  per Storage
Type
>>>>>>>>>>>>> 
>>>>>>>>>>>>> For Security:
>>>>>>>>>>>>> =========
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash
does
>>>> not
>>>>>>>>>>>>> descent
>>>>>>>>>>>>> into child directories to check for permission
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade
>>>> Jsch
>>>>>>>>>>>>> jar
>>>>>>>>>>>>> to
>>>>>>>>>>>>> latest version to fix vulnerability in
old versions
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Regards
>>>>>>>>>>>>> Brahma Reddy Battula
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>> From: Erik Krogen [mailto:ekrogen@linkedin.com.INVALID]
>>>>>>>>>>>>> Sent: 06 May 2017 02:40
>>>>>>>>>>>>> To: Konstantin Shvachko
>>>>>>>>>>>>> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev;
>>>>>>>>>>>>> mapreduce-dev@hadoop.apache.org
>>>>>>>>>>>>> ;
>>>>>>>>>>>>> yarn-dev@hadoop.apache.org
>>>>>>>>>>>>> Subject: Re: About 2.7.4 Release
>>>>>>>>>>>>> 
>>>>>>>>>>>>> List LGTM Konstantin!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Let's say that we will only create a
new tracking JIRA for
>>>> patches
>>>>>>>>>>>>> which
>>>>>>>>>>>>> do not backport cleanly, to avoid having
too many lying around.
>>>>>>>>>>>>> Otherwise
>>>>>>>>>>>>> we can directly attach to old ticket.
If a clean backport does
>>>>>>>>>>>>> happen
>>>>>>>>>>>>> to
>>>>>>>>>>>>> break a test the nightly build will help
us catch it.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Erik
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Thu, May 4, 2017 at 7:21 PM, Konstantin
Shvachko <
>>>>>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Great Zhe. Let's monitor the build.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I marked all jiras I knew of for
inclusion into 2.7.4 as I
>>>>>>>>>>>>>> described
>>>>>>>>>>>>>> before.
>>>>>>>>>>>>>> Target Version/s: 2.7.4
>>>>>>>>>>>>>> Label: release-blocker
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Here is the link to the list: https://s.apache.org/Dzg4
>>>> Please
>>>>>>>>>>>>>> let
>>>>>>>>>>>>>> me
>>>>>>>>>>>>>> know if I missed anything.
>>>>>>>>>>>>>> And feel free to pick up any. Most
of backports are pretty
>>>>>>>>>>>>>> straightforward, but not all.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> We can create tracking jiras for
backporting if you need to
>>>> run
>>>>>>>>>>>>>> Jenkins on the patch (and since Allen
does not allow reopening
>>>>>>>>>>>>>> them).
>>>>>>>>>>>>>> But I think the final patch should
be attached to the original
>>>>>>>>>>>>>> jira.
>>>>>>>>>>>>>> Otherwise history will be hard to
follow.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> --Konstantin
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Wed, May 3, 2017 at 4:53 PM, Zhe
Zhang <zhz@apache.org>
>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for volunteering as RM Konstantin!
The plan LGTM.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I've created a nightly Jenkins
job for branch-2.7 (unit
>>>> tests):
>>>>>>>>>>>>>>> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Wed, May 3, 2017 at 12:42
AM Konstantin Shvachko <
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I and a few of my colleagues
would like to help here and
>>>> move
>>>>>>>>>>>>>>>> 2.7.4
>>>>>>>>>>>>>>>> release forward. A few points
in this regard.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 1. Reading through this thread
since March 1 I see that
>>>> Vinod
>>>>>>>>>>>>>>>> hinted on managing the release.
Vinod, if you still want the
>>>>>>>>>>>>>>>> job /
>>>>>>>>>>>>>>>> have bandwidth will be happy
to work with you.
>>>>>>>>>>>>>>>> Otherwise I am glad to volunteer
as the release manager.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 2. In addition to current
blockers and criticals, I would
>>>> like
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> propose
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> few issues to be included in
the release, see the list below.
>>>>>>>>>>>>>>>> Those
>>>>>>>>>>>>>>>> are mostly bug fixes and
optimizations, which we already
>>>> have in
>>>>>>>>>>>>>>>> our
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> internal
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> branch and run in production. Plus
one minor feature "node
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> labeling", which we found very
handy, when you have
>>>> heterogeneous
>>>>>>>>>>>>>>>> environments and mixed workloads,
like MR and Spark.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 3. For marking issues for
the release I propose to
>>>>>>>>>>>>>>>> - set the target version
to 2.7.4, and
>>>>>>>>>>>>>>>> - add a new label "release-blocker"
>>>>>>>>>>>>>>>> That way we will know issues
targeted for the release
>>>> without
>>>>>>>>>>>>>>>> reopening them for backports.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 4. I see quite a few people
are interested in the release.
>>>> With
>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>> the help I think we can target
to release by the end of May.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Other things include fixing
CHANGES.txt and fixing Jenkins
>>>> build
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 2.7.4
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> branch.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> --Konstantin
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> ==========  List of issue
for 2.7.4  ===========
>>>>>>>>>>>>>>>> ------ Backports
>>>>>>>>>>>>>>>> HADOOP-12975 <https://issues.apache.org/jir
>>>>>>>>>>>>>>>> a/browse/HADOOP-12975>.
>>>>>>>>>>>>>>>> Add
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> du
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> jitters
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>.
>>>> IBR
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> batching
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715
>>>>> .
>>>>>>>>>>>>>> NPE
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> when applying AvailableSpaceBlockPlacementPolicy
>>>>>>>>>>>>>>>> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538
>>>>> .
>>>>>>>>>>>>>>>> fsck
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> removal
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> of dot printing
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
>>>>>>>>>>>>>>>> space-balanced
>>>>>>>>>>>>>>>> policy for balancer
>>>>>>>>>>>>>>>> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549
>>>>> .
>>>>>>>>>>>>>>>> abort
>>>>>>>>>>>>>>>> balancer if upgrade in progress
>>>>>>>>>>>>>>>> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412
>>>>> .
>>>>>>>>>>>>>>>> skip
>>>>>>>>>>>>>>>> small blocks in getBlocks
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471
>>>>> .
>>>>>>>>>>>>>>>> SLS
>>>>>>>>>>>>>>>> simulator
>>>>>>>>>>>>>>>> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302
>>>>> .
>>>>>>>>>>>>>>>> SLS
>>>>>>>>>>>>>>>> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367
>>>>> .
>>>>>>>>>>>>>>>> SLS
>>>>>>>>>>>>>>>> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612
>>>>> .
>>>>>>>>>>>>>>>> SLS
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> ----- Node labeling
>>>>>>>>>>>>>>>> MAPREDUCE-6304
>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
>>>>>>>>>>>>>>>> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
>>>>>>>>>>>>>>>> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
>>>>>>>>>>>>>>>> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
>>>>>>>>>>>>>>>> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
>>>>>>>>>>>>>>>> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Zhe Zhang
>>>>>>>>>>>>>>> Apache Hadoop Committer
>>>>>>>>>>>>>>> http://zhe-thoughts.github.io/about/
| @oldcap
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> ------------------------------------------------------------
>>>> ---------
>>>>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org


Mime
View raw message