Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-issues@hadoop.apache.org
Date: Thu, 5 Mar 2015 20:00:41 +0000 (UTC)
From: "Sean Busbey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.12779871.1425585410000.14355.1425585641505@Atlassian.JIRA>
In-Reply-To: <JIRA.12779871.1425585410000@Atlassian.JIRA>
References: <JIRA.12779871.1425585410000@Atlassian.JIRA>
 <JIRA.12779871.1425585410012@arcas>
Subject: [jira] [Commented] (HADOOP-11680) Deduplicate jars in convenience
 binary distribution
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-11680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349394#comment-14349394 ] 

Sean Busbey commented on HADOOP-11680:
--------------------------------------

The rest of Colin's work

{quote}
And here is the de-duplicator program I wrote (in Go):
{code}
package main

import (
  "path/filepath"
  "flag"
  "fmt"
  "os"
)

var basePathToFullPath map[string][]string = map[string][]string{ }

func visit(path string, f os.FileInfo, err error) error {
	if err != nil {
		panic(err)
	}
	if f.Mode().IsDir() {
		return nil
	}
	base := filepath.Base(path)
	bases := basePathToFullPath[base]
	if bases == nil {
		bases = make([]string, 0, 1)
	}
	bases = append(bases, path)
	basePathToFullPath[base] = bases
	fmt.Printf("%s -> %s\n", base, path)
	return nil
}

func main() {
	flag.Parse()
	if len(os.Args) < 2 {
		fmt.Printf("Usage: %s [path]\n", os.Args[0])
		os.Exit(1)
	}
	root := os.Args[1]
	err := filepath.Walk(root, visit)
	if err != nil {
		fmt.Printf("Error while traversing %s: %s\n", root, err.Error())
		os.Exit(1)
	}
	for basePath, fullPaths := range basePathToFullPath {
		if len(fullPaths) <= 1 {
			continue
		}
		absPath, err := filepath.Abs(fullPaths[0])
		if err != nil {
			fmt.Printf("failed to find abspath of %s: %s\n", fullPaths[0], err.Error())
			os.Exit(1)
		}
		fmt.Printf("Handling %s\n", basePath)
		for i := 1; i < len(fullPaths); i++ {
			fmt.Printf("rm %s\n", fullPaths[i])
			err = os.Remove(fullPaths[i])
			if err != nil {
				panic(err)
			}
			fmt.Printf("ln %s %s\n", absPath, fullPaths[i])
			err = os.Symlink(absPath, fullPaths[i])
			if err != nil {
				panic(err)
			}
		}
	}
}
{code}
{quote}

and things that he thinks are lacking

{quote}
My simple script would also have to be modified (we probably would want relative symlinks rather than absolute ones, for one thing...) Also we probably want to have Maven somehow do this (either by writing a Maven plugin, or tweaking the assembly one...) Not something I know how to do, but maybe there are some maven experts in the audience? 
{quote}

> Deduplicate jars in convenience binary distribution
> ---------------------------------------------------
>
>                 Key: HADOOP-11680
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11680
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Sean Busbey
>            Assignee: Sean Busbey
>
> Pulled from discussion on HADOOP-11656 Colin wrote:
> {quote}
> bq. Andrew wrote: One additional note related to this, we can spend a lot of time right now distributing 100s of MBs of jar dependencies when launching a YARN job. Maybe this is ameliorated by the new shared distributed cache, but I've heard this come up quite a bit as a complaint. If we could meaningfully slim down our client, it could lead to a nice win.
> I'm frustrated that nobody responded to my earlier suggestion that we de-duplicate jars. This would drastically reduce the size of our install, and without rearchitecting anything.
> In fact I was so frustrated that I decided to write a program to do it myself and measure the delta. Here it is:
> Before:
> {code}
> du -h /h
> 249M    /h
> {code}
> After:
> {code}
> du -h /h
> 140M    /h
> {code}
> Seems like deduplicating jars would be a much better project than splitting into a client jar, if we really cared about this.
> <snip>
> {quote}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)