hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Busbey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11680) Deduplicate jars in convenience binary distribution
Date Thu, 05 Mar 2015 20:00:41 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349394#comment-14349394
] 

Sean Busbey commented on HADOOP-11680:
--------------------------------------

The rest of Colin's work

{quote}
And here is the de-duplicator program I wrote (in Go):
{code}
package main

import (
  "path/filepath"
  "flag"
  "fmt"
  "os"
)

var basePathToFullPath map[string][]string = map[string][]string{ }

func visit(path string, f os.FileInfo, err error) error {
	if err != nil {
		panic(err)
	}
	if f.Mode().IsDir() {
		return nil
	}
	base := filepath.Base(path)
	bases := basePathToFullPath[base]
	if bases == nil {
		bases = make([]string, 0, 1)
	}
	bases = append(bases, path)
	basePathToFullPath[base] = bases
	fmt.Printf("%s -> %s\n", base, path)
	return nil
}

func main() {
	flag.Parse()
	if len(os.Args) < 2 {
		fmt.Printf("Usage: %s [path]\n", os.Args[0])
		os.Exit(1)
	}
	root := os.Args[1]
	err := filepath.Walk(root, visit)
	if err != nil {
		fmt.Printf("Error while traversing %s: %s\n", root, err.Error())
		os.Exit(1)
	}
	for basePath, fullPaths := range basePathToFullPath {
		if len(fullPaths) <= 1 {
			continue
		}
		absPath, err := filepath.Abs(fullPaths[0])
		if err != nil {
			fmt.Printf("failed to find abspath of %s: %s\n", fullPaths[0], err.Error())
			os.Exit(1)
		}
		fmt.Printf("Handling %s\n", basePath)
		for i := 1; i < len(fullPaths); i++ {
			fmt.Printf("rm %s\n", fullPaths[i])
			err = os.Remove(fullPaths[i])
			if err != nil {
				panic(err)
			}
			fmt.Printf("ln %s %s\n", absPath, fullPaths[i])
			err = os.Symlink(absPath, fullPaths[i])
			if err != nil {
				panic(err)
			}
		}
	}
}
{code}
{quote}

and things that he thinks are lacking

{quote}
My simple script would also have to be modified (we probably would want relative symlinks
rather than absolute ones, for one thing...) Also we probably want to have Maven somehow do
this (either by writing a Maven plugin, or tweaking the assembly one...) Not something I know
how to do, but maybe there are some maven experts in the audience? 
{quote}

> Deduplicate jars in convenience binary distribution
> ---------------------------------------------------
>
>                 Key: HADOOP-11680
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11680
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Sean Busbey
>            Assignee: Sean Busbey
>
> Pulled from discussion on HADOOP-11656 Colin wrote:
> {quote}
> bq. Andrew wrote: One additional note related to this, we can spend a lot of time right
now distributing 100s of MBs of jar dependencies when launching a YARN job. Maybe this is
ameliorated by the new shared distributed cache, but I've heard this come up quite a bit as
a complaint. If we could meaningfully slim down our client, it could lead to a nice win.
> I'm frustrated that nobody responded to my earlier suggestion that we de-duplicate jars.
This would drastically reduce the size of our install, and without rearchitecting anything.
> In fact I was so frustrated that I decided to write a program to do it myself and measure
the delta. Here it is:
> Before:
> {code}
> du -h /h
> 249M    /h
> {code}
> After:
> {code}
> du -h /h
> 140M    /h
> {code}
> Seems like deduplicating jars would be a much better project than splitting into a client
jar, if we really cared about this.
> <snip>
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message