Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of enis.soz@gmail.com designates
 209.85.219.171 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding;
        b=g/iUgbzrhUk4cmv6vhqLKzsDuaLXNISsu6f6MtKdeF8FAMXfzsHNwCSg04coXRvPsz
         u2uYuISoyWfGiMdideUzs7XiaRE36f61TLXo8tNBhSHgSYQkZ8XKNQwO5uzWxd+EWEG9
         CB824zG2htmDNh8rpcDD6crm/x/cu1OQ8EvxQ=
Message-ID: <49C0F207.4090203@gmail.com>
Date: Wed, 18 Mar 2009 15:07:19 +0200
From: Enis Soztutar <enis.soz@gmail.com>
User-Agent: Thunderbird 2.0.0.19 (X11/20090105)
MIME-Version: 1.0
To: core-user@hadoop.apache.org
Subject: Re: merging files
References: <BAY119-DS726DAF2AA6A4C49D8431FD4990@phx.gbl>
In-Reply-To: <BAY119-DS726DAF2AA6A4C49D8431FD4990@phx.gbl>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Use MultipleInputs and use two different mappers for the inputs. map1 
should be IdentityMapper, mapper 2 should output key, value pairs where 
value is a peudo marker value(same for all keys), which marks that the 
value is null/empty. In the reducer just output the key/value pairs 
which does not include the marker value in their values.

in your example suppose that we use -1 as a marker value, then in 
mapper2, the output will be
4, -1
2, -1

and the reducer will get :

2, {1,3,5,-1}
3, {1,2}
4, {7,9,-1}
6, {3}

then reducer will output :

3, 1
3, 2
6, 3


Nir Zohar wrote:
> Hi,
>
>  
>
> I would like your help with the below question.
>
> I have 2 files: file1 (key, value), file2 (only key) and I need to exclude
> all records from file1 that these key records not in file2.
>
> 1. The output format is key-value, not only keys.
>
> 2. The key is not primary key; hence it's not possible to have joined in the
> end.
>
>  
>
> Can you assist?
>
>  
>
> Thanks,
>
> Nir.
>
>  
>
>  
>
> Example:
>
>  
>
> file1:
>
> 2,1
>
> 2,3
>
> 2,5
>
> 3,1
>
> 3,2
>
> 4,7
>
> 4,9
>
> 6,3
>
>  
>
> file2:
>
> 4
>
> 2
>
>  
>
> Output:
>
> 3,1
>
> 3,2
>
> 6,3
>
>  
>
>  
>
>  
>
>
>