pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-4963) Add a Bloom join
Date Sat, 21 Jan 2017 00:36:27 GMT

     [ https://issues.apache.org/jira/browse/PIG-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Rohini Palaniswamy updated PIG-4963:
    Attachment: PIG-4963-4.patch

> Add a Bloom join
> ----------------
>                 Key: PIG-4963
>                 URL: https://issues.apache.org/jira/browse/PIG-4963
>             Project: Pig
>          Issue Type: New Feature
>            Reporter: Rohini Palaniswamy
>            Assignee: Rohini Palaniswamy
>             Fix For: 0.17.0
>         Attachments: PIG-4963-1.patch, PIG-4963-2.patch, PIG-4963-3.patch, PIG-4963-4.patch
> In PIG-4925, added option to pass BloomFilter as a scalar to bloom function. But found
that actually using it for big data which required huge vector size was very inefficient and
led to OOM.
>    I had initially calculated that it would take around 12MB bytearray for 100 million
vectorsize (100000000 + 7) / 8 = 12500000 bytes) and that would be the scalar value broadcasted
and would not take much space. But problem is 12MB was written out for every input record
with BuildBloom$Initial before the aggregation happens and we arrive at the final BloomFilter
vector. And with POPartialAgg it runs into OOM issues. 
> If we added a bloom join implementation, which can be combined with hash or skewed join
it would boost performance for a lot of jobs. Bloom filter of the smaller tables can be sent
to the bigger tables as scalar and data filtered before hash or skewed join is used.

This message was sent by Atlassian JIRA

View raw message