Return-Path: 
 <dev-return-4803-apmail-falcon-dev-archive=falcon.apache.org@falcon.incubator.apache.org>
X-Original-To: apmail-falcon-dev-archive@minotaur.apache.org
Delivered-To: apmail-falcon-dev-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4987911EE4
	for <apmail-falcon-dev-archive@minotaur.apache.org>;
 Thu, 24 Jul 2014 00:17:33 +0000 (UTC)
Received: (qmail 61228 invoked by uid 500); 24 Jul 2014 00:17:33 -0000
Delivered-To: apmail-falcon-dev-archive@falcon.apache.org
Received: (qmail 61183 invoked by uid 500); 24 Jul 2014 00:17:33 -0000
Mailing-List: contact dev-help@falcon.incubator.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@falcon.incubator.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@falcon.incubator.apache.org>
List-Post: <mailto:dev@falcon.incubator.apache.org>
List-Id: <dev.falcon.incubator.apache.org>
Reply-To: dev@falcon.incubator.apache.org
Delivered-To: mailing list dev@falcon.incubator.apache.org
Received: (qmail 61171 invoked by uid 99); 24 Jul 2014 00:17:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jul 2014 00:17:32 +0000
X-ASF-Spam-Status: No, hits=1.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of johnyu0520@gmail.com designates
 209.85.213.41 as permitted sender)
Received: from [209.85.213.41] (HELO mail-yh0-f41.google.com) (209.85.213.41)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jul 2014 00:17:29 +0000
Received: by mail-yh0-f41.google.com with SMTP id b6so1387031yha.0
        for <dev@falcon.incubator.apache.org>;
 Wed, 23 Jul 2014 17:17:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:from:date:message-id:subject:to:content-type;
        bh=WaY5QgDUUdlcHa8yHKOC59za1vsQ7omh17T57cctYYY=;
        b=F1kw7OKPDNiI0lRTjwCcnZvFyH3DCrzKpaem3MYGsNbXhETsiMrydCK4m2mfJzDkgV
         8M+9D6gz75xg3ztrjPvwtrR9026i2aH61aW5hr5VSLElaBdbM8/AJNOpuyLXLSjiyJAp
         oZgMKsaBzijSs+os2wrdlB5Jjg2ZT6kJYgffxn8RWsWtQ2OrXZxqW7aBfTpKkuK2RJeY
         HY+MG0Sl+IpOb6W6gN8GpzpHLS/V2B1UHb+yq60poKCTSqq8g4sLy7lwnNxpSuo2Ey+n
         8lPiNxx0ozhEwJe82p92mfloO28LdEOT+fMMqBJ9ptx1O+CNmxn+ZQwmGkXzcTfMqUXp
         7+mw==
X-Received: by 10.236.228.161 with SMTP id f31mr6924619yhq.44.1406161024432;
 Wed, 23 Jul 2014 17:17:04 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.170.115.20 with HTTP; Wed, 23 Jul 2014 17:16:34 -0700 (PDT)
From: John Yu <johnyu0520@gmail.com>
Date: Wed, 23 Jul 2014 17:16:34 -0700
Message-ID: 
 <CAJQpoBQixpx8aU_nv4Zh=J6zYA_CkD9o3WY9TYZ9RSAaaneveQ@mail.gmail.com>
Subject: Partitions in Feed definition
To: dev@falcon.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a113335265bbf7f04fee5603a
X-Virus-Checked: Checked by ClamAV on apache.org

--001a113335265bbf7f04fee5603a
Content-Type: text/plain; charset=UTF-8

Hey all,

Few questions about Partitions:

Partitions in the FEED xml like below:

    <partitions>
        <partition name="colo"/>
        <partition name="country"/>
    </partitions>


   1. I see these are partition keys; are the partition keys values
(say country=us or country=uk) need to be defined before-hand or
unbounded?
   2. does the storage location need to have the partition key in
them? Like below (see the colo and country partition keys)

   <location path="/data/${colo}/${country}/${YEAR}/${MONTH}/${DAY}"
type="data"/>

   3.

   if the partition keys are not in the FileSystem path, how does
Falcon identify a feed partition physical location (actually,
how/where is it used)? I understand if it were HCAT, the Feed
definition has the partition key-values.

   4.

   Are these partition keys and values validated against the
FileSystem or HCAT locations?



Partition attribute in the Cluster reference:

Using the example from the documentation page
<http://falcon.incubator.apache.org/docs/FalconArchitecture.html#Replication>


   1. What does it mean to specify partitions in a source cluster ?
   2. vs target cluster? (does it act like a filter to pull only a
subset of data from source? -- if so how does Falcon know to read the
subset in Filesystem feed?)
   3. What data is in sourceCluster1, sourceCluster2 and what location?
   4. Which path does the replicated data end up in the backupCluster (target)?


A few questions.  Hopefully it's something straightforward about
partitions that I have missed.


Thanks for your answers,John

--001a113335265bbf7f04fee5603a--