asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mingda li <limingda1...@gmail.com>
Subject Re: Time of Multiple Joins in AsterixDB
Date Fri, 23 Dec 2016 06:27:29 GMT
That will help a lot for new users. They don't need to worry how to get the
best efficiency :-)

On Thu, Dec 22, 2016 at 4:50 PM, Mike Carey <dtabass@gmail.com> wrote:

> Exactly; I was just thinking of how to lazily set the parameters without
> counting disks or cores - i.e., of how to just ask the system to run
> everything with max-parallelism.
>
>
>
> On 12/22/16 2:51 PM, Yingyi Bu wrote:
>
>> Yes, that would work.
>> However, if the cluster only has 128 available cores, it will use 128-way
>> (instead of 1000) parallelism for the computation.
>>
>> Best,
>> Yingyi
>>
>> On Thu, Dec 22, 2016 at 12:40 PM, Mike Carey <dtabass@gmail.com> wrote:
>>
>> So to use maxi-storage parallelism for scans but maxi-core parallelism
>>> subsequently, one would just specify a large positive value >= the number
>>> of available cores? (E.g., 10000)
>>>
>>>
>>>
>>> On 12/22/16 11:37 AM, Yingyi Bu wrote:
>>>
>>> No need to reload data anymore :-)
>>>>
>>>> Best,
>>>> Yingyi
>>>>
>>>> On Thu, Dec 22, 2016 at 11:36 AM, Yingyi Bu <buyingyi@gmail.com> wrote:
>>>>
>>>> Indeed, the change was merged yesterday.
>>>>
>>>>> If you grab the latest master, the computation parallelism can be set
>>>>> by
>>>>> the parameter compiler.parallelism:
>>>>> -- 0, the default, means to use the storage parallelism as computation
>>>>> parallelism
>>>>> -- negative value,  means to use all available cores in the cluster
>>>>> -- positive value, means to that number of cores, but it will fall to
>>>>> back
>>>>> to use all available cores if the number is too large.
>>>>>
>>>>> Best,
>>>>> Yingyi
>>>>>
>>>>>
>>>>> On Thu, Dec 22, 2016 at 10:59 AM, Mike Carey <dtabass@gmail.com>
>>>>> wrote:
>>>>>
>>>>> It would definitely also be fun to see how the newest more flexible
>>>>>
>>>>>> parallelism stuff handles this!  (Where you'd specify storage
>>>>>> parallelism
>>>>>> based on drives, and compute parallelism based on cores, both spread
>>>>>> across
>>>>>> all of the cluster's resources.)
>>>>>>
>>>>>>
>>>>>> On 12/22/16 10:57 AM, Yingyi Bu wrote:
>>>>>>
>>>>>> Mingda,
>>>>>>
>>>>>>>
>>>>>>>         There is one more things that you can do without the need to
>>>>>>> reload
>>>>>>> your data:
>>>>>>>
>>>>>>>         -- Enlarge the buffercache memory budget to 8GB so that the
>>>>>>> datasets
>>>>>>> can fit into the buffer cache:
>>>>>>>             "storage.buffercache.size": 536870912
>>>>>>>             ->
>>>>>>>             "storage.buffercache.size": 8589934592
>>>>>>>
>>>>>>>         You don't need to reload data but only need to restart the
>>>>>>> AsterixDB
>>>>>>> instance.
>>>>>>>         Thanks!
>>>>>>>
>>>>>>> Best,
>>>>>>> Yingyi
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Dec 21, 2016 at 9:22 PM, Mike Carey <dtabass@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Nice!!
>>>>>>>
>>>>>>> On Dec 21, 2016 8:43 PM, "Yingyi Bu" <buyingyi@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Cool, thanks, Mingda!
>>>>>>>>
>>>>>>>> Look forward to the new numbers!
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Yingyi
>>>>>>>>>
>>>>>>>>> On Wed, Dec 21, 2016 at 7:13 PM, mingda li <limingda1993@gmail.com
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>> Dear Yingyi,
>>>>>>>>
>>>>>>>>> Thanks for your suggestion. I have reset the AsterixDB and retest
>>>>>>>>>>
>>>>>>>>>> AsterixDB
>>>>>>>>>>
>>>>>>>>> using new environment on 100G, 10G. The efficiency for all the
>>>>>>>>> tests
>>>>>>>>>
>>>>>>>>>> (good
>>>>>>>>>>
>>>>>>>>> and bad order) have been all improved to twice speed.
>>>>>>>>>
>>>>>>>>>> I will finish all the tests and update the result later.
>>>>>>>>>>
>>>>>>>>>> Bests,
>>>>>>>>>> Mingda
>>>>>>>>>>
>>>>>>>>>> On Tue, Dec 20, 2016 at 10:20 PM, Yingyi Bu <buyingyi@gmail.com>
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>> Hi Mingda,
>>>>>>>>>
>>>>>>>>>         I think that in your setting, a better configuration for
>>>>>>>>>>
>>>>>>>>>>> AsterixDB
>>>>>>>>>>>
>>>>>>>>>> might be to use 64 partitions, i.e., 4 cores *16.
>>>>>>>>>
>>>>>>>>>         1. To achieve that, you have to have 4 iodevices on each
>>>>>>>>>> NC,
>>>>>>>>>>
>>>>>>>>>>> e.g.:
>>>>>>>>>>>
>>>>>>>>>>             iodevices=/home/clash/asterixStorage/asterixdb5/red16
>>>>>>>>>
>>>>>>>>>             -->
>>>>>>>>>>
>>>>>>>>>>>             iodevices=/home/clash/asterix
>>>>>>>>>>> Storage/asterixdb5/red16-1,
>>>>>>>>>>> iodevices=/home/clash/asterixStorage/asterixdb5/red16-2,
>>>>>>>>>>> iodevices=/home/clash/asterixStorage/asterixdb5/red16-3,
>>>>>>>>>>> iodevices=/home/clash/asterixStorage/asterixdb5/red16-4
>>>>>>>>>>>
>>>>>>>>>>>         2. Assume you have 64 partitions, 31.01G/64 ~= 0.5G.
>>>>>>>>>>> That
>>>>>>>>>>> means
>>>>>>>>>>>
>>>>>>>>>>> you'd
>>>>>>>>>>>
>>>>>>>>>> better have 512MB memory budget for each joiner so as to make the
>>>>>>>>>>
>>>>>>>>>>> join
>>>>>>>>>>>
>>>>>>>>>> memory-resident.
>>>>>>>>>
>>>>>>>>>             To achieve that,  in the cc section in the cc.conf, you
>>>>>>>>>>
>>>>>>>>>>> could
>>>>>>>>>>>
>>>>>>>>>> add:
>>>>>>>>>
>>>>>>>>>             compiler.joinmemory=536870912
>>>>>>>>>>
>>>>>>>>>>>         3. For the JVM setting, 1024MB is too small for the NC.
>>>>>>>>>>>             In the shared NC section in cc.conf, you can add:
>>>>>>>>>>>             jvm.args=-Xmx16G
>>>>>>>>>>>
>>>>>>>>>>>         4. For Pig and Hive, you can set the maximum
>>>>>>>>>>> mapper/reducer
>>>>>>>>>>>
>>>>>>>>>>> numbers
>>>>>>>>>>>
>>>>>>>>>> in
>>>>>>>>>>
>>>>>>>>>> the MapReduce configuration, e.g., at most 4 mappers per machine
>>>>>>>>>> and
>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>> most 4 reducers per machine.
>>>>>>>>>
>>>>>>>>>         5. I'm not super-familiar with hyper threading, but it
>>>>>>>>>> might
>>>>>>>>>>
>>>>>>>>>>> be
>>>>>>>>>>>
>>>>>>>>>>> worth
>>>>>>>>>>>
>>>>>>>>>> trying 8 partitions per machine, i.e., 128 partitions in total.
>>>>>>>>>>
>>>>>>>>>>>         To validate if the new settings work, you can go to the
>>>>>>>>>>>
>>>>>>>>>>> admin/cluster
>>>>>>>>>>>
>>>>>>>>>> page to double check.
>>>>>>>>>>
>>>>>>>>>>>         Pls keep us updated and let us know if you run into any
>>>>>>>>>>> issue.
>>>>>>>>>>>         Thanks!
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Yingyi
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Dec 20, 2016 at 9:26 PM, mingda li <
>>>>>>>>>>> limingda1993@gmail.com
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>> Dear Yingyi,
>>>>>>>>>>
>>>>>>>>>>> 1. For the returned of  :http://<master
>>>>>>>>>>>> node>:19002/admin/cluster
>>>>>>>>>>>>
>>>>>>>>>>>> {
>>>>>>>>>>>>        "cc": {
>>>>>>>>>>>>            "configUri": "http://scai01.cs.ucla.edu:
>>>>>>>>>>>> 19002/admin/cluster/cc/config",
>>>>>>>>>>>>            "statsUri": "http://scai01.cs.ucla.edu:
>>>>>>>>>>>> 19002/admin/cluster/cc/stats",
>>>>>>>>>>>>            "threadDumpUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/cc/threaddump"
>>>>>>>>>>>>        },
>>>>>>>>>>>>        "config": {
>>>>>>>>>>>>            "api.port": 19002,
>>>>>>>>>>>>            "cc.java.opts": "-Xmx1024m",
>>>>>>>>>>>>            "cluster.partitions": {
>>>>>>>>>>>>                "0": "ID:0, Original Node: red16, IODevice: 0,
>>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>>> Node:
>>>>>>>>>>>>
>>>>>>>>>>> red16",
>>>>>>>>>>
>>>>>>>>>>                "1": "ID:1, Original Node: red15, IODevice: 0,
>>>>>>>>>>> Active
>>>>>>>>>>>
>>>>>>>>>>>> Node:
>>>>>>>>>>>>
>>>>>>>>>>> red15",
>>>>>>>>>>
>>>>>>>>>>                "2": "ID:2, Original Node: red14, IODevice: 0,
>>>>>>>>>>> Active
>>>>>>>>>>>
>>>>>>>>>>>> Node:
>>>>>>>>>>>>
>>>>>>>>>>> red14",
>>>>>>>>>>
>>>>>>>>>>                "3": "ID:3, Original Node: red13, IODevice: 0,
>>>>>>>>>>> Active
>>>>>>>>>>>
>>>>>>>>>>>> Node:
>>>>>>>>>>>>
>>>>>>>>>>> red13",
>>>>>>>>>>
>>>>>>>>>>                "4": "ID:4, Original Node: red12, IODevice: 0,
>>>>>>>>>>> Active
>>>>>>>>>>>
>>>>>>>>>>>> Node:
>>>>>>>>>>>>
>>>>>>>>>>> red12",
>>>>>>>>>>
>>>>>>>>>>                "5": "ID:5, Original Node: red11, IODevice: 0,
>>>>>>>>>>> Active
>>>>>>>>>>>
>>>>>>>>>>>> Node:
>>>>>>>>>>>>
>>>>>>>>>>> red11",
>>>>>>>>>>
>>>>>>>>>>                "6": "ID:6, Original Node: red10, IODevice: 0,
>>>>>>>>>>> Active
>>>>>>>>>>>
>>>>>>>>>>>> Node:
>>>>>>>>>>>>
>>>>>>>>>>> red10",
>>>>>>>>>>
>>>>>>>>>>                "7": "ID:7, Original Node: red9, IODevice: 0,
>>>>>>>>>>> Active
>>>>>>>>>>>
>>>>>>>>>>>> Node:
>>>>>>>>>>>>
>>>>>>>>>>> red9",
>>>>>>>>>>
>>>>>>>>>>                "8": "ID:8, Original Node: red8, IODevice: 0,
>>>>>>>>>>> Active
>>>>>>>>>>>
>>>>>>>>>>>> Node:
>>>>>>>>>>>>
>>>>>>>>>>> red8",
>>>>>>>>>>
>>>>>>>>>>                "9": "ID:9, Original Node: red7, IODevice: 0,
>>>>>>>>>>> Active
>>>>>>>>>>>
>>>>>>>>>>>> Node:
>>>>>>>>>>>>
>>>>>>>>>>> red7",
>>>>>>>>>>
>>>>>>>>>>                "10": "ID:10, Original Node: red6, IODevice: 0,
>>>>>>>>>>>
>>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>>> Node:
>>>>>>>>>>>>
>>>>>>>>>>> red6",
>>>>>>>>>>>
>>>>>>>>>>>                "11": "ID:11, Original Node: red5, IODevice: 0,
>>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>>> Node:
>>>>>>>>>>>>
>>>>>>>>>>> red5",
>>>>>>>>>>>
>>>>>>>>>>>                "12": "ID:12, Original Node: red4, IODevice: 0,
>>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>>> Node:
>>>>>>>>>>>>
>>>>>>>>>>> red4",
>>>>>>>>>>>
>>>>>>>>>>>                "13": "ID:13, Original Node: red3, IODevice: 0,
>>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>>> Node:
>>>>>>>>>>>>
>>>>>>>>>>> red3",
>>>>>>>>>>>
>>>>>>>>>>>                "14": "ID:14, Original Node: red2, IODevice: 0,
>>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>>> Node:
>>>>>>>>>>>>
>>>>>>>>>>> red2",
>>>>>>>>>>>
>>>>>>>>>>>                "15": "ID:15, Original Node: red, IODevice: 0,
>>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>>> Node:
>>>>>>>>>>>>
>>>>>>>>>>> red"
>>>>>>>>>>
>>>>>>>>>>            },
>>>>>>>>>>>
>>>>>>>>>>>>            "compiler.framesize": 32768,
>>>>>>>>>>>>            "compiler.groupmemory": 33554432,
>>>>>>>>>>>>            "compiler.joinmemory": 33554432,
>>>>>>>>>>>>            "compiler.pregelix.home": "~/pregelix",
>>>>>>>>>>>>            "compiler.sortmemory": 33554432,
>>>>>>>>>>>>            "core.dump.paths": {
>>>>>>>>>>>>                "red": "/home/clash/asterixStorage/
>>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red/coredump",
>>>>>>>>>>>>
>>>>>>>>>>>                "red10": "/home/clash/asterixStorage/
>>>>>>>>>>>
>>>>>>>>>>> asterixdb5/red10/coredump",
>>>>>>>>>>>>                "red11": "/home/clash/asterixStorage/
>>>>>>>>>>>> asterixdb5/red11/coredump",
>>>>>>>>>>>>                "red12": "/home/clash/asterixStorage/
>>>>>>>>>>>> asterixdb5/red12/coredump",
>>>>>>>>>>>>                "red13": "/home/clash/asterixStorage/
>>>>>>>>>>>> asterixdb5/red13/coredump",
>>>>>>>>>>>>                "red14": "/home/clash/asterixStorage/
>>>>>>>>>>>> asterixdb5/red14/coredump",
>>>>>>>>>>>>                "red15": "/home/clash/asterixStorage/
>>>>>>>>>>>> asterixdb5/red15/coredump",
>>>>>>>>>>>>                "red16": "/home/clash/asterixStorage/
>>>>>>>>>>>> asterixdb5/red16/coredump",
>>>>>>>>>>>>                "red2": "/home/clash/asterixStorage/
>>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red2/coredump",
>>>>>>>>>>>>
>>>>>>>>>>>                "red3": "/home/clash/asterixStorage/
>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red3/coredump",
>>>>>>>>>>>>
>>>>>>>>>>>                "red4": "/home/clash/asterixStorage/
>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red4/coredump",
>>>>>>>>>>>>
>>>>>>>>>>>                "red5": "/home/clash/asterixStorage/
>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red5/coredump",
>>>>>>>>>>>>
>>>>>>>>>>>                "red6": "/home/clash/asterixStorage/
>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red6/coredump",
>>>>>>>>>>>>
>>>>>>>>>>>                "red7": "/home/clash/asterixStorage/
>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red7/coredump",
>>>>>>>>>>>>
>>>>>>>>>>>                "red8": "/home/clash/asterixStorage/
>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red8/coredump",
>>>>>>>>>>>>
>>>>>>>>>>>                "red9": "/home/clash/asterixStorage/
>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red9/coredump"
>>>>>>>>>>>>
>>>>>>>>>>>            },
>>>>>>>>>>>
>>>>>>>>>>>>            "feed.central.manager.port": 4500,
>>>>>>>>>>>>            "feed.max.threshold.period": 5,
>>>>>>>>>>>>            "feed.memory.available.wait.timeout": 10,
>>>>>>>>>>>>            "feed.memory.global.budget": 67108864,
>>>>>>>>>>>>            "feed.pending.work.threshold": 50,
>>>>>>>>>>>>            "feed.port": 19003,
>>>>>>>>>>>>            "instance.name": "DEFAULT_INSTANCE",
>>>>>>>>>>>>            "log.level": "WARNING",
>>>>>>>>>>>>            "max.wait.active.cluster": 60,
>>>>>>>>>>>>            "metadata.callback.port": 0,
>>>>>>>>>>>>            "metadata.node": "red16",
>>>>>>>>>>>>            "metadata.partition": "ID:0, Original Node: red16,
>>>>>>>>>>>>
>>>>>>>>>>>> IODevice:
>>>>>>>>>>>>
>>>>>>>>>>> 0, Active Node: red16",
>>>>>>>>>>            "metadata.port": 0,
>>>>>>>>>>
>>>>>>>>>>>            "metadata.registration.timeout.secs": 60,
>>>>>>>>>>>>            "nc.java.opts": "-Xmx1024m",
>>>>>>>>>>>>            "node.partitions": {
>>>>>>>>>>>>                "red": ["ID:15, Original Node: red, IODevice: 0,
>>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>>> Node:
>>>>>>>>>>>>
>>>>>>>>>>> red"],
>>>>>>>>>>>
>>>>>>>>>>>                "red10": ["ID:6, Original Node: red10, IODevice:
>>>>>>>>>>>> 0,
>>>>>>>>>>>>
>>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>> Node: red10"],
>>>>>>>>>>
>>>>>>>>>>                "red11": ["ID:5, Original Node: red11, IODevice: 0,
>>>>>>>>>>>
>>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>> Node: red11"],
>>>>>>>>>>
>>>>>>>>>>                "red12": ["ID:4, Original Node: red12, IODevice: 0,
>>>>>>>>>>>
>>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>> Node: red12"],
>>>>>>>>>>
>>>>>>>>>>                "red13": ["ID:3, Original Node: red13, IODevice: 0,
>>>>>>>>>>>
>>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>> Node: red13"],
>>>>>>>>>>
>>>>>>>>>>                "red14": ["ID:2, Original Node: red14, IODevice: 0,
>>>>>>>>>>>
>>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>> Node: red14"],
>>>>>>>>>>
>>>>>>>>>>                "red15": ["ID:1, Original Node: red15, IODevice: 0,
>>>>>>>>>>>
>>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>> Node: red15"],
>>>>>>>>>>
>>>>>>>>>>                "red16": ["ID:0, Original Node: red16, IODevice: 0,
>>>>>>>>>>>
>>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>> Node: red16"],
>>>>>>>>>>
>>>>>>>>>>                "red2": ["ID:14, Original Node: red2, IODevice: 0,
>>>>>>>>>>>
>>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>> Node: red2"],
>>>>>>>>>>                "red3": ["ID:13, Original Node: red3, IODevice: 0,
>>>>>>>>>>
>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>> Node: red3"],
>>>>>>>>>>                "red4": ["ID:12, Original Node: red4, IODevice: 0,
>>>>>>>>>>
>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>> Node: red4"],
>>>>>>>>>>                "red5": ["ID:11, Original Node: red5, IODevice: 0,
>>>>>>>>>>
>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>> Node: red5"],
>>>>>>>>>>                "red6": ["ID:10, Original Node: red6, IODevice: 0,
>>>>>>>>>>
>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>> Node: red6"],
>>>>>>>>>>                "red7": ["ID:9, Original Node: red7, IODevice: 0,
>>>>>>>>>>
>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>> Node: red7"],
>>>>>>>>>>                "red8": ["ID:8, Original Node: red8, IODevice: 0,
>>>>>>>>>>
>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>> Node: red8"],
>>>>>>>>>>                "red9": ["ID:7, Original Node: red9, IODevice: 0,
>>>>>>>>>>
>>>>>>>>>>> Active
>>>>>>>>>>>>
>>>>>>>>>>> Node: red9"]
>>>>>>>>>>            },
>>>>>>>>>>
>>>>>>>>>>>            "node.stores": {
>>>>>>>>>>>>                "red": ["/home/clash/asterixStorage/
>>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red/storage"],
>>>>>>>>>>>>
>>>>>>>>>>>                "red10": ["/home/clash/asterixStorage/
>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red10/storage"],
>>>>>>>>>>>>                "red11": ["/home/clash/asterixStorage/
>>>>>>>>>>>> asterixdb5/red11/storage"],
>>>>>>>>>>>>                "red12": ["/home/clash/asterixStorage/
>>>>>>>>>>>> asterixdb5/red12/storage"],
>>>>>>>>>>>>                "red13": ["/home/clash/asterixStorage/
>>>>>>>>>>>> asterixdb5/red13/storage"],
>>>>>>>>>>>>                "red14": ["/home/clash/asterixStorage/
>>>>>>>>>>>> asterixdb5/red14/storage"],
>>>>>>>>>>>>                "red15": ["/home/clash/asterixStorage/
>>>>>>>>>>>> asterixdb5/red15/storage"],
>>>>>>>>>>>>                "red16": ["/home/clash/asterixStorage/
>>>>>>>>>>>> asterixdb5/red16/storage"],
>>>>>>>>>>>>                "red2": ["/home/clash/asterixStorage/
>>>>>>>>>>>> asterixdb5/red2/storage"],
>>>>>>>>>>>>                "red3": ["/home/clash/asterixStorage/
>>>>>>>>>>>> asterixdb5/red3/storage"],
>>>>>>>>>>>>                "red4": ["/home/clash/asterixStorage/
>>>>>>>>>>>> asterixdb5/red4/storage"],
>>>>>>>>>>>>                "red5": ["/home/clash/asterixStorage/
>>>>>>>>>>>> asterixdb5/red5/storage"],
>>>>>>>>>>>>                "red6": ["/home/clash/asterixStorage/
>>>>>>>>>>>> asterixdb5/red6/storage"],
>>>>>>>>>>>>                "red7": ["/home/clash/asterixStorage/
>>>>>>>>>>>> asterixdb5/red7/storage"],
>>>>>>>>>>>>                "red8": ["/home/clash/asterixStorage/
>>>>>>>>>>>> asterixdb5/red8/storage"],
>>>>>>>>>>>>                "red9": ["/home/clash/asterixStorage/
>>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red9/storage"]
>>>>>>>>>>>>
>>>>>>>>>>>            },
>>>>>>>>>>>
>>>>>>>>>>>>            "plot.activate": false,
>>>>>>>>>>>>            "replication.enabled": false,
>>>>>>>>>>>>            "replication.factor": 2,
>>>>>>>>>>>>            "replication.log.batchsize": 4096,
>>>>>>>>>>>>            "replication.log.buffer.numpages": 8,
>>>>>>>>>>>>            "replication.log.buffer.pagesize": 131072,
>>>>>>>>>>>>            "replication.max.remote.recovery.attempts": 5,
>>>>>>>>>>>>            "replication.timeout": 30,
>>>>>>>>>>>>            "storage.buffercache.maxopenfiles": 2147483647,
>>>>>>>>>>>>            "storage.buffercache.pagesize": 131072,
>>>>>>>>>>>>            "storage.buffercache.size": 536870912,
>>>>>>>>>>>>            "storage.lsm.bloomfilter.falsepositiverate": 0.01,
>>>>>>>>>>>>            "storage.memorycomponent.globalbudget": 536870912,
>>>>>>>>>>>>            "storage.memorycomponent.numcomponents": 2,
>>>>>>>>>>>>            "storage.memorycomponent.numpages": 256,
>>>>>>>>>>>>            "storage.memorycomponent.pagesize": 131072,
>>>>>>>>>>>>            "storage.metadata.memorycomponent.numpages": 256,
>>>>>>>>>>>>            "transaction.log.dirs": {
>>>>>>>>>>>>                "red": "/home/clash/asterixStorage/
>>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red/txnlog",
>>>>>>>>>>>>
>>>>>>>>>>>                "red10": "/home/clash/asterixStorage/
>>>>>>>>>>
>>>>>>>>>> asterixdb5/red10/txnlog",
>>>>>>>>>>>
>>>>>>>>>>>                "red11": "/home/clash/asterixStorage/
>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red11/txnlog",
>>>>>>>>>>>>
>>>>>>>>>>>                "red12": "/home/clash/asterixStorage/
>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red12/txnlog",
>>>>>>>>>>>>
>>>>>>>>>>>                "red13": "/home/clash/asterixStorage/
>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red13/txnlog",
>>>>>>>>>>>>
>>>>>>>>>>>                "red14": "/home/clash/asterixStorage/
>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red14/txnlog",
>>>>>>>>>>>>
>>>>>>>>>>>                "red15": "/home/clash/asterixStorage/
>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red15/txnlog",
>>>>>>>>>>>>
>>>>>>>>>>>                "red16": "/home/clash/asterixStorage/
>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red16/txnlog",
>>>>>>>>>>>>
>>>>>>>>>>>                "red2": "/home/clash/asterixStorage/
>>>>>>>>>>>
>>>>>>>>>>>> asterixdb5/red2/txnlog",
>>>>>>>>>>>>
>>>>>>>>>>>                "red3": "/home/clash/asterixStorage/
>>>>>>>>>>> asterixdb5/red3/txnlog",
>>>>>>>>>>>                "red4": "/home/clash/asterixStorage/
>>>>>>>>>>> asterixdb5/red4/txnlog",
>>>>>>>>>>>                "red5": "/home/clash/asterixStorage/
>>>>>>>>>>> asterixdb5/red5/txnlog",
>>>>>>>>>>>                "red6": "/home/clash/asterixStorage/
>>>>>>>>>>> asterixdb5/red6/txnlog",
>>>>>>>>>>>                "red7": "/home/clash/asterixStorage/
>>>>>>>>>>> asterixdb5/red7/txnlog",
>>>>>>>>>>>                "red8": "/home/clash/asterixStorage/
>>>>>>>>>>> asterixdb5/red8/txnlog",
>>>>>>>>>>>                "red9": "/home/clash/asterixStorage/
>>>>>>>>>>> asterixdb5/red9/txnlog"
>>>>>>>>>>>            },
>>>>>>>>>>>
>>>>>>>>>>>            "txn.commitprofiler.reportinterval": 5,
>>>>>>>>>>>>            "txn.job.recovery.memorysize": 67108864,
>>>>>>>>>>>>            "txn.lock.escalationthreshold": 1000,
>>>>>>>>>>>>            "txn.lock.shrinktimer": 5000,
>>>>>>>>>>>>            "txn.lock.timeout.sweepthreshold": 10000,
>>>>>>>>>>>>            "txn.lock.timeout.waitthreshold": 60000,
>>>>>>>>>>>>            "txn.log.buffer.numpages": 8,
>>>>>>>>>>>>            "txn.log.buffer.pagesize": 131072,
>>>>>>>>>>>>            "txn.log.checkpoint.history": 0,
>>>>>>>>>>>>            "txn.log.checkpoint.lsnthreshold": 67108864,
>>>>>>>>>>>>            "txn.log.checkpoint.pollfrequency": 120,
>>>>>>>>>>>>            "txn.log.partitionsize": 268435456,
>>>>>>>>>>>>            "web.port": 19001,
>>>>>>>>>>>>            "web.queryinterface.port": 19006,
>>>>>>>>>>>>            "web.secondary.port": 19005
>>>>>>>>>>>>        },
>>>>>>>>>>>>        "fullShutdownUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/shutdown?all=true",
>>>>>>>>>>>>        "metadata_node": "red16",
>>>>>>>>>>>>        "ncs": [
>>>>>>>>>>>>            {
>>>>>>>>>>>>                "configUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red15/
>>>>>>>>>>>> config
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "node_id": "red15",
>>>>>>>>>>>>                "partitions": [{
>>>>>>>>>>>>                    "active": true,
>>>>>>>>>>>>                    "partition_id": "partition_1"
>>>>>>>>>>>>                }],
>>>>>>>>>>>>                "state": "ACTIVE",
>>>>>>>>>>>>                "statsUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red15/stats
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "threadDumpUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/
>>>>>>>>>>>>
>>>>>>>>>>>> red15/threaddump
>>>>>>>>>>>>
>>>>>>>>>>> "
>>>>>>>>>>
>>>>>>>>>            },
>>>>>>>>>
>>>>>>>>>>            {
>>>>>>>>>>>
>>>>>>>>>>>>                "configUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red14/
>>>>>>>>>>>> config
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "node_id": "red14",
>>>>>>>>>>>>                "partitions": [{
>>>>>>>>>>>>                    "active": true,
>>>>>>>>>>>>                    "partition_id": "partition_2"
>>>>>>>>>>>>                }],
>>>>>>>>>>>>                "state": "ACTIVE",
>>>>>>>>>>>>                "statsUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red14/stats
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "threadDumpUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/
>>>>>>>>>>>>
>>>>>>>>>>>> red14/threaddump
>>>>>>>>>>>>
>>>>>>>>>>> "
>>>>>>>>>>
>>>>>>>>>            },
>>>>>>>>>
>>>>>>>>>>            {
>>>>>>>>>>>
>>>>>>>>>>>>                "configUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red16/
>>>>>>>>>>>> config
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "node_id": "red16",
>>>>>>>>>>>>                "partitions": [{
>>>>>>>>>>>>                    "active": true,
>>>>>>>>>>>>                    "partition_id": "partition_0"
>>>>>>>>>>>>                }],
>>>>>>>>>>>>                "state": "ACTIVE",
>>>>>>>>>>>>                "statsUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red16/stats
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "threadDumpUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/
>>>>>>>>>>>>
>>>>>>>>>>>> red16/threaddump
>>>>>>>>>>>>
>>>>>>>>>>> "
>>>>>>>>>>
>>>>>>>>>            },
>>>>>>>>>
>>>>>>>>>>            {
>>>>>>>>>>>
>>>>>>>>>>>>                "configUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red11/
>>>>>>>>>>>> config
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "node_id": "red11",
>>>>>>>>>>>>                "partitions": [{
>>>>>>>>>>>>                    "active": true,
>>>>>>>>>>>>                    "partition_id": "partition_5"
>>>>>>>>>>>>                }],
>>>>>>>>>>>>                "state": "ACTIVE",
>>>>>>>>>>>>                "statsUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red11/stats
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "threadDumpUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/
>>>>>>>>>>>>
>>>>>>>>>>>> red11/threaddump
>>>>>>>>>>>>
>>>>>>>>>>> "
>>>>>>>>>>
>>>>>>>>>            },
>>>>>>>>>
>>>>>>>>>>            {
>>>>>>>>>>>
>>>>>>>>>>>>                "configUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red10/
>>>>>>>>>>>> config
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "node_id": "red10",
>>>>>>>>>>>>                "partitions": [{
>>>>>>>>>>>>                    "active": true,
>>>>>>>>>>>>                    "partition_id": "partition_6"
>>>>>>>>>>>>                }],
>>>>>>>>>>>>                "state": "ACTIVE",
>>>>>>>>>>>>                "statsUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red10/stats
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "threadDumpUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/
>>>>>>>>>>>>
>>>>>>>>>>>> red10/threaddump
>>>>>>>>>>>>
>>>>>>>>>>> "
>>>>>>>>>>
>>>>>>>>>            },
>>>>>>>>>
>>>>>>>>>>            {
>>>>>>>>>>>
>>>>>>>>>>>>                "configUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red13/
>>>>>>>>>>>> config
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "node_id": "red13",
>>>>>>>>>>>>                "partitions": [{
>>>>>>>>>>>>                    "active": true,
>>>>>>>>>>>>                    "partition_id": "partition_3"
>>>>>>>>>>>>                }],
>>>>>>>>>>>>                "state": "ACTIVE",
>>>>>>>>>>>>                "statsUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red13/stats
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "threadDumpUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/
>>>>>>>>>>>>
>>>>>>>>>>>> red13/threaddump
>>>>>>>>>>>>
>>>>>>>>>>> "
>>>>>>>>>>
>>>>>>>>>            },
>>>>>>>>>
>>>>>>>>>>            {
>>>>>>>>>>>
>>>>>>>>>>>>                "configUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red12/
>>>>>>>>>>>> config
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "node_id": "red12",
>>>>>>>>>>>>                "partitions": [{
>>>>>>>>>>>>                    "active": true,
>>>>>>>>>>>>                    "partition_id": "partition_4"
>>>>>>>>>>>>                }],
>>>>>>>>>>>>                "state": "ACTIVE",
>>>>>>>>>>>>                "statsUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red12/stats
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "threadDumpUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/
>>>>>>>>>>>>
>>>>>>>>>>>> red12/threaddump
>>>>>>>>>>>>
>>>>>>>>>>> "
>>>>>>>>>>
>>>>>>>>>            },
>>>>>>>>>
>>>>>>>>>>            {
>>>>>>>>>>>
>>>>>>>>>>>>                "configUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red6/config
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "node_id": "red6",
>>>>>>>>>>>>                "partitions": [{
>>>>>>>>>>>>                    "active": true,
>>>>>>>>>>>>                    "partition_id": "partition_10"
>>>>>>>>>>>>                }],
>>>>>>>>>>>>                "state": "ACTIVE",
>>>>>>>>>>>>                "statsUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red6/stats
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "threadDumpUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red6/
>>>>>>>>>>>>
>>>>>>>>>>>> threaddump"
>>>>>>>>>>>>
>>>>>>>>>>>            },
>>>>>>>>>>            {
>>>>>>>>>>
>>>>>>>>>>>                "configUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red/config
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "node_id": "red",
>>>>>>>>>>>>                "partitions": [{
>>>>>>>>>>>>                    "active": true,
>>>>>>>>>>>>                    "partition_id": "partition_15"
>>>>>>>>>>>>                }],
>>>>>>>>>>>>                "state": "ACTIVE",
>>>>>>>>>>>>                "statsUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red/stats",
>>>>>>>>>>>>                "threadDumpUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red/thre
>>>>>>>>>>>> addump
>>>>>>>>>>>>
>>>>>>>>>>>> "
>>>>>>>>>>>>
>>>>>>>>>>>            },
>>>>>>>>>>            {
>>>>>>>>>>
>>>>>>>>>>>                "configUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red5/config
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "node_id": "red5",
>>>>>>>>>>>>                "partitions": [{
>>>>>>>>>>>>                    "active": true,
>>>>>>>>>>>>                    "partition_id": "partition_11"
>>>>>>>>>>>>                }],
>>>>>>>>>>>>                "state": "ACTIVE",
>>>>>>>>>>>>                "statsUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red5/stats
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "threadDumpUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red5/
>>>>>>>>>>>>
>>>>>>>>>>>> threaddump"
>>>>>>>>>>>>
>>>>>>>>>>>            },
>>>>>>>>>>            {
>>>>>>>>>>
>>>>>>>>>>>                "configUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red8/config
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "node_id": "red8",
>>>>>>>>>>>>                "partitions": [{
>>>>>>>>>>>>                    "active": true,
>>>>>>>>>>>>                    "partition_id": "partition_8"
>>>>>>>>>>>>                }],
>>>>>>>>>>>>                "state": "ACTIVE",
>>>>>>>>>>>>                "statsUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red8/stats
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "threadDumpUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red8/
>>>>>>>>>>>>
>>>>>>>>>>>> threaddump"
>>>>>>>>>>>>
>>>>>>>>>>>            },
>>>>>>>>>>            {
>>>>>>>>>>
>>>>>>>>>>>                "configUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red7/config
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "node_id": "red7",
>>>>>>>>>>>>                "partitions": [{
>>>>>>>>>>>>                    "active": true,
>>>>>>>>>>>>                    "partition_id": "partition_9"
>>>>>>>>>>>>                }],
>>>>>>>>>>>>                "state": "ACTIVE",
>>>>>>>>>>>>                "statsUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red7/stats
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "threadDumpUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red7/
>>>>>>>>>>>>
>>>>>>>>>>>> threaddump"
>>>>>>>>>>>>
>>>>>>>>>>>            },
>>>>>>>>>>            {
>>>>>>>>>>
>>>>>>>>>>>                "configUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red2/config
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "node_id": "red2",
>>>>>>>>>>>>                "partitions": [{
>>>>>>>>>>>>                    "active": true,
>>>>>>>>>>>>                    "partition_id": "partition_14"
>>>>>>>>>>>>                }],
>>>>>>>>>>>>                "state": "ACTIVE",
>>>>>>>>>>>>                "statsUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red2/stats
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "threadDumpUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red2/
>>>>>>>>>>>>
>>>>>>>>>>>> threaddump"
>>>>>>>>>>>>
>>>>>>>>>>>            },
>>>>>>>>>>            {
>>>>>>>>>>
>>>>>>>>>>>                "configUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red4/config
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "node_id": "red4",
>>>>>>>>>>>>                "partitions": [{
>>>>>>>>>>>>                    "active": true,
>>>>>>>>>>>>                    "partition_id": "partition_12"
>>>>>>>>>>>>                }],
>>>>>>>>>>>>                "state": "ACTIVE",
>>>>>>>>>>>>                "statsUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red4/stats
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "threadDumpUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red4/
>>>>>>>>>>>>
>>>>>>>>>>>> threaddump"
>>>>>>>>>>>>
>>>>>>>>>>>            },
>>>>>>>>>>            {
>>>>>>>>>>
>>>>>>>>>>>                "configUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red3/config
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "node_id": "red3",
>>>>>>>>>>>>                "partitions": [{
>>>>>>>>>>>>                    "active": true,
>>>>>>>>>>>>                    "partition_id": "partition_13"
>>>>>>>>>>>>                }],
>>>>>>>>>>>>                "state": "ACTIVE",
>>>>>>>>>>>>                "statsUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red3/stats
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "threadDumpUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red3/
>>>>>>>>>>>>
>>>>>>>>>>>> threaddump"
>>>>>>>>>>>>
>>>>>>>>>>>            },
>>>>>>>>>>            {
>>>>>>>>>>
>>>>>>>>>>>                "configUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red9/config
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "node_id": "red9",
>>>>>>>>>>>>                "partitions": [{
>>>>>>>>>>>>                    "active": true,
>>>>>>>>>>>>                    "partition_id": "partition_7"
>>>>>>>>>>>>                }],
>>>>>>>>>>>>                "state": "ACTIVE",
>>>>>>>>>>>>                "statsUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red9/stats
>>>>>>>>>>>> ",
>>>>>>>>>>>>                "threadDumpUri":
>>>>>>>>>>>> "http://scai01.cs.ucla.edu:19002/admin/cluster/node/red9/
>>>>>>>>>>>>
>>>>>>>>>>>> threaddump"
>>>>>>>>>>>>
>>>>>>>>>>>            }
>>>>>>>>>>        ],
>>>>>>>>>>
>>>>>>>>>>>        "shutdownUri": "http://scai01.cs.ucla.edu:190
>>>>>>>>>>>> 02/admin/shutdown
>>>>>>>>>>>>
>>>>>>>>>>>> ",
>>>>>>>>>>>>
>>>>>>>>>>>        "state": "ACTIVE",
>>>>>>>>>>        "versionUri": "http://scai01.cs.ucla.edu:190
>>>>>>>>>> 02/admin/version"
>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> 2.Catalog_return:2.28G
>>>>>>>>>>>>
>>>>>>>>>>>> catalog_sales:31.01G
>>>>>>>>>>>>
>>>>>>>>>>>> inventory:8.63G
>>>>>>>>>>>>
>>>>>>>>>>>> 3.As for Pig and Hive, I always use the default configuration. I
>>>>>>>>>>>>
>>>>>>>>>>>> didn't
>>>>>>>>>>>>
>>>>>>>>>>> set
>>>>>>>>>>
>>>>>>>>>> the partition things for them. And for Spark, we use 200
>>>>>>>>>>>
>>>>>>>>>>>> partitions,
>>>>>>>>>>>>
>>>>>>>>>>> which
>>>>>>>>>> may be improved and just not bad. For AsterixDB, I also set the
>>>>>>>>>>
>>>>>>>>>>> cluster
>>>>>>>>>>>>
>>>>>>>>>>> using default value of partition and JVM things (I didn't
>>>>>>>>>> manually
>>>>>>>>>>
>>>>>>>>>> set
>>>>>>>>>>>
>>>>>>>>>>> these parameters).
>>>>>>>>>>
>>>>>>>>>> On Tue, Dec 20, 2016 at 5:58 PM, Yingyi Bu <buyingyi@gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>> Mingda,
>>>>>>>>>>
>>>>>>>>>>         1. Can you paste the returned JSON of http://<master
>>>>>>>>>>>
>>>>>>>>>>>> node>:19002/admin/cluster at your side? (Pls replace <master
>>>>>>>>>>>>>
>>>>>>>>>>>>> node>
>>>>>>>>>>>>>
>>>>>>>>>>>> with
>>>>>>>>>>>
>>>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>>> actual master node name or IP)
>>>>>>>>>>>>
>>>>>>>>>>>>>         2. Can you list the individual size of each dataset
>>>>>>>>>>>>> involved
>>>>>>>>>>>>>
>>>>>>>>>>>>> in
>>>>>>>>>>>>>
>>>>>>>>>>>> the
>>>>>>>>>>> query, e.g., catalog_returns, catalog_sales, and inventory?  (I
>>>>>>>>>>>
>>>>>>>>>>>> assume
>>>>>>>>>>>>
>>>>>>>>>>>> 100GB is the overall size?)
>>>>>>>>>>>
>>>>>>>>>>>         3. Do Spark/Hive/Pig saturate all CPUs on all machines,
>>>>>>>>>>>>
>>>>>>>>>>>>> i.e.,
>>>>>>>>>>>>>
>>>>>>>>>>>> how
>>>>>>>>>>>
>>>>>>>>>> many
>>>>>>>>>>
>>>>>>>>>>> partitions are running on each machine?  (It seems that your
>>>>>>>>>>>>
>>>>>>>>>>>>> AsterixDB
>>>>>>>>>>>>>
>>>>>>>>>>>> configuration wouldn't saturate all CPUs for queries --- in the
>>>>>>>>>>>
>>>>>>>>>>> current
>>>>>>>>>>>>
>>>>>>>>>>>> AsterixDB master, the computation parallelism is set to be the
>>>>>>>>>>>
>>>>>>>>>>> same
>>>>>>>>>>>>
>>>>>>>>>>>> as
>>>>>>>>>>>
>>>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>>> storage parallelism (i.e., the number of iodevices on each NC).
>>>>>>>>>>>>
>>>>>>>>>>>>> I've
>>>>>>>>>>>>>
>>>>>>>>>>>> submitted a new patch that allow flexible computation
>>>>>>>>>>> parallelism,
>>>>>>>>>>> which
>>>>>>>>>>>
>>>>>>>>>> should be able to get merged into master very soon.)
>>>>>>>>>>
>>>>>>>>>>>         Thanks!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Yingyi
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Dec 20, 2016 at 5:44 PM, mingda li <
>>>>>>>>>>>>>
>>>>>>>>>>>>> limingda1993@gmail.com
>>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>> Oh, sure. When we test the 100G multiple join, we find
>>>>>>>>>>
>>>>>>>>>>> AsterixDB
>>>>>>>>>>>>>
>>>>>>>>>>>>> is
>>>>>>>>>>>>
>>>>>>>>>>> slower
>>>>>>>>>
>>>>>>>>>> than Spark (but still faster than Pig and Hive).
>>>>>>>>>>>
>>>>>>>>>>>> I can share with you the both plots: 1-10G.eps and 1-100G.eps.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (We
>>>>>>>>>>>>>>
>>>>>>>>>>>>> will
>>>>>>>>>>>>
>>>>>>>>>>> only use 1-10G.eps in our paper).
>>>>>>>>>>>
>>>>>>>>>>>> And thanks for Ian's advice:* The dev list generally strips
>>>>>>>>>>>>>
>>>>>>>>>>>>>> attachments.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Maybe you can just put the config inline? Or link to a
>>>>>>>>>>>>> pastebin/gist?*
>>>>>>>>>>>>>
>>>>>>>>>>>>> I know why you can't see the attachments. So I move the plots
>>>>>>>>>>>>
>>>>>>>>>>>> with
>>>>>>>>>>>>>
>>>>>>>>>>>>> two
>>>>>>>>>>>>
>>>>>>>>>>> documents to my Dropbox.
>>>>>>>>>>>
>>>>>>>>>>>> You can find the
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1-10G.eps here: https://www.dropbox.com/s/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> rk3xg6gigsfcuyq/1-10G.eps?dl=0
>>>>>>>>>>>>>>
>>>>>>>>>>>>> 1-100G.eps here:https://www.dropbox.com/
>>>>>>>>>>>>> s/tyxnmt6ehau2ski/1-100G.eps
>>>>>>>>>>>>>
>>>>>>>>>>>>> ?
>>>>>>>>>>>>
>>>>>>>>>>> dl=0
>>>>>>>>>>>
>>>>>>>>>>>> cc_conf.pdf here: https://www.dropbox.com/s/
>>>>>>>>>>>>>
>>>>>>>>>>>>>> y3of1s17qdstv5f/cc_conf.pdf?
>>>>>>>>>>>>>>
>>>>>>>>>>>>> dl=0
>>>>>>>>>>>>>
>>>>>>>>>>>>> CompleteQuery.pdf here:
>>>>>>>>>>>>>> https://www.dropbox.com/s/lml3fzxfjcmf2c1/CompleteQuery.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> pdf?dl=0
>>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Dec 20, 2016 at 4:40 PM, Tyson Condie <
>>>>>>>>>>>>
>>>>>>>>>>> tcondie.ucla@gmail.com>
>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Mingda: Please also share the numbers for 100GB, which show
>>>>>>>>>>>>>
>>>>>>>>>>>>>> AsterixDB
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> not
>>>>>>>>>>>>> quite doing as well as Spark. These 100GB results will not be
>>>>>>>>>>>>>
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> our
>>>>>>>>>>>>>
>>>>>>>>>>>> submission version, since they’re not needed for the desired
>>>>>>>>>>>
>>>>>>>>>>>> message:
>>>>>>>>>>>>> picking the right join order matters. Nevertheless, I’d like
>>>>>>>>>>>>> to
>>>>>>>>>>>>> get a
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message