Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C3B1317600 for ; Wed, 28 Jan 2015 07:05:21 +0000 (UTC) Received: (qmail 35497 invoked by uid 500); 28 Jan 2015 07:05:19 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 35454 invoked by uid 500); 28 Jan 2015 07:05:19 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 35444 invoked by uid 99); 28 Jan 2015 07:05:19 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jan 2015 07:05:19 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of wanshenghua@gmail.com designates 209.85.214.180 as permitted sender) Received: from [209.85.214.180] (HELO mail-ob0-f180.google.com) (209.85.214.180) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jan 2015 07:04:53 +0000 Received: by mail-ob0-f180.google.com with SMTP id uz6so17734677obc.11 for ; Tue, 27 Jan 2015 23:04:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=nFvQigI9aYqC7ydzF68/kHKptYH8+LlB1Ep0vVG0LMw=; b=n0kypWIWiAcbsTNf1l6OTa1RoC+NN3Ta7dYlFW2+gVYBjKGgHwwHt8cCiCIJgIY24p 99qUGs64Am8JI4Yt48LjtxjX1CM9tvLF0FNdnC34AN1VayKGTUkhpRgKfzRMvkj2ufqz HWxKrucrebClbGYjmT6/nPIqkuz2q7WiGuo3C0pRW+G/6oB8JzHNCbEwhSw4dV0+biMZ BIdtuLu5U/rsMuMJMriqqc3VwZOVl35k+vCE6iQCdJtrwcTdKyiJpDY2hf9YYImJrbcI l7V9JXmehLoJwGwZMM8Kl/sdMgzCf40L25eCfczrCzP4ixwqt+GLoEcbs3uszV147JX/ 5R4w== MIME-Version: 1.0 X-Received: by 10.182.153.39 with SMTP id vd7mr1187535obb.78.1422428691229; Tue, 27 Jan 2015 23:04:51 -0800 (PST) Received: by 10.182.79.36 with HTTP; Tue, 27 Jan 2015 23:04:51 -0800 (PST) In-Reply-To: References: Date: Tue, 27 Jan 2015 23:04:51 -0800 Message-ID: Subject: Re: cqlinputformat and retired cqlpagingingputformat creates lots of connections to query the server From: "Shenghua(Daniel) Wan" To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=089e013a01e6dbf39d050db0fcb0 X-Virus-Checked: Checked by ClamAV on apache.org --089e013a01e6dbf39d050db0fcb0 Content-Type: text/plain; charset=UTF-8 Hi, Huiliang, Great to hear from you, again! Image you have 3 nodes, replication factor=1, and using default number of tokens. You will have 3*256 mappers... In that case, you will be soon out of mappers or reach the limit. On Tue, Jan 27, 2015 at 10:59 PM, Huiliang Zhang wrote: > Hi Shenghua, as I understand, each range is assigned to a mapper. Mapper > will not share connections. So, it needs at least 256 connections to read > all. But all 256 connections should not be set up at the same time unless > you have 256 mappers running at the same time. > > On Tue, Jan 27, 2015 at 9:34 PM, Shenghua(Daniel) Wan < > wanshenghua@gmail.com> wrote: > >> By default, each C* node is set with 256 tokens. On a local 1-node C* >> server, my hadoop drop creates 256 connections to the server. Is there any >> way to control this behavior? e.g. reduce the number of connections to a >> pre-configured gap. >> >> I debugged C* source code and found the client asks for partition ranges, >> or virtual nodes. Then the client was told by server there were 257 ranges, >> corresponding to 257 column family splits. >> >> Here is a snapshot of my logs >> >> 15/01/27 18:02:20 DEBUG hadoop.AbstractColumnFamilyInputFormat: adding >> ColumnFamilySplit((9121856086738887846, '-9223372036854775808] @[localhost]) >> ... >> totally 257 splits. >> >> The problem is the user might only want all the data via a "select *" >> like statement. It seems that 257 connections to query the rows are >> necessary. However, is there any way to prohibit 257 concurrent >> connections? >> >> My C* version is 2.0.11 and I also tried CqlPagingInputFormat, which has >> same behavior. >> >> Thank you. >> >> -- >> >> Regards, >> Shenghua (Daniel) Wan >> > > -- Regards, Shenghua (Daniel) Wan --089e013a01e6dbf39d050db0fcb0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi, Huiliang,=C2=A0
Great to hear from you, again!
Image you have = 3 nodes, replication factor=3D1, and using default number of tokens. You wi= ll have 3*256 mappers... In that case, you will be soon out of mappers or r= each the limit.


On Tue, Jan 27, 2015 at 10:59 PM, Huiliang Zhang <zhlntu@gmai= l.com> wrote:
Hi Shenghua, as I understand, each range is assigned to a mapper. Mappe= r will not share connections. So, it needs at least 256 connections to read= all. But all 256 connections should not be set up at the same time unless = you have 256 mappers running at the same time.
<= div class=3D"h5">

= On Tue, Jan 27, 2015 at 9:34 PM, Shenghua(Daniel) Wan <= ;wanshenghua@gma= il.com> wrote:
B= y default, each C* node is set with 256 tokens. On a local 1-node C* server= , my hadoop drop creates 256 connections to the server. Is there any way to= control this behavior? e.g. reduce the number of connections to a pre-conf= igured gap.

I debugged C* source code and found the client asks for part= ition ranges, or virtual nodes. Then the client was told by server there we= re 257 ranges, corresponding to 257 column family splits.=C2=A0

Here is = a snapshot of my logs

15/01/27 18:02:20 DEBUG hadoop.AbstractColumnFamil= yInputFormat: adding ColumnFamilySplit((9121856086738887846, '-92233720= 36854775808] @[localhost])
...
totally 257 splits.

The problem is t= he user might only want all the data via a "select *" like statem= ent. It seems that 257 connections to query the rows are necessary. However= , is there any way to prohibit 257 concurrent connections?=C2=A0

<= div class=3D"gmail_default" style=3D"font-family:tahoma,sans-serif">My C* v= ersion is 2.0.11 and I also tried CqlPagingInputFormat, which has same beha= vior.

Thank you.

--
Regards,
Shenghua (Daniel= ) Wan




--
=

Regards,
Shenghua (Daniel) Wan
--089e013a01e6dbf39d050db0fcb0--