hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From m0j0j0j0 <silvio.technolog...@gmail.com>
Subject Help with Multiple Connections
Date Sun, 19 Feb 2012 01:00:32 GMT

Hello all, ive been developing an automation tool for myself.

After modifying the example code for a multithreaded get request, and
setting up all my logic, 
i had everything working great. Testing with 1340 urls everything worked
fine.

Further testing i loaded over 3000 urls and connection pool closed
IllegalStateException
Ive been playing with this for a day or 2.

Keep getting an exception that i cant seem to track down!

here is my httpget request runnable with all this testing ive done i
completely removed my logic
and placed a simple alive check for testing purposes..

	public void run() {
		HttpEntity entity = null;
		ResponseHandler<byte[]> handler = new ResponseHandler<byte[]>() {
		    public byte[] handleResponse(
		            HttpResponse response) throws ClientProtocolException,
IOException {
		        HttpEntity entity = response.getEntity();
		        if (entity != null) {
		        	mdl1.setValueAt(response.getStatusLine().getStatusCode(), id-1,
0);		        
		            return EntityUtils.toByteArray(entity);
		        } else {
		        	mdl1.setValueAt("TO", id-1, 0);
		        	httpget.abort();
		            return null;
		        }
		    }
		};

		try {
			byte[] response = httpClient.execute(httpget, handler);
			EntityUtils.consume(entity);
		} catch (ClientProtocolException e) {
			mdl1.setValueAt("TO", id-1, 0);
			System.out.println("CLIENTPROTOEXE " + e.getMessage() + "\n" +
e.getCause());
			httpget.abort();
			//e.printStackTrace();
		} catch (IOException e) {
			mdl1.setValueAt("TO", id-1, 0);
			System.out.println(e.getMessage());
			httpget.abort();
			//e.printStackTrace();
		}
		try {
			EntityUtils.consume(entity);
		} catch (IOException e) {
			
			// TODO Auto-generated catch block
			System.out.println("CONSUME + " + e.getMessage());
			//e.printStackTrace();
		}
}



in my starting thread ive tried multiple things.  My last attempt was to try
an executor 
to limit the ammount of threads, this seem to work until almost halfway
through the list
and i keep getting java.lang.IllegalStateException: Connection pool shut
down

				for (int j = 0; j < dmdl1.getRowCount(); j++) {
					exec.execute(new GetThread(httpclient, new HttpGet(dmdl1.getValueAt(j,
2).toString()), j + 1, dmdl1, (ThreadSafeClientConnManager) cm));
				}


My original try was to use the traditional method which i though would do
the work for me.
after creating each getrequest thread i used the code below.  Which was
working fine 
when i was doing testing and loaded 1340 urls, but when i try over 3000 urls
it fails.  Im kindof 
new to java mostly im out of practice from not using it for quite some time
but im eager to get 
back into it.

//            for (int j = 0; j < threads.length; j++) {
//            		//System.out.println(cm.getConnectionsInPool() + "-- thread
# " + j + " " + dmdl1.getValueAt(j, 2).toString());
//                	threads[j].start(); 
//            }
//
//            // join the threads
//            for (int j = 0; j < threads.length; j++) {
//                  threads[j].join();
//                     System.out.println("Joined id " + j);
//            } 


Im using HttpClient 4.1.2 in my project, please any insight on how to
initiate these connections
or manage them would be helpful.

			SchemeRegistry schemeRegistry = new SchemeRegistry();
				schemeRegistry.register(
				        new Scheme("http", 80, PlainSocketFactory.getSocketFactory()));

				ClientConnectionManager cm = new
ThreadSafeClientConnManager(schemeRegistry);
				HttpClient httpClient = new DefaultHttpClient(cm);

	    ((ThreadSafeClientConnManager) cm).setMaxTotal(50);
        ((ThreadSafeClientConnManager) cm).setDefaultMaxPerRoute(10);
        HttpClient httpclient = new DefaultHttpClient(cm);
        httpclient.getParams().setParameter("http.socket.timeout", new
Integer(10000));

I thought using threadsafeconnmanager it would do all the work for me:(

Then i thought of a way to have a locking thread, that waits for
connectionmanager.getconnectionsinpool to drop below half th total and start
initiating more connections.  Again im kinda confused i'd love any help you
can provide. 

Thank you for your time.






-- 
View this message in context: http://old.nabble.com/Help-with-Multiple-Connections-tp33350254p33350254.html
Sent from the HttpClient-User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message