`

HttpClient4.3.3 使用样例—获取静态资源

阅读更多

编写不易,转载请注明(http://shihlei.iteye.com/blog/2067688)!

一 概述

 

    HttpClient是最常见的Http请求工具,很好的封装 Http请求,响应,Cookies操作,适合网络爬虫处理静态页面及其他资源。

    HttpCient能自动处理302客户端重定向,这个很喜欢!

    HttpClient4.3.3 API较之前的版本有不少改变,本文描述请求响应,设置cookies,设置代理等方法。

 

二 版本

 

<dependency>
	<groupId>org.apache.httpcomponents</groupId>
	<artifactId>httpclient</artifactId>
	<version>4.3.3</version>
</dependency>

 

 

三 样例

(1)请求响应

 

/**
 * 发送get请求
 * 
 * @param url
 * @throws Exception
 */
public byte[] sendGetRequest(String url) throws Exception {
	HttpClient httpClient = HttpClients.createDefault();
	HttpGet httpGet = new HttpGet(url);
	System.out.println("Send Get Request : " + url);
	return sendRequest(httpClient,httpGet);
}

/**
 * 发送post 请求
 * 
 * @param request
 * @throws Exception
 */
public byte[] sendPostRequest(String url, Map<String, String> params) throws Exception {
	HttpClient httpClient = HttpClients.createDefault();
	HttpPost httpPost = new HttpPost(url);
	// 设置参数
	if (params != null && params.size() > 0) {
		List<NameValuePair> nvps = new ArrayList<NameValuePair>();
		for (Entry<String, String> param : params.entrySet()) {
			nvps.add(new BasicNameValuePair(param.getKey(), param.getValue()));
		}
		httpPost.setEntity(new UrlEncodedFormEntity(nvps));
	}

	System.out.println("Send Post Request : " + url + " , " + params);
	return sendRequest(httpClient,httpPost);
}
// 底层请求获得响应数据
private byte[] sendRequest(HttpClient httpClient ,HttpRequestBase request) throws Exception {
	CloseableHttpResponse response = httpclient.execute(request);

	// 获取返回的状态列表
	StatusLine statusLine = response.getStatusLine();
	System.out.println("StatusLine : " + statusLine);

	if (statusLine.getStatusCode() == HttpStatus.SC_OK) {
		try {
			// 获取response entity
			HttpEntity entity = response.getEntity();
			System.out.println("getContentType : " + entity.getContentType().getValue());
			System.out.println("getContentEncoding : " + entity.getContentEncoding().getValue());

			// 读取响应内容
			byte[] responseBody = EntityUtils.toByteArray(entity);

			// 关闭响应流
			EntityUtils.consume(entity);
			return responseBody;
		} finally {
			response.close();
		}
	}
	return null;
}

 

(2)超时和重定向

在execute前针对Request设置超时的Config

 

//设置超时
private void setTimeOut(HttpRequestBase request, int timeout) {
	RequestConfig requestConfig = RequestConfig.custom().setConnectionRequestTimeout(timeout)
			.setConnectTimeout(timeout).setSocketTimeout(timeout).setRedirectsEnabled(true).build();
	request.setConfig(requestConfig);
}

 

(3)代理

在创建HttpClient实例的时候,设置代理

//获得设置代理的httpClent实例
private CloseableHttpClient getHttpClientWithProxy(HttpProxy proxy){
	HttpHost proxyHost = new HttpHost(proxy.getHost(), proxy.getPort());
	CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
	credentialsProvider.setCredentials(new AuthScope(proxy.getHost(), proxy.getPort()),
			new UsernamePasswordCredentials(proxy.getUser(), proxy.getPassword()));

	return HttpClients.custom().setProxy(proxyHost).setDefaultCredentialsProvider(credentialsProvider)
			.build();
}

 

这里有个辅助类:

 

package x.http.simple;


/**
 * Http代理
 * 
 * @author shilei
 * 
 */
public class HttpProxy {
	private String proxy = "http";
	private String host;
	private int port;
	private String user;
	private String password;
	public String getProxy() {
		return proxy;
	}
	public void setProxy(String proxy) {
		this.proxy = proxy;
	}
	public String getHost() {
		return host;
	}
	public void setHost(String host) {
		this.host = host;
	}
	public int getPort() {
		return port;
	}
	public void setPort(int port) {
		this.port = port;
	}
	public String getUser() {
		return user;
	}
	public void setUser(String user) {
		this.user = user;
	}
	public String getPassword() {
		return password;
	}
	public void setPassword(String password) {
		this.password = password;
	}
}

 

(4)Cookies:可以用于认证数据设置

1)全局Cookies设置:在创建HttpClient时

 

/**
 * 设置全局Cookies
 * 
 * @param cookies
 * @return
 */
public HttpClient setGlobeCookieHttpClient(Map<String, String> cookies) {
	HttpClient client = null;
	if (cookies != null && cookies.size() > 0) {
		BasicCookieStore cookieStore = new BasicCookieStore();

		for (Entry<String, String> c : cookies.entrySet()) {
			BasicClientCookie cookie = new BasicClientCookie(c.getKey(), c.getValue());
			cookie.setDomain("sina.com");
			cookie.setPath("/");
			cookieStore.addCookie(cookie);
		}
		client = HttpClientBuilder.create().setDefaultCookieStore(cookieStore).build();
	} else {
		client = HttpClients.createDefault();
	}
	return client;
}

 

2)请求Cookies设置

法一:直接在请求头添加

 

/**
 * 设置Cookie方法1
 * 
 * @param request
 * @param cookies
 */
public void setRequestCookiesStr(HttpUriRequest request, Map<String, String> cookies) {
	StringBuilder cookieStr = new StringBuilder();
	if (cookies != null && cookies.size() > 0) {
		for (Entry<String, String> cookie : cookies.entrySet()) {
			cookieStr.append(cookie.getKey()).append('=').append(cookie.getValue()).append(';');
		}
		cookieStr.deleteCharAt(cookieStr.length() - 1);
		request.addHeader("Cookie", cookieStr.toString());
	}
}

 

法二:请求是通过context设置

 

/**
 * 设置Cookie 方法2
 * 
 * httpclient.execute(request, context) 方法进行添加
 * 
 * @param cookies
 * @return
 */
public HttpContext getetRequestCookieContext(Map<String, String> cookies) {
	HttpContext localContext = null;
	if (cookies != null && cookies.size() > 0) {
		BasicCookieStore cookieStore = new BasicCookieStore();

		for (Entry<String, String> c : cookies.entrySet()) {
			BasicClientCookie cookie = new BasicClientCookie(c.getKey(), c.getValue());
			cookie.setDomain("sina.com");
			cookie.setPath("/");
			cookieStore.addCookie(cookie);
		}
		localContext = new BasicHttpContext();
		localContext.setAttribute(HttpClientContext.COOKIE_STORE, cookieStore);
	}
	return localContext;
}

 

3)读取响应的Cookies

 

/**
	 * 响应中获取cookie
	 * 
	 * @param response
	 */
	public void getResponseCookies(HttpResponse response) {
		//法1 
		HeaderIterator headerIter = response.headerIterator("Set-Cookie");
		while (headerIter.hasNext()) {
			Object obj = headerIter.next();
			if (obj != null) {
				String cookie = ((Header) headerIter.next()).getValue();

				String cookieRecord = cookie.substring(0, cookie.indexOf(';'));
				System.out.println(cookieRecord);
			}

		}

		//法2
		HeaderElementIterator headerElementIter = new BasicHeaderElementIterator(response.headerIterator("Set-Cookie"));
		while (headerElementIter.hasNext()) {
			HeaderElement elem = headerElementIter.nextElement();
			System.out.println("Element : " + elem.getName() + " = " + elem.getValue());
			NameValuePair[] params = elem.getParameters();
			for (int i = 0; i < params.length; i++) {
				System.out.println(" Params : " + params[i].getName() + " | " + params[i].getValue());
			}
		}
	}

 

 附录:完整代码

 

package x.http.simple.httpclient;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;

import org.apache.http.Header;
import org.apache.http.HeaderElement;
import org.apache.http.HeaderElementIterator;
import org.apache.http.HeaderIterator;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHost;
import org.apache.http.HttpResponse;
import org.apache.http.HttpStatus;
import org.apache.http.NameValuePair;
import org.apache.http.StatusLine;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.client.HttpClient;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.methods.HttpRequestBase;
import org.apache.http.client.methods.HttpUriRequest;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.impl.client.BasicCookieStore;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.cookie.BasicClientCookie;
import org.apache.http.message.BasicHeaderElementIterator;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.protocol.BasicHttpContext;
import org.apache.http.protocol.HttpContext;
import org.apache.http.util.EntityUtils;

import x.http.simple.HttpProxy;

public class HttpClientDemo {
	private CloseableHttpClient httpclient;

	public HttpClientDemo() {
		this(null);
	}

	/**
	 * 代理方式
	 * 
	 * @param proxy
	 */
	public HttpClientDemo(HttpProxy proxy) {
		if (proxy != null) {
			httpclient = getHttpClientWithProxy(proxy);
		} else {
			httpclient = HttpClients.createDefault();
		}
	}

	// 获得设置代理的httpClent实例
	private CloseableHttpClient getHttpClientWithProxy(HttpProxy proxy) {
		HttpHost proxyHost = new HttpHost(proxy.getHost(), proxy.getPort());
		CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
		credentialsProvider.setCredentials(new AuthScope(proxy.getHost(), proxy.getPort()),
				new UsernamePasswordCredentials(proxy.getUser(), proxy.getPassword()));

		return HttpClients.custom().setProxy(proxyHost).setDefaultCredentialsProvider(credentialsProvider).build();
	}

	/**
	 * 发送get请求
	 * 
	 * @param url
	 * @throws Exception
	 */
	public byte[] sendGetRequest(String url) throws Exception {
		HttpGet httpGet = new HttpGet(url);
		System.out.println("Send Get Request : " + url);
		return sendRequest(httpGet);
	}

	/**
	 * 发送post 请求
	 * 
	 * @param request
	 * @throws Exception
	 */
	public byte[] sendPostRequest(String url, Map<String, String> params) throws Exception {
		HttpPost httpPost = new HttpPost(url);
		// 设置参数
		if (params != null && params.size() > 0) {
			List<NameValuePair> nvps = new ArrayList<NameValuePair>();
			for (Entry<String, String> param : params.entrySet()) {
				nvps.add(new BasicNameValuePair(param.getKey(), param.getValue()));
			}
			httpPost.setEntity(new UrlEncodedFormEntity(nvps));
		}

		System.out.println("Send Post Request : " + url + " , " + params);
		return sendRequest(httpPost);
	}

	// 底层请求打印页面
	private byte[] sendRequest(HttpRequestBase request) throws Exception {
		CloseableHttpResponse response = httpclient.execute(request);

		// 设置超时
		setTimeOut(request, 5000);

		// 获取返回的状态列表
		StatusLine statusLine = response.getStatusLine();
		System.out.println("StatusLine : " + statusLine);

		if (statusLine.getStatusCode() == HttpStatus.SC_OK) {
			try {
				// 获取response entity
				HttpEntity entity = response.getEntity();
				System.out.println("getContentType : " + entity.getContentType().getValue());
				System.out.println("getContentEncoding : " + entity.getContentEncoding().getValue());

				// 读取响应内容
				byte[] responseBody = EntityUtils.toByteArray(entity);

				// 关闭响应流
				EntityUtils.consume(entity);
				return responseBody;
			} finally {
				response.close();
			}
		}
		return null;
	}

	// 设置超时重定向
	private void setTimeOut(HttpRequestBase request, int timeout) {
		RequestConfig requestConfig = RequestConfig.custom().setConnectionRequestTimeout(timeout)
				.setConnectTimeout(timeout).setSocketTimeout(timeout).setRedirectsEnabled(true).build();
		request.setConfig(requestConfig);
	}

	/**
	 * 设置全局Cookies
	 * 
	 * @param cookies
	 * @return
	 */
	public HttpClient setGlobeCookieHttpClient(Map<String, String> cookies) {
		HttpClient client = null;
		if (cookies != null && cookies.size() > 0) {
			BasicCookieStore cookieStore = new BasicCookieStore();

			for (Entry<String, String> c : cookies.entrySet()) {
				BasicClientCookie cookie = new BasicClientCookie(c.getKey(), c.getValue());
				cookie.setDomain("sina.com");
				cookie.setPath("/");
				cookieStore.addCookie(cookie);
			}
			client = HttpClientBuilder.create().setDefaultCookieStore(cookieStore).build();
		} else {
			client = HttpClients.createDefault();
		}
		return client;
	}

	/**
	 * 设置Cookie方法1
	 * 
	 * @param request
	 * @param cookies
	 */
	public void setRequestCookiesStr(HttpUriRequest request, Map<String, String> cookies) {
		StringBuilder cookieStr = new StringBuilder();
		if (cookies != null && cookies.size() > 0) {
			for (Entry<String, String> cookie : cookies.entrySet()) {
				cookieStr.append(cookie.getKey()).append('=').append(cookie.getValue()).append(';');
			}
			cookieStr.deleteCharAt(cookieStr.length() - 1);
			request.addHeader("Cookie", cookieStr.toString());
		}
	}

	/**
	 * 设置Cookie 方法2
	 * 
	 * httpclient.execute(request, context) 方法进行添加
	 * 
	 * @param cookies
	 * @return
	 */
	public HttpContext getetRequestCookieContext(Map<String, String> cookies) {
		HttpContext localContext = null;
		if (cookies != null && cookies.size() > 0) {
			BasicCookieStore cookieStore = new BasicCookieStore();

			for (Entry<String, String> c : cookies.entrySet()) {
				BasicClientCookie cookie = new BasicClientCookie(c.getKey(), c.getValue());
				cookie.setDomain("sina.com");
				cookie.setPath("/");
				cookieStore.addCookie(cookie);
			}
			localContext = new BasicHttpContext();
			localContext.setAttribute(HttpClientContext.COOKIE_STORE, cookieStore);
		}
		return localContext;
	}

	/**
	 * 响应中获取cookie
	 * 
	 * @param response
	 */
	public void getResponseCookies(HttpResponse response) {
		// 法1
		HeaderIterator headerIter = response.headerIterator("Set-Cookie");
		while (headerIter.hasNext()) {
			Object obj = headerIter.next();
			if (obj != null) {
				String cookie = ((Header) headerIter.next()).getValue();

				String cookieRecord = cookie.substring(0, cookie.indexOf(';'));
				System.out.println(cookieRecord);
			}

		}

		// 法2
		HeaderElementIterator headerElementIter = new BasicHeaderElementIterator(response.headerIterator("Set-Cookie"));
		while (headerElementIter.hasNext()) {
			HeaderElement elem = headerElementIter.nextElement();
			System.out.println("Element : " + elem.getName() + " = " + elem.getValue());
			NameValuePair[] params = elem.getParameters();
			for (int i = 0; i < params.length; i++) {
				System.out.println(" Params : " + params[i].getName() + " | " + params[i].getValue());
			}
		}
	}

	public void shutdown() throws Exception {
		httpclient.close();
	}

	public static void main(String[] args) throws Exception {
		String url = "http://www.google.com.hk";

		HttpClientDemo httpclient = new HttpClientDemo();

		byte[] getResponse = httpclient.sendGetRequest(url);
		System.out.println("Get Body : " + new String(getResponse, "utf-8"));

		byte[] postResponse = httpclient.sendPostRequest(url, null);
		System.out.println("Get Body : " + new String(postResponse, "utf-8"));

		httpclient.shutdown();
	}
}

 

 

分享到:
评论

相关推荐

    最全最新httpclient4.3.3

    1. 使用连接池:HttpClient支持连接池,可提高并发性能并减少资源消耗。 2. 正确处理异常:对可能出现的网络异常进行捕获和处理,避免程序意外终止。 3. 使用异步模式:HttpClient 4.3.3支持异步操作,对于大量并发...

    httpclient4.3.3

    《HttpClient 4.3.3:构建高效网络通信的核心库》 HttpClient 4.3.3 是 Apache 组织提供的一款强大的 HTTP 客户端库,它为开发者提供了丰富的功能,便于实现各种复杂的HTTP请求和处理。HttpClient 4.x 系列在功能、...

    HttpClient4.3.3及其依赖的包

    在使用HttpClient 4.3.3时,开发者需要注意以下几点: 1. **版本兼容性**:确保项目中的其他依赖与HttpClient 4.3.3兼容,避免出现版本冲突。 2. **正确配置连接池**:合理设置连接池的最大连接数和超时时间,防止...

    最新 httpclient-4.3.3

    10. **性能优化**:HttpClient 4.3.3 版本针对内存管理、线程安全和资源释放等方面进行了优化,确保在高并发场景下的稳定运行。 在实际开发中,HttpClient 4.3.3 可以与各种 Java 框架(如 Spring、Struts 等)无缝...

    httpclient-4.3.3.jar和httpcore-4.3.2.jar

    在实际项目中,HTTPClient 4.3.3和HTTPCore 4.3.2的组合使用,可以构建出高效、灵活的网络通信模块。开发者可以根据需要定制请求处理逻辑,比如自定义认证机制、错误处理策略等。同时,这两个库都遵循Apache软件许可...

    httpclient-4.3.3.jar

    httpclient-4.3.3.jar 三、使用方法 使用HttpClient发送请求、接收响应很简单,一般需要如下几步即可。 1. 创建HttpClient对象。 2. 创建请求方法的实例,并指定请求URL。如果需要发送GET请求,创建HttpGet对象;...

    httpclient-4.3.3和httpcore-4.2.1资源下载

    在使用HTTPClient和HTTPCore时,需要注意的是,这两个库都需要正确配置连接管理,以确保有效利用系统资源,避免过多的并发连接导致的问题。此外,对于安全性要求较高的应用,应考虑使用SSL/TLS加密,设置信任的证书...

    httpclient-4.5.3 api 中文版

    10. **拦截器**:HttpClient 允许使用`HttpRequestInterceptor`和`HttpResponseInterceptor`来在请求发送前或响应接收后添加自定义逻辑,这对于日志记录、请求修改或特殊处理非常有用。 HttpClient 4.5.3的文档...

    httpclient-4.3.3+jsoup-1.7.3模拟登陆csdn

    本项目涉及的是使用Apache HttpClient 4.3.3和Jsoup 1.7.3这两个开源库来实现对CSDN(China Software Developer Network)网站的模拟登录,并在成功登录后抓取用户的个人主页内容。下面将详细阐述这两个库的功能以及...

    通过HTTPClient获取网络资源

    本篇文章将深入探讨如何使用`HttpClient`来获取网络资源,以及处理返回的XML数据。 首先,我们需要了解`HttpClient`的基本用法。在Java中,创建一个`HttpClient`实例是获取网络资源的第一步。`HttpClient`提供了...

    httpClient 调用远程接口 获取数据到本地文件夹

    在这个场景中,我们使用HttpClient来调用远程接口,从电信公司的网站获取可用的新手机号码信息,并将这些数据存储到本地文件中。以下是对这个过程的详细解释: 1. **HttpClient的基本使用**: HttpClient提供了一...

    java实现httpClient样例

    GET用于获取资源,POST用于提交数据。比如: ```java HttpGet httpGet = new HttpGet("http://example.com"); HttpPost httpPost = new HttpPost("http://example.com"); ``` 3. **设置请求头**:你可以通过`...

    httpclient-4.3.1.jar,httpcore-4.3.2.jar和httpmime-4.3.5.jar

    标题中的“httpclient-4.3.1.jar”,“httpcore-4.3.2.jar”和“httpmime-4.3.5.jar”分别指的是Apache HttpClient库的不同组件和版本。这些是Java开发中用于HTTP通信的重要库,由Apache软件基金会提供。下面将详细...

    httpclient 静态化网站 project

    【标题】"httpclient 静态化网站 project"是一个基于Apache HttpClient库的项目,旨在实现网站内容的抓取和静态化。HttpClient是一个强大的HTTP客户端编程工具包,它允许开发者在Java环境中执行HTTP请求,获取网页...

    httpcore-4.3.2和httpclient-4.3.3.jar

    在4.3.3版本中,HttpClient对SSL/TLS支持进行了增强,改进了连接池管理,并修复了一些已知问题,提高了整体的稳定性和兼容性。 对于POST请求来说,HttpClient是理想的工具。POST请求常用于向服务器提交数据,例如...

    android:通过httpclient向网络端获取数据的精短小实例

    Android 中使用 HttpClient 获取网络数据的实例 Android 平台中,HttpClient 是一个常用的网络请求组件,通过它可以与远程服务器进行交互,获取所需的数据。在 Android 应用程序中,我们可以使用 HttpClient 来获取...

    httpClient调用远程接口获取数据到本地文件夹

    在这个例子中,我们使用HttpClient来调用远程接口,从电信公司的网站获取可用手机号码的数据,并将其存储到本地文件系统。以下是对这个过程的详细解释: 1. **HttpClient的引入与配置**: 首先,我们需要在项目中...

    ATL HttpClient 代码样例

    ATL自带 HTTP Client访问 HTTP Server的Server侧代码。

    HttpClient获取OAuth2.0中的code

    通过httpclient post去获取,response返回码是302,返回的code放在header的Location中。 请求的时候client_id,response_type,redirect_uri,state拼接在url后面,account和password放在body表单(x-...

    HttpClient网络获取图片

    从网络获取数据和图片,用gson解析并用listview展示

Global site tag (gtag.js) - Google Analytics