Java多线程下载文件
优化：合理利用服务器资源，将资源利用最大化，加快下载速度
一般有两种方式：
- 线程池里面有N个线程，多线程下载单个文件，将网络路径的文件流切割成多快，每个线程下载一小部分，然后写入到文件里面，组成一个文件
- 当有很多个文件需要下载的时候，调用某个方法，有个线程池，线程池大小假定是10，当有10个文件过来的时候，每个线程去下载一个文件即可
多线程如果只知道Thread和Runnable，那你就Out了，Java1.5以后有个concurrent包，就是Java并发编程。
N个线程下载单个文件
定义一个单个线程下载的部分
public class DownloadWithRange implements Runnable {
private String urlLocation;
private String filePath;
private long start;
private long end;
DownloadWithRange(String urlLocation, String filePath, long start, long end) {
this.urlLocation = urlLocation;
this.filePath = filePath;
this.start = start;
this.end = end;
}
@Override
public void run() {
try {
HttpURLConnection conn = getHttp();
conn.setRequestProperty("Range", "bytes=" + start + "-" + end);
File file = new File(filePath);
RandomAccessFile out = null;
if (file != null) {
out = new RandomAccessFile(file, "rw");
}
out.seek(start);
InputStream in = conn.getInputStream();
byte[] b = new byte[1024];
int len = 0;
while ((len = in.read(b)) >= 0) {
out.write(b, 0, len);
}
in.close();
out.close();
} catch (Exception e) {
e.getMessage();
}
}
public HttpURLConnection getHttp() throws IOException {
URL url = null;
if (urlLocation != null) {
url = new URL(urlLocation);
}
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(5000);
conn.setRequestMethod("GET");
return conn;
}
}
定义线程Pool
public class DownloadFileWithThreadPool {
public void getFileWithThreadPool(String urlLocation, String filePath, int poolLength) throws IOException {
ExecutorService threadPool = Executors.newCachedThreadPool();
long len = getContentLength(urlLocation);
System.out.println(len);
for (int i = 0; i < poolLength; i++) {
long start = i * len / poolLength;
long end = (i + 1) * len / poolLength - 1;
if (i == poolLength - 1) {
end = len;
}
System.out.println(start+"---------------"+end);
DownloadWithRange download = new DownloadWithRange(urlLocation, filePath, start, end);
threadPool.execute(download);
}
threadPool.shutdown();
}
public static long getContentLength(String urlLocation) throws IOException {
URL url = null;
if (urlLocation != null) {
url = new URL(urlLocation);
}
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(5000);
conn.setRequestMethod("GET");
long len = conn.getContentLength();
return len;
}
}
测试下载单个文件
public static void main(String[] args) {
Date startDate = new Date();
DownloadFileWithThreadPool pool = new DownloadFileWithThreadPool();
try {
pool.getFileWithThreadPool("http://mpge.5nd.com/2016/2016-11-15/74847/1.mp3", "D:\\1.mp3", 100);
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(new Date().getTime() - startDate.getTime());
}
测试发现，我目前的网速，下载一个3.6M左右mp3音频文件，使用多线程，下载大概需要800ms左右，而不使用多线程，则需要1600ms以上，有时候甚至3000ms
多文件下载，单文件单线程
/**
* @author Nick 带线程池的文件下载类，线程大小10
* 文件数少的情况下，体现不大
* @version V1.0.0
* @Date 2017/8/2 20:43
*/
public class FileDownConnManager {
private static final Logger logger = LoggerFactory.getLogger(FileDownConnManager.class);
private static final FileDownConnManager connManager = new FileDownConnManager();
private static ExecutorService executorService = Executors.newFixedThreadPool(10); //10个线程跑
public static FileDownConnManager getDefaultManager() {
return connManager;
}
public static byte[] fileDown(final String netURL) throws ExecutionException, InterruptedException {
Future<byte[]> future = executorService.submit(new Callable<byte[]>() {
@Override
public byte[] call() throws Exception {
Date date = new Date();
URL url;
byte[] getData = new byte[0];
InputStream is = null;
try {
url = new URL(netURL);
URLConnection connection = url.openConnection();
is = connection.getInputStream();
getData = readInputStream(is);
} catch (IOException e) {
logger.error("从URL获得字节流数组失败 " + ExceptionUtils.getMessage(e));
} finally {
try {
is.close();
} catch (IOException e) {
logger.error("从URL获得字节流数组流释放失败 " + ExceptionUtils.getMessage(e));
}
}
return getData;
}
});
return future.get();
}
}
测试
@Test
public void test2() throws ExecutionException, InterruptedException, IOException {
long time1 = System.currentTimeMillis();
for(int i = 0; i < 15; i++) {
byte[] by1 = FileDownConnManager.fileDown("http://mpge.5nd.com/2016/2016-11-15/74847/1.mp3");
FileUtils.writeByteArrayToFile(new File("D:\\test2_"+i+".mp3"), by1);
}
System.out.println(System.currentTimeMillis() - time1);
}
@Test
public void test3() throws IOException {
long time1 = System.currentTimeMillis();
for(int i = 0; i < 15; i++) {
byte[] by1 = FileFromUrlUtil.getInputStreamFromUrl("http://mpge.5nd.com/2016/2016-11-15/74847/1.mp3");
FileUtils.writeByteArrayToFile(new File("D:\\test3_"+i+".mp3"), by1);
}
System.out.println(System.currentTimeMillis() - time1);
}
结论：同样的文件，测试发现，在一定数量(文件数量大概15)的文件下载的时候，每个文件是3.6M的情况下，使用多线程大概时间是30xxxms，而不实用多线程大概是38xxxms，还是有一定时间上的提升
此处的测试还是会存在一定的误差，因为ExecutorService线程池的创建是需要时间，类的初始化也是需要一定的时间，假如在Web应用中，第一次创建出来了，第二次不需要创建，优势还是稍微能够体现出来！！
