Android Netroid解析之——断点续传下载及问题修正
Posted gccbuaa
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Android Netroid解析之——断点续传下载及问题修正相关的知识，希望对你有一定的参考价值。
提到Netroid也许非常多人不知道这个框架，但我假设说Volley想必没有人不知道吧。
Netroid是一个基于Volley实现的android Http库。提供运行网络请求、缓存返回结果、批量图片载入、大文件断点下载的常见Http交互功能，关于网络请求，图片载入没什么好说的，Volley已经有非常多人解析过了，这里来说一下大文件断点下载。
关于大文件断点下载，网上也有非常多实现的demo,为什么要单单说Netroid呢？由于Netroid断点续传不依赖数据库，我在网上看到过非常多的断点续传的样例，无一例外都是依赖于数据库。包含DownloadManager，大名鼎鼎的xutils,可是这两个都有一定的问题。
1.DownloadManager在三星手机上必须打开下载管理才干应用，而打开这个管理必须须要手动打开，普通情况下无伤大雅。视情况而定
2.xutils这个框架别的不知道。文件下载这块慎用
好了。进入正题，Netroid的地址:https://github.com/vince-styling/,以下简单的说一下这个框架文件下载的实现和原理,
// 1
RequestQueue queue = Netroid.newRequestQueue(getApplicationContext(), null);
// 2
mDownloder = new FileDownloader(queue, 1) {
@Override
public FileDownloadRequest buildRequest(String storeFilePath, String url) {
return new FileDownloadRequest(storeFilePath, url) {
@Override
public void prepare() {
addHeader("Accept-Encoding", "identity");
super.prepare();
}
};
}
};
// 3
task.controller = mDownloder.add(mSaveDirPath + task.storeFileName, task.url, new Listener<Void>() {
@Override
public void onPreExecute() {
task.invalidate();
}
@Override
public void onSuccess(Void response) {
showToast(task.storeFileName + " Success!");
}
@Override
public void onError(NetroidError error) {
NetroidLog.e(error.getMessage());
}
@Override
public void onFinish() {
NetroidLog.e("onFinish size : " + Formatter.formatFileSize(
FileDownloadActivity.this, new File(mSaveDirPath + task.storeFileName).length()));
task.invalidate();
}
@Override
public void onProgressChange(long fileSize, long downloadedSize) {
task.onProgressChange(fileSize, downloadedSize);
// NetroidLog.e("---- fileSize : " + fileSize + " downloadedSize : " + downloadedSize);
}
});
实现的话非常easy，主要分为三步就能够了
1.创建一个请求队列
2.构建一个文件下载管理器
3.将下载任务加入到队列
如今依据上面的三步来看一下它的实现原理：
第一步：创建一个请求队列:RequestQueue queue = Netroid.newRequestQueue(getApplicationContext(), null);
/**
* Creates a default instance of the worker pool and calls {@link RequestQueue#start()} on it.
* @param context A {@link Context} to use for creating the cache dir.
* @return A started {@link RequestQueue} instance.
*/
public static RequestQueue newRequestQueue(Context context, DiskCache cache) {
int poolSize = RequestQueue.DEFAULT_NETWORK_THREAD_POOL_SIZE;
HttpStack stack;
String userAgent = "netroid/0";
try {
String packageName = context.getPackageName();
PackageInfo info = context.getPackageManager().getPackageInfo(packageName, 0);
userAgent = packageName + "/" + info.versionCode;
} catch (NameNotFoundException e) {
}
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.GINGERBREAD) {
stack = new HurlStack(userAgent, null);
} else {
// Prior to Gingerbread, HttpUrlConnection was unreliable.
// See: http://android-developers.blogspot.com/2011/09/androids-http-clients.html
stack = new HttpClientStack(userAgent);
}
//实例化BasicNetwork，主要用于运行下载请求
Network network = new BasicNetwork(stack, HTTP.UTF_8);
//创建请求队列
RequestQueue queue = new RequestQueue(network, poolSize, cache);
//非常重要的一步
queue.start();
return queue;
}com.duowan.mobile.netroid.RequestQueue.start():
/**
* Starts the dispatchers in this queue.
*/
public void start() {
stop(); // Make sure any currently running dispatchers are stopped.
// Create the cache dispatcher and start it.
mCacheDispatcher = new CacheDispatcher(mCacheQueue, mNetworkQueue, mCache, mDelivery);
mCacheDispatcher.start();
// Create network dispatchers (and corresponding threads) up to the pool size.
for (int i = 0; i < mDispatchers.length; i++) {
//一个线程，从请求队列中获取任务并运行
NetworkDispatcher networkDispatcher =
new NetworkDispatcher(mNetworkQueue, mNetwork, mCache, mDelivery);
mDispatchers[i] = networkDispatcher;
//Thread run()
networkDispatcher.start();
}
}
/**
* Stops the cache and network dispatchers.
*/
public void stop() {
if (mCacheDispatcher != null) {
mCacheDispatcher.quit();
}
for (NetworkDispatcher mDispatcher : mDispatchers) {
//Thread interrupt()线程中断
if (mDispatcher != null) mDispatcher.quit();
}
}框架中对于文件是没有缓存机制的。所以mCacheDispatcher能够不用理它。看一下NetworkDispatcher这个线程做了什么：com.duowan.mobile.netroid.NetworkDispatcher
public class NetworkDispatcher extends Thread {
@Override
public void run() {
//设置线程优先级
Process.setThreadPriority(Process.THREAD_PRIORITY_BACKGROUND);
Request request;
while (true) {
try {
// Take a request from the queue.假设队列为空，则堵塞
request = mQueue.take();
} catch (InterruptedException e) {
// We may have been interrupted because it was time to quit.唯有线程中断的时候mQuit才为true,InterruptedException为中断异常
//mQueue.take()假设队列为null,仅仅会堵塞，不会跑出异常
if (mQuit) return;
continue;
}
try {
request.addMarker("network-queue-take");
//准备运行
mDelivery.postPreExecute(request);
// If the request was cancelled already,
// do not perform the network request.
if (request.isCanceled()) {
request.finish("network-discard-cancelled");
mDelivery.postCancel(request);
mDelivery.postFinish(request);
continue;
}
// Perform the network request.最重要一步。Netroid实例化的BasicNetwork在这里运行网络请求
NetworkResponse networkResponse = mNetwork.performRequest(request);
request.addMarker("network-http-complete");
// Parse the response here on the worker thread.重命名一下。没做什么
Response<?> response = request.parseNetworkResponse(networkResponse);
request.addMarker("network-parse-complete");
// Write to cache if applicable.
if (mCache != null && request.shouldCache() && response.cacheEntry != null) {
response.cacheEntry.expireTime = request.getCacheExpireTime();
mCache.putEntry(request.getCacheKey(), response.cacheEntry);
request.addMarker("network-cache-written");
}
// Post the response back.
request.markDelivered();
mDelivery.postResponse(request, response);
} catch (NetroidError netroidError) {
mDelivery.postError(request, request.parseNetworkError(netroidError));
} catch (Exception e) {
NetroidLog.e(e, "Unhandled exception %s", e.toString());
mDelivery.postError(request, new NetroidError(e));
}
}
}
}这里最重要的一步就是NetworkResponse networkResponse = mNetwork.performRequest(request);运行网络请求，可是我们不要忘记我们的mQueue还是空的，mQueue.take()正在堵塞着呢，所以，如今还没有办法进行网络请求，因此我们须要在mQueue中填充任务，才干进行我们的网络请求。
不要忘记这里哦。由于我们还会回到这里！
第二步：创建一个文件下载管理器：new FileDownloader(queue, 1)
mDownloder = new FileDownloader(queue, 1) {
@Override
public FileDownloadRequest buildRequest(String storeFilePath, String url) {
return new FileDownloadRequest(storeFilePath, url) {
@Override
public void prepare() {
addHeader("Accept-Encoding", "identity");
super.prepare();
}
};
}
};
这里有没有看着非常吓人，我起初看的时候也吓了一跳，事实上就是实例化的时候，顺手override了一下
/** The parallel task count, recommend less than 3. */
private final int mParallelTaskCount;
/** The linked Task Queue. */
private final LinkedList<DownloadController> mTaskQueue;
/**
* Construct Downloader and init the Task Queue.
* @param queue The RequestQueue for dispatching Download task.
* @param parallelTaskCount
* Allows parallel task count,
* don't forget the value must less than ThreadPoolSize of the RequestQueue.
*/
public FileDownloader(RequestQueue queue, int parallelTaskCount) {
if (parallelTaskCount >= queue.getThreadPoolSize()) {
throw new IllegalArgumentException("parallelTaskCount[" + parallelTaskCount
+ "] must less than threadPoolSize[" + queue.getThreadPoolSize() + "] of the RequestQueue.");
}
mTaskQueue = new LinkedList<DownloadController>();
mParallelTaskCount = parallelTaskCount;
mRequestQueue = queue;
}
这里是须要注意的一点，mParallelTaskCount并发的数量最好<3.
第三步：将下载任务加入到队列，task.controller = mDownloder.add(mSaveDirPath + task.storeFileName, task.url, new Listener<Void>()：
/**
* Create a new download request, this request might not run immediately because the parallel task limitation,
* you can check the status by the {@link DownloadController} which you got after invoke this method.
*
* Note: don't perform this method twice or more with same parameters, because we didn't check for
* duplicate tasks, it rely on developer done.
*
* Note: this method should invoke in the main thread.
*
* @param storeFilePath Once download successed, we'll find it by the store file path.
* @param url The download url.
* @param listener The event callback by status;
* @return The task controller allows pause or resume or discard operation.
*/
public DownloadController add(String storeFilePath, String url, Listener<Void> listener) {
// only fulfill requests that were initiated from the main thread.(reason for the Delivery?)
//看名字就知道
throwIfNotOnMainThread();
//创建一个下载控制器
DownloadController controller = new DownloadController(storeFilePath, url, listener);
synchronized (mTaskQueue) {
//这可不是mQueue,这里仅仅是一个DownloadController的LinkedList集合
mTaskQueue.add(controller);
}
//重点来了
schedule();
return controller;
} /**
* Traverse the Task Queue, count the running task then deploy more if it can be.
*/
private void schedule() {
// make sure only one thread can manipulate the Task Queue.
synchronized (mTaskQueue) {
// counting ran task.
int parallelTaskCount = 0;
for (DownloadController controller : mTaskQueue) {
//累计队列中正在下载的的任务数
if (controller.isDownloading()) parallelTaskCount++;
}
//当正在下载的个数大于并行任务数的时候，不在运行下载任务
/*
* 这里举个样例说明一下：我们默认mParallelTaskCount=1
* 当我们加入第一个任务的时候。这个的controller.isDownloading()肯定是false
* 所以parallelTaskCount >= mParallelTaskCount是不成立的，当我们再加入一个任务的时候，如今mTaskQueue.size是2了
* 且第一个isDownloading，为了保证并发数量为1，会return,说的有点乱。不知道说明确了没有
*/
if (parallelTaskCount >= mParallelTaskCount) return;
// try to deploy all Task if they're await.
for (DownloadController controller : mTaskQueue) {
//deploy()，将任务加入到队列中
if (controller.deploy() && ++parallelTaskCount == mParallelTaskCount) return;
}
}
} /**
* For the parallel reason, only the {@link FileDownloader#schedule()} can call this method.
* @return true if deploy is successed.
*/
private boolean deploy() {
if (mStatus != STATUS_WAITING) return false;
//第二步我说非常吓人那个地方
mRequest = buildRequest(mStoreFilePath, mUrl);
// we create a Listener to wrapping that Listener which developer specified,
// for the onFinish(), onSuccess(), onError() won't call when request was cancel reason.
mRequest.setListener(new Listener<Void>() {
boolean isCanceled;
@Override
public void onPreExecute() {
mListener.onPreExecute();
}
@Override
public void onFinish() {
// we don't inform FINISH when it was cancel.
if (!isCanceled) {
mStatus = STATUS_PAUSE;
mListener.onFinish();
// when request was FINISH, remove the task and re-schedule Task Queue.
// remove(DownloadController.this);
}
}
@Override
public void onSuccess(Void response) {
// we don't inform SUCCESS when it was cancel.
if (!isCanceled) {
mListener.onSuccess(response);
mStatus = STATUS_SUCCESS;
remove(DownloadController.this);
}
}
@Override
public void onError(NetroidError error) {
// we don't inform ERROR when it was cancel.
if (!isCanceled) mListener.onError(error);
}
@Override
public void onCancel() {
mListener.onCancel();
isCanceled = true;
}
@Override
public void onProgressChange(long fileSize, long downloadedSize) {
mListener.onProgressChange(fileSize, downloadedSize);
}
});
mStatus = STATUS_DOWNLOADING;
//我擦，最终把任务加到队列中了
mRequestQueue.add(mRequest);
return true;
} mRequestQueue.add(mRequest);任务加到队列中了，都到了这里了看一下怎么加的吧
public Request add(Request request) {
// Tag the request as belonging to this queue and add it to the set of current requests.
request.setRequestQueue(this);
synchronized (mCurrentRequests) {
mCurrentRequests.add(request);
}
// Process requests in the order they are added.
request.setSequence(getSequenceNumber());
request.addMarker("add-to-queue");
// If the request is uncacheable or forceUpdate, skip the cache queue and go straight to the network.
if (request.isForceUpdate() || !request.shouldCache()) {
mDelivery.postNetworking(request);
mNetworkQueue.add(request);
return request;
}
}request.shouldCache()有兴趣的能够自己去看一下。这里说明了文件下载没有缓存机制，这里就不多说了，由于假设你还没有忘记的话。mQueue.take()还在堵塞着呢。好了让我们回到第一步，运行网络请求
NetworkResponse networkResponse = mNetwork.performRequest(request);
@Override
public NetworkResponse performRequest(Request<?> request) throws NetroidError {
// Determine if request had non-http perform.
NetworkResponse networkResponse = request.perform();
if (networkResponse != null) return networkResponse;
long requestStart = SystemClock.elapsedRealtime();
while (true) {
// If the request was cancelled already,
// do not perform the network request.
if (request.isCanceled()) {
request.finish("perform-discard-cancelled");
mDelivery.postCancel(request);
throw new NetworkError(networkResponse);
}
HttpResponse httpResponse = null;
byte[] responseContents = null;
try {
// prepare to perform this request, normally is reset the request headers.
request.prepare();
httpResponse = mHttpStack.performRequest(request);
StatusLine statusLine = httpResponse.getStatusLine();
int statusCode = statusLine.getStatusCode();
responseContents = request.handleResponse(httpResponse, mDelivery);
if (statusCode < 200 || statusCode > 299) throw new IOException();
// if the request is slow, log it.
long requestLifetime = SystemClock.elapsedRealtime() - requestStart;
logSlowRequests(requestLifetime, request, responseContents, statusLine);
return new NetworkResponse(statusCode, responseContents, parseCharset(httpResponse));
} catch (SocketTimeoutException e) {
attemptRetryOnException("socket", request, new TimeoutError());
} catch (ConnectTimeoutException e) {
attemptRetryOnException("connection", request, new TimeoutError());
} catch (MalformedURLException e) {
throw new RuntimeException("Bad URL " + request.getUrl(), e);
} catch (IOException e) {
if (httpResponse == null) throw new NoConnectionError(e);
int statusCode = httpResponse.getStatusLine().getStatusCode();
NetroidLog.e("Unexpected response code %d for %s", statusCode, request.getUrl());
if (responseContents != null) {
networkResponse = new NetworkResponse(statusCode, responseContents, parseCharset(httpResponse));
if (statusCode == HttpStatus.SC_UNAUTHORIZED || statusCode == HttpStatus.SC_FORBIDDEN) {
attemptRetryOnException("auth", request, new AuthFailureError(networkResponse));
} else {
// TODO: Only throw ServerError for 5xx status codes.
throw new ServerError(networkResponse);
}
} else {
throw new NetworkError(networkResponse);
}
}
}
}这里我给改了一下，详细的能够看一下作者的。他有一块dead code,网络请求这一块没什么好说的。可是这里有一句非常重要的代码
responseContents = request.handleResponse(httpResponse, mDelivery);。写文件,断点续传的原理
/**
* In this method, we got the Content-Length, with the TemporaryFile length,
* we can calculate the actually size of the whole file, if TemporaryFile not exists,
* we'll take the store file length then compare to actually size, and if equals,
* we consider this download was already done.
* We used {@link RandomAccessFile} to continue download, when download success,
* the TemporaryFile will be rename to StoreFile.
*/
@Override
public byte[] handleResponse(HttpResponse response, Delivery delivery) throws IOException, ServerError {
// Content-Length might be negative when use HttpURLConnection because it default header Accept-Encoding is gzip,
// we can force set the Accept-Encoding as identity in prepare() method to slove this problem but also disable gzip response.
HttpEntity entity = response.getEntity();
//获取文件的总大小
long fileSize = entity.getContentLength();
if (fileSize <= 0) {
NetroidLog.d("Response doesn't present Content-Length!");
}
long downloadedSize = mTemporaryFile.length();
/*
* 是否支持断点续传
*
* client每次提交下载请求时。服务端都要加入这两个响应头，以保证client和服务端将此下载识别为能够断点续传的下载：
* Accept-Ranges：告知下载client这是一个能够恢复续传的下载，存放本次下载的開始字节位置、文件的字节大小；
* ETag：保存文件的唯一标识（我在用的文件名称+文件最后改动时间，以便续传请求时对文件进行验证）。
* Last-Modified：可选响应头，存放服务端文件的最后改动时间，用于验证
*/
boolean isSupportRange = HttpUtils.isSupportRange(response);
if (isSupportRange) {
fileSize += downloadedSize;
// Verify the Content-Range Header, to ensure temporary file is part of the whole file.
// Sometime, temporary file length add response content-length might greater than actual file length,
// in this situation, we consider the temporary file is invalid, then throw an exception.
String realRangeValue = HttpUtils.getHeader(response, "Content-Range");
// response Content-Range may be null when "Range=bytes=0-"
if (!TextUtils.isEmpty(realRangeValue)) {
String assumeRangeValue = "bytes " + downloadedSize + "-" + (fileSize - 1);
if (TextUtils.indexOf(realRangeValue, assumeRangeValue) == -1) {
throw new IllegalStateException(
"The Content-Range Header is invalid Assume[" + assumeRangeValue + "] vs Real[" + realRangeValue + "], " +
"please remove the temporary file [" + mTemporaryFile + "].");
}
}
}
// Compare the store file size(after download successes have) to server-side Content-Length.
// temporary file will rename to store file after download success, so we compare the
// Content-Length to ensure this request already download or not.
if (fileSize > 0 && mStoreFile.length() == fileSize) {
// Rename the store file to temporary file, mock the download success. ^_^
mStoreFile.renameTo(mTemporaryFile);
// Deliver download progress.
delivery.postDownloadProgress(this, fileSize, fileSize);
return null;
}
//之所以能够实现断点续传的原因所在
RandomAccessFile tmpFileRaf = new RandomAccessFile(mTemporaryFile, "rw");
// If server-side support range download, we seek to last point of the temporary file.
if (isSupportRange) {
//移动文件读写指针位置
tmpFileRaf.seek(downloadedSize);
} else {
// If not, truncate the temporary file then start download from beginning.
tmpFileRaf.setLength(0);
downloadedSize = 0;
}
try {
InputStream in = entity.getContent();
// Determine the response gzip encoding, support for HttpClientStack download.
if (HttpUtils.isGzipContent(response) && !(in instanceof GZIPInputStream)) {
in = new GZIPInputStream(in);
}
byte[] buffer = new byte[6 * 1024]; // 6K buffer
int offset;
while ((offset = in.read(buffer)) != -1) {
//写文件
tmpFileRaf.write(buffer, 0, offset);
downloadedSize += offset;
long currTime = SystemClock.uptimeMillis();
//控制下载进度的速度
if (currTime - lastUpdateTime >= DEFAULT_TIME) {
lastUpdateTime = currTime;
delivery.postDownloadProgress(this, fileSize,
downloadedSize);
}
if (isCanceled()) {
delivery.postCancel(this);
break;
}
}
} finally {
try {
// Close the InputStream and release the resources by "consuming the content".
if (entity != null) entity.consumeContent();
} catch (Exception e) {
// This can happen if there was an exception above that left the entity in
// an invalid state.
NetroidLog.v("Error occured when calling consumingContent");
}
tmpFileRaf.close();
}
return null;
}
实现断点续传主要靠的RandomAccessFile,你假设对c语言不陌生的话tmpFileRaf.seek(downloadedSize)和int fseek(FILE *stream, long
offset, int fromwhere);是不是有点眼熟，仅仅与RandomAccessFile就不说了。
好了，Netroid的原理基本上就是这些了，讲一下我用的时候遇到的两个问题：
1.下载进度的速度太快。你假设用notifition来显示，会出现ANR,所以我们要控制一下它的速度，详细方法在上面
//控制下载进度的速度
if (currTime - lastUpdateTime >= DEFAULT_TIME) {
lastUpdateTime = currTime;
delivery.postDownloadProgress(this, fileSize,
downloadedSize);
}2.第二个问题是当你下载的时候。假设把WiFi关掉。即使没下完。也会被标记为done,改动主要是在在FileDownloader.DownloadController的deploy()中
@Override
public void onFinish() {
// we don't inform FINISH when it was cancel.
if (!isCanceled) {
mStatus = STATUS_PAUSE;
mListener.onFinish();
// when request was FINISH, remove the task and re-schedule Task Queue.
// remove(DownloadController.this);
}
}
@Override
public void onSuccess(Void response) {
// we don't inform SUCCESS when it was cancel.
if (!isCanceled) {
mListener.onSuccess(response);
mStatus = STATUS_SUCCESS;
remove(DownloadController.this);
}
}
把onFinish的status改成STATUS_PAUSE。并去掉remove(DownloadController.this);。在onSuccess中再将status改动为STATUS_SUCCESS，并remove，当然这个办法治标不治本。假设有谁知道请告之，谢谢。
以上是关于Android Netroid解析之——断点续传下载及问题修正的主要内容，如果未能解决你的问题，请参考以下文章
Android应用开发之使用Socket进行大文件断点上传续传