下载文件时内存不足

Posted

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了下载文件时内存不足相关的知识,希望对你有一定的参考价值。

我想在android下载一个CSV文件。下载后,我使用Jackson进行映射,然后继续使用sqlite3 db。我面临的问题是当我拨打下载文件的电话时,我得到了一个OutOfMemoryError。文件大小在12MB附近。当我在浏览器中尝试相同时,我能够下载该文件,因此我不认为它与端点有关。

要下载,我正在使用AsyncTask。

class DownloadCSVTask extends AsyncTask<Void, String, Void> {

    private final Call<ResponseBody> csvDump;
    private final Context context;

    public DownloadCSVTask(Call<ResponseBody> csvDump, Context context) {
        this.csvDump = csvDump;
        this.context = context;
    }

    @Override
    protected Void doInBackground(Void... voids) {
        try {
            Response<ResponseBody> response = csvDump.execute();
            File fileFromResponse = createFileFromResponse(response);
            mapFileContentsToModelAndPersist(fileFromResponse);
        } catch (IOException e) {
            Log.e(getClass().getSimpleName(), "CSV pull failed");
        }
        return null;
    }

    private File createFileFromResponse(Response<ResponseBody> response) throws IOException {
        File path = context.getFilesDir();
        File csvFile = File.createTempFile(TEMP_PREFIX, CSV_EXTENSION, path);
        FileOutputStream fileOutputStream = context.openFileOutput(csvFile.getName(), Context.MODE_PRIVATE);
        fileOutputStream.write(response.body().bytes());
        Log.i(getClass().getCanonicalName(), "Writing file completed" + csvFile.getAbsolutePath());
        fileOutputStream.close();
        return csvFile;
    }

    private void mapFileContentsToModelAndPersist(File fileFromResponse) {
        try {
            MappingIterator<City> cityIter = new CsvMapper().readerWithTypedSchemaFor(City.class).readValues(fileFromResponse);
            List<City> cities = cityIter.readAll();
            Stream.of(cities).forEach(City::persist);
        } catch (IOException e) {
            Log.e(getClass().getSimpleName(), "Failed to map contents of file to City Class");
        }
    }
}

我得到的错误日志:

 java.lang.RuntimeException: An error occured while executing doInBackground()
                         E      at android.os.AsyncTask$3.done(AsyncTask.java:300)
                         E      at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:355)
                         E      at java.util.concurrent.FutureTask.setException(FutureTask.java:222)
                         E      at java.util.concurrent.FutureTask.run(FutureTask.java:242)
                         E      at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:231)
                         E      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
                         E      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
                         E      at java.lang.Thread.run(Thread.java:841)
                         E  Caused by: java.lang.OutOfMemoryError
                         E      at java.lang.String.<init>(String.java:354)
                         E      at java.lang.String.<init>(String.java:393)
                         E      at okio.Buffer.readString(Buffer.java:616)
                         E      at okio.Buffer.readString(Buffer.java:599)
                         E      at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:263)
                         E      at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                         E      at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
                         E      at com.someclass.somepath.network.util.CustomHeadersInterceptor.intercept(CustomHeadersInterceptor.java:42)
                         E      at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                         E      at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
                         E      at com.someclass.somepath.network.util.NetworkInterceptor.intercept(NetworkInterceptor.java:23)
                         E      at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                         E      at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
                         E      at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170)
                         E      at okhttp3.RealCall.execute(RealCall.java:60)
                         E      at retrofit2.OkHttpCall.execute(OkHttpCall.java:174)
                         E      at retrofit2.ExecutorCallAdapterFactory$ExecutorCallbackCall.execute(ExecutorCallAdapterFactory.java:89)
                         E      at com.someclass.somepath.service.DownloadCSVTask.doInBackground(CsvPullHandler.java:60)
                         E      at com.someclass.somepath.service.DownloadCSVTask.doInBackground(CsvPullHandler.java:47)
                         E      at android.os.AsyncTask$2.call(AsyncTask.java:288)
                         E      at java.util.concurrent.FutureTask.run(FutureTask.java:237)

我在这里做错了吗?还有另一种方法吗?

答案

Android上Java代码的内存量限制为16 Mb(因设备而异)。因此,您需要将此部分移动到本机NDK代码,其中malloc()可以与设备一样多的内存,或者将文件直接保存到磁盘,然后处理它而不将整个文件加载到Java缓冲区中。在带有16Kb字节数组的循环中使用URLConnection.getInputStream()URLConnection.read(),而不是将整个文件复制到response数组中。或者,您可以使用BufferedReader.readLine()直接从HTTP流中逐行读取,并逐行填充数据库。

以上是关于下载文件时内存不足的主要内容,如果未能解决你的问题,请参考以下文章

Android下载文件内存不足的问题

Azure SDK for php blob 下载导致内存不足

将视频文件推入矢量时矢量内存不足?

安卓手机下载软件总是提示内存不足,怎么办?

使用 perl 提取 tar.gz 文件时内存不足

使用PHPExcel读取大文件(20M左右)时,报内存不足。