manually download dataset and import in tensorflow

Posted 2023-02-14 seamanj

somtimes, you may have problem with tensorflow build-in downloading function like using

tfds.load(name="the300w_lp", split="train", with_info=True)

This could be different reason. One way to workaround is to download the dataset manually as mentioned in https://www.tensorflow.org/datasets/overview#manual_download_if_download_fails

The official doc is quite general. It took me hours to figure out how to build/import the downloaded dataset in tensorflow. At this time, the tensorflow I use is 2.6.0.

here, I want to use 300W-LP dataset in my project.

First, I manually download the dataset from the official project space, the downloaded file is named as 300W-LP.zip
(if you don’t know the official dataset download address, then you can go to the corresponding source code to find out, Here, when I open /home/seamanj/Software/miniconda3/envs/tf2/lib/python3.8/site-packages/tensorflow_datasets/image/the300w_lp.py I could find

_DATASET_URL = "https://drive.google.com/uc?export=download&id=0B7OEHD3T4eCkVGs0TkhUWFN6N1k"

_PROJECT_URL = "http://www.cbsr.ia.ac.cn/users/xiangyuzhu/projects/3DDFA/main.htm"

try both in your browser.

)

Second and most important step, you need to find out where you should put the downloaded file or point the related link to your location.

To do that, you need to set a breakpoint in download_manager.py (mine location is /home/seamanj/Software/miniconda3/envs/tf2/lib/python3.8/site-packages/tensorflow_datasets/core/download/download_manager.py) to find out what manual_path is

def _get_manually_downloaded_path(
    manual_dir: Optional[ReadOnlyPath],
    expected_url_info: Optional[checksums.UrlInfo],
) -> Optional[ReadOnlyPath]:
  """Checks if file is already downloaded in manual_dir."""
  if not manual_dir:  # Manual dir not passed
    return None

  if not expected_url_info or not expected_url_info.filename:
    return None  # Filename unknown.

  manual_path = manual_dir / expected_url_info.filename
  if not manual_path.exists():  # File not manually downloaded
    return None

here, my manual_path is /home/seamanj/tensorflow_datasets/downloads/manual/300W-LP.zip, then just put the downloaded file to this location.

Next, you need to prepare the dataset including extracting and creating tfrecords, this is done by builder’s download_and_prepare function, the complete code is as follows:

import tensorflow_datasets as tfds
from tensorflow_datasets import *


builder = tfds.image.The300wLp() # reference https://www.tensorflow.org/datasets/overview#manual_download_if_download_fails to construct the builder
download_config = download.DownloadConfig(manual_dir="~/tensorflow_datasets/downloads/manual",
                                          try_download_gcs=False,
                                          )

builder.download_and_prepare(download_dir="~/tensorflow_datasets", download_config=download_config)

note that you could skip the download_config unless you want to specify a different dir for your download dataset, the default manual_dir is ~/tensorflow_datasets/downloads/manual on my ubuntu 20.04

after this step, you will find the downloaded dataset is precossed


now, you can use it as normally you do

dataset, info = tfds.load(name="the300w_lp", split="train", with_info=True)