Cache a Parquet Export of a Hugging Face Dataset

Downloads and caches the Parquet files for a dataset hosted on the Hugging Face Hub, using the auto-generated refs/convert/parquet branch. This branch contains a Parquet conversion of the dataset that Hugging Face maintains automatically, which avoids the need to load the dataset via its original (often script-based) loading format.

Usage

cache_parquet(
  repo_id,
  config,
  split = NULL,
  revision = "refs%2Fconvert%2Fparquet",
  ...
)

Arguments

repo_id: Character string. The repository ID of the dataset on the Hugging Face Hub, in the form "user_id/repo_name" (e.g. "fancyzhx/amazon_polarity").
config: Character string. The dataset configuration name. Many datasets expose only one configuration, in which case "default" can usually be used. If unsure, inspect the repository's refs/convert/parquet branch on the Hub to see the available configuration folders.
split: Character string. The dataset split to cache, such as "train", "test", or "validation". If NULL (default), all available splits for the given configuration are cached.
revision: Character string. A specific commit hash to pin the download to, instead of using the latest revision of the refs/convert/parquet branch. Defaults to NULL, which resolves to the most recent commit.
...: Additional arguments, reserved for future use.

Value

A named list with one entry per cached split. Each entry is a character vector of local file paths to the cached Parquet shard(s) for that split (a single path if the split has one shard, multiple paths if it was sharded across several files).

Details

Files are downloaded from the refs/convert/parquet branch maintained by Hugging Face, not from the dataset's default branch. Large datasets may be split across multiple Parquet shards; all matching shards for the requested split are downloaded and cached.

Examples

if (FALSE) { # interactive()
cache_parquet(
  repo_id = "scikit-learn/iris",
  config  = "default"
)

cache_parquet(
  repo_id = "fancyzhx/amazon_polarity",
  config  = "amazon_polarity",
  split   = "train"
)
}