Get verified datasets for speech and accents from $0.59/clip
Back to Documentation

Dataset Management

Organize, validate, and maintain production-ready voice datasets with consistent metadata and governance controls.

12 min readUpdated Mar 28, 2025

1. Recommended Dataset Structure

Use a stable folder and manifest pattern so downstream teams can process data without custom transforms for every delivery.

  • Keep speaker IDs pseudonymous and stable.
  • Store transcripts and audio references using deterministic naming.
  • Version manifests when records are corrected or replaced.

Suggested structure

dataset/
  audio/
    speaker_001_clip_001.wav
    speaker_001_clip_002.wav
  transcripts/
    speaker_001_clip_001.txt
  metadata/
    manifest.jsonl
  consent/
    consent_summary.csv

2. Metadata Standards That Matter

Metadata quality directly affects training reliability and auditability.

Checklist

  • - Language and locale tags are normalized (for example, en-US, en-GB)
  • - Accent label taxonomy is documented and consistently applied
  • - Consent status is attached to each record batch
  • - Recording conditions include device/channel context where available
  • - Quality fields include pass/fail reason for rejected samples

3. Versioning and Change Control

Treat datasets like production assets. Every correction or augmentation should generate a new version reference and changelog entry.

Do not silently overwrite records used in active model experiments. Instead, publish additive updates and mark deprecated entries for removal windows.

1. Create immutable release tags

Use clear version labels such as v1.0, v1.1, v2.0 for traceability.

2. Publish change notes

List what changed, why it changed, and whether retraining is recommended.

3. Track downstream impact

Record which model experiments consumed each dataset version for reproducibility.

4. Secure Delivery and Retention

  • Use access-scoped download links and expiration windows.
  • Encrypt stored data and access logs for sensitive workflows.
  • Define retention rules for raw audio, derived artifacts, and backups.
  • Document deletion and revocation workflows for compliance response.

Frequently Asked Questions

How often should we refresh dataset versions?

Most teams align refresh cycles to model release cadence, usually monthly or quarterly depending on product velocity.

Should we merge all datasets into one giant manifest?

Use separate manifests per release plus a curated index. This preserves traceability without blocking cross-pack search and analysis.