GenVarLoader: an accelerated dataloader for applying deep learning to personalized genomics
TL;DR
GenVarLoader is a new dataloader that creates personalized genomic sequences on-the-fly, reading variants up to 26,000 times faster and reducing storage by 2,000-fold compared to existing methods.
Problem / question
Training and applying deep learning sequence models to personalized genome sequences is severely bottlenecked by the massive computational and storage costs of reading and storing these large datasets.
Methods
The authors developed GenVarLoader, a software tool that stores personalized genomic data in machine learning-optimized formats and generates personalized genomic sequences and functional tracks on-the-fly during model training or inference.
Key findings
GenVarLoader reads genetic variants up to 26,000 times faster than using a pre-subset BCF file. It loads personalized genomic data up to 130 times faster than optimized FASTA and pyBigWig pipelines. Additionally, it reduces the required data storage footprint by approximately 2,000-fold compared to existing alternative approaches.
Why it matters
By removing severe data loading and storage bottlenecks, this tool enables the scalable integration of biobank-scale personalized genomic data into deep learning applications.
Limitations
The provided text does not specify any limitations, biases, or constraints of the tool.
Takeaway
GenVarLoader drastically accelerates deep learning on personalized genomes by generating sequences on-the-fly, achieving up to 130x faster data loading and a 2,000-fold reduction in storage.