Improving RNA Secondary Structure Prediction Through Expanded Training Data
TL;DR
The authors created RNASSTR, a massive dataset of over 4.7 million RNA sequence-structure pairs, and found that retraining existing deep learning models on this expanded data exposed severe generalization and computational limitations in current RNA folding architectures.
Problem / question
Deep learning models for RNA secondary structure prediction fail to generalize to novel RNA folds, largely because existing training datasets (like bpRNA and ArchiveII) are small, structurally biased, and lack rigorous structural partitioning between training and testing sets, leading models to memorize sequences rather than learn underlying folding rules.
Methods
The researchers built the RNA Secondary Structure Repository (RNASSTR) by using Infernal and 4,170 Rfam covariance models to search GTDB and NCBI RefSeq genomes, yielding 4,779,435 filtered sequence-structure pairs. They strictly partitioned the data into training (90%), validation (5%), and test (5%) sets based on mutually exclusive structural families to prevent data leakage, and then retrained two deep learning models, SincFold and MXfold2, evaluating them using F1 and Matthews Correlation Coefficient (MCC) scores.
Key findings
RNASSTR provides nearly 50 times more data than the standard bpRNA-1m dataset. Retraining SincFold to convergence (15 epochs) yielded a training F1 of 0.983 but a test F1 of 0.420, which was lower than the published SincFold model's test F1 of 0.561 but higher than MFold's F1 of 0.321. Retraining MXfold2 was computationally prohibitive (268 GPU hours per epoch) and halted after 3 epochs, resulting in a poor test F1 of 0.171. While the retrained SincFold model had a lower average F1 score, it outperformed the published model on specific families, such as correctly predicting all canonical base pairs in a CRISPR RNA direct repeat where the published model failed.
Why it matters
By providing a rigorously partitioned dataset that is orders of magnitude larger than previous benchmarks, this work reveals that simply adding more data does not fix the generalization failures of current RNA models, highlighting the need for new, more computationally efficient architectures.
Limitations
The paper notes that current models are too computationally expensive to train on large datasets (e.g., MXfold2 taking 268 GPU hours per epoch), that the retrained models failed to improve overall average accuracy, and that sequence and secondary structure data alone may be insufficient for true generalization without incorporating experimental chemical probing data.
Takeaway
Expanding RNA training data to 4.7 million sequences reveals that current deep learning models for RNA structure still struggle with generalization and computational scaling, establishing RNASSTR as a rigorous new benchmark for future algorithm development.