Were RNNs All We Needed?
Interesting work on reviving RNNs. arxiv.org/abs/2410.01201 -- in general the fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't fundamentally important in the curve-fitting paradigm (aka deep learning)
Curve-fitting is about embedding a dataset on a curve. The critical factor is the dataset, not the specific hard-coded bells and whistles that constrain the curve's shape. As long as your curve is sufficiently expressive all architectures will converge to the same performance in the large-data regime.
#مقاله #ایده_جذاب
🔸 مطالب بیشتر 👇👇
✅ @AI_DeepMind
🔸 @AI_Person
Interesting work on reviving RNNs. arxiv.org/abs/2410.01201 -- in general the fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't fundamentally important in the curve-fitting paradigm (aka deep learning)
Curve-fitting is about embedding a dataset on a curve. The critical factor is the dataset, not the specific hard-coded bells and whistles that constrain the curve's shape. As long as your curve is sufficiently expressive all architectures will converge to the same performance in the large-data regime.
#مقاله #ایده_جذاب
🔸 مطالب بیشتر 👇👇
✅ @AI_DeepMind
🔸 @AI_Person