Skip to content

Deep learning resources

A curated list of high-quality resources that complement the implementations and documentation in this hub.

Core deep learning texts

Research papers

  • Backpropagation: Rumelhart, Hinton & Williams (1986).
  • Adam Optimizer: Kingma & Ba (2014) – https://arxiv.org/abs/1412.6980
  • Batch Normalization: Ioffe & Szegedy (2015) – https://arxiv.org/abs/1502.03167
  • Dropout: Srivastava et al. (2014) – https://jmlr.org/papers/v15/srivastava14a.html
  • Long Short-Term Memory: Hochreiter & Schmidhuber (1997) – Neural Computation 9(8), 1735–1780.
  • Learning to Forget (LSTM forget gate): Gers, Schmidhuber & Cummins (2000) – Neural Computation 12(10), 2451–2471.
  • Gated Recurrent Unit (RNN Encoder–Decoder): Cho et al. (2014) – https://arxiv.org/abs/1406.1078
  • Empirical Evaluation of Gated RNNs (GRU vs LSTM): Chung, Gulcehre, Cho & Bengio (2014) – https://arxiv.org/abs/1412.3555
  • Were RNNs All We Needed? (minGRU / minLSTM): Feng et al. (2024) – https://arxiv.org/abs/2410.01201
  • Parallelizing Linear Recurrences (log-space scan): Heinsen (2023) – https://arxiv.org/abs/2311.06281
  • Attention Is All You Need (Transformer): Vaswani et al. (2017) – https://arxiv.org/abs/1706.03762
  • Efficiently Modeling Long Sequences with Structured State Spaces (S4): Gu, Goel & Ré (2022) – https://arxiv.org/abs/2111.00396
  • RWKV: Reinventing RNNs for the Transformer Era: Peng et al. (2023) – https://arxiv.org/abs/2305.13048
  • Retentive Network (RetNet): Sun et al. (2023) – https://arxiv.org/abs/2307.08621
  • Mamba: Linear-Time Sequence Modeling with Selective State Spaces: Gu & Dao (2023) – https://arxiv.org/abs/2312.00752
  • Griffin: Mixing Gated Linear Recurrences with Local Attention: De et al. (2024) – https://arxiv.org/abs/2402.19427
  • Transformers are SSMs: State Space Duality (Mamba-2): Dao & Gu (2024) – https://arxiv.org/abs/2405.21060
  • xLSTM: Extended Long Short-Term Memory: Beck et al. (2024) – https://arxiv.org/abs/2405.04517
  • Distributional Structure (the distributional hypothesis): Harris (1954) – Word 10(2–3), 146–162.
  • A synopsis of linguistic theory 1930–1955 ("a word by the company it keeps"): Firth (1957) – in Studies in Linguistic Analysis, Blackwell.
  • A Neural Probabilistic Language Model: Bengio, Ducharme, Vincent & Jauvin (2003) – JMLR 3, 1137–1155. Learned distributed word-feature vectors.
  • Efficient Estimation of Word Representations in Vector Space (Word2Vec): Mikolov, Chen, Corrado & Dean (2013) – https://arxiv.org/abs/1301.3781
  • Linguistic Regularities in Continuous Space Word Representations (analogies): Mikolov, Yih & Zweig (2013) – NAACL-HLT, 746–751.
  • GloVe: Global Vectors for Word Representation: Pennington, Socher & Manning (2014) – https://nlp.stanford.edu/pubs/glove.pdf
  • Improving Distributional Similarity with Lessons Learned from Word Embeddings (3CosMul): Levy, Goldberg & Dagan (2015) – TACL 3, 211–225.
  • Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings (hard debiasing, identify-subspace / neutralize / equalize): Bolukbasi, Chang, Zou, Saligrama & Kalai (2016) – https://arxiv.org/abs/1607.06520
  • Semantics derived automatically from language corpora contain human-like biases (WEAT — the Word Embedding Association Test): Caliskan, Bryson & Narayanan (2017) – Science 356(6334), 183–186.
  • Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings (the "geometric debiasing is cosmetic" critique): Gonen & Goldberg (2019) – https://arxiv.org/abs/1903.03862
  • Distributed Representations of Words and Phrases and their Compositionality (negative sampling): Mikolov, Sutskever, Chen, Corrado & Dean (2013) – https://arxiv.org/abs/1310.4546
  • Neural Word Embedding as Implicit Matrix Factorization: Levy & Goldberg (2014) – NeurIPS.
  • word2vec Parameter Learning Explained: Rong (2014) – https://arxiv.org/abs/1411.2738
  • Neural Machine Translation of Rare Words with Subword Units (BPE): Sennrich, Haddow & Birch (2016) – https://arxiv.org/abs/1508.07909
  • Enriching Word Vectors with Subword Information (fastText): Bojanowski, Grave, Joulin & Mikolov (2017) – https://arxiv.org/abs/1607.04606
  • Visualizing Data using t-SNE: van der Maaten & Hinton (2008) – JMLR 9, 2579–2605.
  • Deep contextualized word representations (ELMo): Peters, Neumann, Iyyer, Gardner, Clark, Lee & Zettlemoyer (2018) – https://arxiv.org/abs/1802.05365
  • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding: Devlin, Chang, Lee & Toutanova (2019) – https://arxiv.org/abs/1810.04805
  • RoBERTa: A Robustly Optimized BERT Pretraining Approach: Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis, Zettlemoyer & Stoyanov (2019) – https://arxiv.org/abs/1907.11692
  • Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks: Reimers & Gurevych (2019) – https://arxiv.org/abs/1908.10084
  • SimCSE: Simple Contrastive Learning of Sentence Embeddings: Gao, Yao & Chen (2021) – https://arxiv.org/abs/2104.08821
  • Dense Passage Retrieval for Open-Domain Question Answering (DPR): Karpukhin et al. (2020) – https://arxiv.org/abs/2004.04906
  • ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT: Khattab & Zaharia (2020) – https://arxiv.org/abs/2004.12832
  • BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models: Thakur, Reimers, Rücklé, Srivastava & Gurevych (2021) – https://arxiv.org/abs/2104.08663
  • MTEB: Massive Text Embedding Benchmark: Muennighoff, Tazi, Magne & Reimers (2023) – https://arxiv.org/abs/2210.07316
  • Text Embeddings by Weakly-Supervised Contrastive Pre-training (E5): Wang et al. (2022) – https://arxiv.org/abs/2212.03533
  • C-Pack: Packed Resources For General Chinese Embeddings (BGE): Xiao, Liu, Zhang, Muennighoff, Lian & Nie (2024) – https://arxiv.org/abs/2309.07597
  • Matryoshka Representation Learning: Kusupati et al. (2022) – https://arxiv.org/abs/2205.13147
  • One Embedder, Any Task: Instruction-Finetuned Text Embeddings (Instructor): Su et al. (2022) – https://arxiv.org/abs/2212.09741
  • Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (RAG): Lewis et al. (2020) – https://arxiv.org/abs/2005.11401

Online courses

Tools and frameworks

Supplementary resources

Contributing resources

To recommend an additional high-quality resource, follow the instructions in the repository's contributing guide.