Deep Learning
=================

Tutorial
--------------

- `CS230: Deep Learning, Stanford | Fall 2018 <https://cs230.stanford.edu/>`_
    - `Cheatsheets <https://stanford.edu/~shervine/l/ja/teaching/cs-230/>`_
- `DS-GA 1008: Deep Learning, NYU | Spring 2020 <https://atcold.github.io/pytorch-Deep-Learning/>`_
- `MIT 6.S191: Introduction to Deep Learning, MIT | 2020 <http://introtodeeplearning.com/>`_
- `CS 182: Deep Learning, UC Berkeley | Spring 2021 <https://cs182sp21.github.io/>`_

- `CS294-158-SP20: Deep Unsupervised Learning, UC Berkeley | Spring 2020 <https://sites.google.com/view/berkeley-cs294-158-sp20/home>`_

- `CS330: Deep Multi-Task and Meta Learning, Stanford | Fall 2019 <https://cs330.stanford.edu/>`_
- `MIT 6.S192: Deep Learning for Art, Aesthetics, and Creativity, MIT | 2021 <https://ali-design.github.io/deepcreativity/>`_


GitHub
--------------

- `Zhang, A., Lipton, Z., Li, M., & Smola, A. (2021). Dive into Deep Learning. arXiv preprint arXiv:2106.11342. <https://d2l.ai/index.html>`_
- `Chollet, F. (2017). Deep Learning with Python. Manning. <https://github.com/fchollet/deep-learning-with-python-notebooks>`_
- `Foster, D. (2019). Generative Deep Learning: Teaching Machines to Paint, Write, Compose, and Play. O'Reilly Media. <https://github.com/davidADSP/GDL_code>`_
- `Patel, A. (2019). Hands-On Unsupervised Learning Using Python: How to Build Applied Machine Learning Solutions from Unlabeled Data. O'Reilly Media. <https://github.com/aapatel09/handson-unsupervised-learning>`_
- https://github.com/bharathgs/Awesome-pytorch-list
- https://github.com/rasbt/deeplearning-models

GAN
^^^^^^^^

- https://github.com/eriklindernoren/Keras-GAN
- https://github.com/robbiebarrat/art-DCGAN


Book
--------------

- `Ian Goodfellow, Yoshua Bengio, & Aaron Courville (2016). Deep Learning. MIT Press. <https://www.deeplearningbook.org/>`_

Web Page
--------------

Troubleshooting and Debugging
^^^^^^^^

- `A Recipe for Training Neural Networks - Andrej Karpathy <http://karpathy.github.io/2019/04/25/recipe/>`_
- `Troubleshooting Deep Neural Networks - Josh Tobin <http://josh-tobin.com/troubleshooting-deep-neural-networks.html>`_
- `37 Reasons why your Neural Network is not working - Slav Ivanov <https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607>`_
- `CS231n: Convolutional Neural Networks for Visual Recognition <https://cs231n.github.io/neural-networks-3/>`_


Hyperparameters
^^^^^^^^

- `hypeparameters tuning neural network according to loss vs according to scoring function - Stack Exchange <https://datascience.stackexchange.com/questions/42758/hypeparameters-tuning-neural-network-according-to-loss-vs-according-to-scoring-f>`_

Optimizer
^^^^^^^^
- `AdamW and Super-convergence is now the fastest way to train neural nets - fast.ai <https://www.fast.ai/2018/07/02/adam-weight-decay/>`_
- `Optimizerはどれが優れているか（ON EMPIRICAL COMPARISONS OF OPTIMIZERS FOR DEEP LEARNINGの紹介） - Akihiro FUJII <https://akichan-f.medium.com/optimizer%E3%81%AF%E3%81%A9%E3%82%8C%E3%81%8C%E5%84%AA%E3%82%8C%E3%81%A6%E3%81%84%E3%82%8B%E3%81%8B-on-empirical-comparisons-of-optimizers-for-deep-learning%E3%81%AE%E7%B4%B9%E4%BB%8B-f843179e8a8d>`_

Batch Normalization
^^^^^^^^

- `Ordering of batch normalization and dropout? - stackoverflow <https://stackoverflow.com/a/62806906/15383706>`_
    | 順番はBN → Dropoutにする（Dropout → BNだと学習と推論でBNに入力する値の分散が変わってしまい性能が悪化する）


MLP (Multilayer Perceptron)
^^^^^^^^

- `How to Configure the Number of Layers and Nodes in a Neural Network - Jason Brownlee <https://machinelearningmastery.com/how-to-configure-the-number-of-layers-and-nodes-in-a-neural-network/>`_
- `How to choose the number of hidden layers and nodes in a feedforward neural network? - Cross Validated <https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw/1097#1097>`_

    | "One hidden layer is sufficient for the large majority of problems."
    | "the optimal size of the hidden layer is usually between the size of the input and size of the output layers."
    | "The number of hidden neurons should be 2/3 the size of the input layer, plus the size of the output layer."
    | "The number of hidden neurons should be less than twice the size of the input layer."
    | "the number of neurons in that layer is the mean of the neurons in the input and output layers."

- `multi-layer perceptron (MLP) architecture: criteria for choosing number of hidden layers and size of the hidden layer? - Stack Overflow <https://stackoverflow.com/questions/10565868/multi-layer-perceptron-mlp-architecture-criteria-for-choosing-number-of-hidde/10568938#10568938>`_
- `Number of nodes in hidden layers of neural network - Cross Validated <https://stats.stackexchange.com/questions/160887/number-of-nodes-in-hidden-layers-of-neural-network/160901#160901>`_


PyTorch
^^^^^^^^

- `PERFORMANCE TUNING GUIDE - PyTorch <https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html>`_
- `PyTorchでの学習・推論を高速化するコツ集 - 小川雄太郎 <https://qiita.com/sugulu_Ogawa_ISID/items/62f5f7adee083d96a587>`_