Deep Learning

AlexNet's victory in the ImageNet competition officially ushered in the era of deep learning.

Deep learning achieved a breakthrough performance in the 2012 ImageNet competition, driving the revival of connectionism

Frontier models exhibit emergent capabilities, meaning they suddenly display abilities not explicitly included in the training objective once a certain parameter-scale threshold is reached.

The massively parallel computing architecture of GPUs is naturally suited for matrix operations in deep learning, and their thousands of computing cores can accelerate the training process by tens to hundreds of times.

The poor scalability of statistical modeling methods on high-dimensional data is the core reason for the gradual rise of deep learning methods.

Deep learning is a subfield of machine learning whose core is multi-layer artificial neural networks, with ideas traceable as far back as the 1940s.

Backpropagation is essentially a dynamic programming algorithm for efficiently computing gradients, which reduces the computational complexity of gradient calculation by propagating error signals layer by layer from the output layer to the input layer and reusing intermediate computation results.

AI coding tools are showing an evolutionary trend from "rapid generation" toward "deliberate reasoning."

Deep learning is the underlying support technology for current mainstream AI capabilities such as image recognition, natural language processing, and speech synthesis.

Artificial intelligence, machine learning, deep learning, and large language models are in a nested relationship: artificial intelligence is the broadest, machine learning is a subset of it, deep learning is a branch of machine learning, and large language models are an application direction of deep learning.

If samples are too easy the model has nothing to learn; if they are too difficult, noisy learning signals may interfere with training; the most effective learning occurs in the region slightly above current capability but still within reach.

Low-level features such as edges and textures learned by the shallow layers of deep learning models exhibit strong domain-agnostic properties, while high-level semantic features encoded by deep layers are more domain-specific.

Gradient descent is the most fundamental optimization algorithm in deep learning, and its mathematical essence is iteratively updating parameters in the direction of the negative gradient of the loss function.

Compared to deep learning, gradient boosting is generally more stable, more interpretable, and faster to train on small-to-medium-sized datasets.

ReLU, defined as max(0, x), rapidly gained widespread adoption in the 2010s due to its computational efficiency and gradient stability, becoming the default activation function in modern deep learning.

The GPU's parallel computing architecture originally stemmed from the demands of game graphics rendering, and its computation mode is similar to the tensor batch operations in deep learning.

It is recommended to first master classical machine learning principles before entering deep learning; otherwise, one risks becoming a practitioner who only knows how to call pre-built packages.

The energy consumption during the training phase of large models is approximately hundreds of times that of the inference phase.

Deep learning training has inherent randomness, and retraining with the same configuration does not necessarily yield the same results; sources of randomness include the random seed for weight initialization, the shuffling order of data batches, random noise introduced by Dropout, and non-deterministic accumulated floating-point errors in distributed training.

There is a progressive containment relationship among artificial intelligence, machine learning, and deep learning: artificial intelligence > machine learning > deep learning.