Chapter 1 Introduction 1.1 Optimization Introduction 1.1.1 What does Engineering do? 1.1.2 Optimization Factors 1.2 Machine Learning Introduction 1.2.1 What is Machine Learning? 1.2.2 Mathematics Comparisons 1.2.3 Correlations between Optimization and Machine Learning 1.3 Application of AI in Real-life 1.3.1 Low-Level Image Processing 1.3.2 Radar Signal Processing 1.3.3 Target Classification 1.3.4 Data Generation 1.3.5 Scene Reconstruction and Understanding 1.3.6 Multi-Modal Sensors Data Fusion 1.4 Summary 1.5 Exercises Chapter 2 Optimality Theory 2.1 Basic Principles 2.2 Mathematical Preliminary 2.2.1 Convex Function 2.2.2 Convexity 2.2.3 Norm 2.2.4 Gradient 2.3 Optimization Modeling 2.3.1 Cost, Loss, and Profit Functions 2.3.2 Constraints 2.4 Convex Optimization Problems 2.4.1 Optimization Problem 2.4.2 Standard Convex Problems 2.5 Summary 2.6 Exercises Chapter 3 Unconstrained Optimization 3.1 Overview 3.1.1 Mathematical Formulation and Geometric Interpretation 3.1.2 Typical Optimization Examples in Machine Learning 3.1.3 Theoretical Foundations: Convexity and Conditioning 3.1.4 Numerical Solution Pipelines 3.1.5 Optimization Strategies: Line Search and Trust Region 3.2 Solution Algorithms 3.2.1 Line-Search Algorithm 3.2.2 Gradient Descent Algorithm 3.2.3 Linear Regression and Solutions 3.2.4 Comparisons between LSE and GD 3.3 Application Instance: Image Super-Resolution 3.3.1 Problem Statement 3.3.2 Solution Pipelines 3.3.3 External Rewards 3.4 Summary 3.5 Exercises
Chapter 4 Constrained Optimization 4.1 Constraints and Regularization 4.1.1 Constraints 4.1.2 Regularization 4.1.3 Comparisons and Summary 4.2 Lagrangian Multiplier Method 4.2.1 Equality Constraint 4.2.2 Inequality Constraint 4.2.3 Karush-Kuhn-Tucker Conditions 4.2.4 Augmented Lagrangian Method 4.3 ADMM Algorithm 4.3.1 The Concept of ADMM Algorithm 4.3.2 Expression and Solution of the ADMM Algorithm 4.3.3 Variants of the ADMM Algorithm 4.4 Summary 4.5 Exercises Chapter 5 Optimization and Machine Learning 5.1 End-to-end Network Optimization 5.1.1 Black Box Modeling 5.1.2 Network Parameters Optimization Algorithms 5.1.3 Understanding the Network Based Optimization 5.2 Deep-unfolding Optimization 5.2.1 Statement 5.2.2 Algorithm Unrolling 5.2.3 Summary 5.3 Learning-aided Optimization 5.3.1 Statement 5.3.2 Learning-aided Algorithm 5.3.3 Combination 5.4 Summary 5.5 Exercises Chapter 6 Application of Neural Networks in Optimization 6.1 Application of Neural Networks in Sequence Modeling 6.1.1 Recurrent Neural Network (RNN) 6.1.2 Transformer 6.2 Vision Transformer (ViT) 6.2.1 Core Design: Migration Breakthrough from NLP to Vision 6.2.2 Applications of Vision Transformer Beyond Classification 6.2.3 Summary 6.3 Applications of Vision Transformers 6.3.1 Self-Supervised Representation Learning 6.3.2 Object Detection 6.3.3 Image Segmentation 6.3.4 3D Plane Reconstruction 6.3.5 Vision Language Models(VLM) 6.4 Summary 6.5 Exercises Chapter 7 Advanced Applications of NNs 7.1 Learning from a Single Image 7.1.1 Introduction to Deep Internal Learning
7.1.2 Deep Image Prior 7.1.3 Double-DIP 7.1.4 Advanced Single-Image Applications 7.1.5 Conclusion 7.2 Learning from a Single Scene 7.2.1 Introduction 7.2.2 Neural Radiance Field 7.2.3 3D Gaussian Splatting 7.2.4 Applications of Randiance-field Representations 7.3 Summary 7.4 Exercises Chapter 8 Generative Models 8.1 Introduction to Generative Models 8.1.1 Discriminative and Generative Models 8.1.2 Objectives and Approaches in Generative Modeling 8.1.3 Why Generative Models Matter 8.1.4 A Taxonomy of Generative Model 8.1.5 Recent Advances in Generative Models 8.1.6 Key Challenges and Future Directions 8.1.7 PixelRNN and PixelCNN: Autoregressive Image Models 8.2 Variational Autoencoder 8.2.1 From Autoencoders to the Need for Generative Models 8.2.2 The Probabilistic Modeling Framework of VAEs 8.2.3 Variational Inference and the Evidence Lower Bound 8.2.4 The Reparameterization Trick 8.2.5 Optimizing VAEs 8.3 Generative Adversarial Networks (GAN) 8.3.1 Problem Statement: Distribution Matching for Generation 8.3.2 Adversarial Modeling: Generator and Discriminator 8.3.3 Value Function 8.3.4 Training GANs 8.3.5 Summary 8.4 Diffusion Model 8.4.1 Background of Generative Modeling and the Diffusion Paradigm 8.4.2 The Forward Diffusion Process 8.4.3 The Reverse Denoising Process 8.4.4 Training Objective and Noise Prediction Parameterization 8.4.5 Sampling Algorithms and the Generative Process 8.4.6 Network Architecture and Implementation Considerations 8.4.7 Conclusion 8.5 Summary 8.6 Exercises
[