Optimization Algorithms in Deep Learning Models for Improving the Forecasting Accuracy in Sequential Datasets with Application in the South African Stock Market Index: A Review

Abstract

In this paper, we review different popular optimization algorithms for machine learning models, we then evaluate the model performance and convergence rates for each optimizer using a multilayer fully connected neural network. Using a sequential dataset of index returns (time-series data) spanning over of 20-years, we demonstrate Adam and RMSprop optimizers can efficiently solve practical deep learning problems dealing with sequential datasets. We use the same parameter initialization when comparing different optimization algorithms. The hyperparameters, such as learning rate and momentum, are searched over a dense grid and the results are reported using the best hyperparameter setting.