The popularity of deep reinforcement learning (DRL) applications in economics has increased exponentially. DRL, through a wide range of capabilities from reinforcement learning (RL) to deep learning (DL), offers vast opportunities for handling sophisticated dynamic economics systems. DRL is characterized by scalability with the potential to be applied to high-dimensional problems in conjunction with noisy and nonlinear patterns of economic data. In this paper, we initially consider a brief review of DL, RL, and deep RL methods in diverse applications in economics, providing an in-depth insight into the state-of-the-art. Furthermore, the architecture of DRL applied to economic applications is investigated in order to highlight the complexity, robustness, accuracy, performance, computational tasks, risk constraints, and profitability. The survey results indicate that DRL can provide better performance and higher efficiency as compared to the traditional algorithms while facing real economic problems in the presence of risk parameters and the ever-increasing uncertainties. View Full-Text
1. Introduction
Deep learning (DL) techniques are based on the use of multi-neurons that rely on the multi-layer architectures to accomplish a learning task. In DL, the neurons are linked to the input data in conjunction with a loss function for the purpose of updating their weights and maximizing the fitting to the inbound data [1][2][1,2]. In the structure of a multi-layer, every node takes the outputs of all the prior layers in order to represent outputs set by diminishing the approximation of the primary input data, while multi-neurons learn various weights for the same data at the same time. There is a great demand for the appropriate mechanisms to improve productivity and product quality in the current market development. DL enables predicting and investigating complicated market trends compared to the traditional algorithms in ML. DL presents great potential to provide powerful tools to learn from stochastic data arising from multiple sources that can efficiently extract complicated relationships and features from the given data. DL is reported as an efficient predictive tool to analyze the market [3][4][3,4]. Additionally, compared to the traditional algorithms, DL is able to prevent the over-fitting problem, to provide more efficient sample fitting associated with complicated interactions, and to outstretch input data to cover all the essential features of the relevant problem [5].
Reinforcement learning (RL) [6] is a powerful mathematical framework for experience-driven autonomous learning [7]. In RL, the agents interact directly with the environment by taking actions to enhance its efficiency by trial-and-error to optimize the cumulative reward without requiring labeled data. Policy search and value function approximation are critical tools of autonomous learning. The search policy of RL is to detect an optimal (stochastic) policy applying gradient-based or gradient-free approaches dealing with both continuous and discrete state-action settings [8]. The value function strategy is to estimate the expected return in order to find the optimal policy dealing with all possible actions based on the given state. While considering an economic problem, despite traditional approaches [9], reinforcement learning methods prevent suboptimal performance, namely, by imposing significant market constraints that lead to finding an optimal strategy in terms of market analysis and forecast [10]. Despite RL successes in recent years [11][12][13][11,12,13], these results suffer the lack of scalability and cannot manage high dimensional problems. The DRL technique, by combining both RL and DL methods, where DL is equipped with the vigorous function approximation, representation learning properties of deep neural networks (DNN), and handling complex and nonlinear patterns of economic data, can efficiently overcome these problems [14][15][14,15]. Ultimately, the purpose of this paper is to comprehensively provide an overview of the state-of-the-art in the application of both DL and DRL approaches in economics. However, in this paper, we focus on the state-of-the-art papers that employ DL, RL, and DRL methods in economics issues. The main contributions of this paper can be summarized as follows:
-
Classification of the existing DL, RL, and DRL approaches in economics.
-
Providing extensive insights into the accuracy and applicability of DL-, RL-, and DRL-based economic models.
-
Discussing the core technologies and architecture of DRL in economic technologies.
-
Proposing a general architecture of DRL in economics.
-
Presenting open issues and challenges in current deep reinforcement learning models in economics.
2. Review Section
This section discusses an overview of various interesting uses of both DL and deep RL approaches in economics.
2.1. Deep Learning Application in Economics
The recent attractive application of deep learning in a variety of economics domains is discussed in this section.
2.1.1. Deep Learning in Stock Pricing
From an economic point of view, the stock market value and its development are essential to business growth. In the current economic situation, there are many investors around the world that are interested in the stock market in order to receive quick and better return compared to other sectors. The presence of uncertainty and risk in the forecasting of stock pricing bring challenges to the researcher to design a market model for prediction. Despite all advances to develop mathematical models for forecasting, they are still not that successful [16][61]. The deep learning topic attracts scientists and practitioners as it is useful for high revenue while enhancing the prediction accuracy with DL methods. presents recent research.
Table 1. Application of deep learning in stock price prediction.
| References |
Methods |
Application |
| Reference |
Methods |
Application |
|---|
| [28] | [76] |
Augmented Lagrangian Technique |
Optimal auction design |
| [29] | [77] |
Extended RegretNet method |
Maximized return in auction |
| [30] | [78] |
Data-Driven Method |
Mechanism design in auction |
| [31] | [79] |
Multi-layer neural Network method |
Auction in mobile networks |
2.1.4. Deep Learning in Banking and Online Markets
In current technology improvement, fraud detection is a challenging application of deep learning, namely, in online shopping and credit cards. There is a high market demand to construct an efficient system for fraud detection in order to keep the involved system safe (see ).
Table 4. Application of deep learning in the banking system and online market.
| Reference |
Methods |
Application |
| [32] | [81] |
AE |
Fraud detection in unbalanced datasets |
| [33] | [82] |
Network topology |
credit card transactions |
| [34] | [83] |
Natural language Processing |
Anti-money laundering detection |
| [35] | [84] |
AE and RBM architecture |
Fraud detection in credit cards |
2.1.5. Deep Learning in Macroeconomics
Macroeconomic prediction approaches have gained much interest in recent years, which are helpful for investigating economics growth and business changes [36][85]. There are many proposed methods that can forecast macroeconomic indicators, but these approaches require huge amounts of data and suffer from model dependency. shows the recent results which are more acceptable than the previous ones.
Table 5. Application of deep learning in macroeconomics.
| [17][62] |
Two-Streamed gated recurrent unit network |
Deep learning framework for stock value prediction |
| [18][63] |
| [ |
| 20 |
). In recent years, ML techniques have been widely used to develop practical algorithms in this field due to the high market demand for new approaches compared with traditional methods to practically measure all types of risks (Brockett et al. 2002; Pathak et al. 2005, Derrig, 2002). For instance, there are many demands for car insurance that forces companies to find novel strategies in order to meliorate and upgrade their system. summarizes the most notable studies for the application of DL techniques in insurance.
Table 2. Application of deep learning in the Insurance industry.
| Reference |
Methods |
Application |
| [21][69] |
Cycling algorithms |
Fraud detection in car insurance |
| Filtering methods |
Novel filtering approach |
| [ |
| [22][70] |
LDA-based appraoch |
Insurance fraud |
19][64] |
Pattern techniques |
Pattern matching algorithm for forecasting the stock value |
| [23][71] |
Autoencoder technique |
Evaluation of risk in car insurance |
][65] |
Multilayer deep Approach |
Advanced DL framework for the stock value price |
2.1.2. Deep Learning in Insurance
Another application of DL methods is the insurance sector. One of the challenges of insurance companies is to efficiently manage fraud detection (see
2.1.3. Deep Learning in Auction Mechanisms
Auction design has a major importance in practice that allows the organizations to present better services to their customers. A great challenge to learn a trustable auction is that its bidders require optimal strategy for maximizing profit. In this direction, Myerson designed an optimal auction with only a single item [24][72]. There are many works with results for single bidders but most often with partial optimality [25][26][27][73,74,75]. presents the notable studies developed by DL techniques in Auction Mechanisms.
Table 3. Application of deep learning in auction design.
| Reference |
Methods |
Application |
| [37] | [86] |
Encoder-decoder |
Indicator prediction |
| [38] | [87] |
Backpropagation Approach |
Forecasting inflation |
| [39] | [88] |
Feed-Forward neural Network |
Asset allocation |
2.1.6. Deep Learning in Financial Markets (Service & Risk Management)
In financial markets, it is crucial to efficiently handle the risk arising from credits. Due to recent advance in big data technology, DL models can design a reliable financial model in order to forecast credit risk in banking systems (see ).
Table 67. Application of deep learning in financial markets (services and risk management).
| Reference |
Methods |
Application |
comprises the recent results of using deep learning approaches in financial investment.
comprises the recent results of using deep learning approaches in financial investment.
Table 78. Application of deep learning in stock price prediction.
| Reference |
Methods |
Application |
| [93] |
LSTM and AE |
Market investment |
| [94] |
Hyper-parameter |
Option pricing in finance |
| 117] |
SS-RTB method |
Bidding optimization in advertising |
[95] |
LSTM and SVR |
Quantitative strategy in investment |
| [68] |
DDPG and DQN |
Pricing algorithm for online market |
| [118] |
DDPG and DQN |
Pricing algorithm for online market |
[96] |
| [ |
| R-NN and genetic method |
Smart financial investment |
| 69 |
| ] |
| DQN scheme |
| Online news recommendation |
| [119] |
DQN scheme |
Online news recommendation |
| ] |
| Binary Classification Technique |
Loan pricing |
| [89] |
Binary Classification Technique |
Loan pricing |
| Reference |
Methods |
Application |
| [44] |
LSTM and AE |
Market investment |
| [45] |
Hyper-parameter |
Option pricing in finance |
| [46] |
LSTM and SVR |
Quantitative strategy in investment |
| [47] |
R-NN and genetic method |
Smart financial investment |
2.1.8. Deep Learning in Retail
New applications of DL as well as novel DRL in retail industry are emerging in a fast pace
[48][49][50][51][52][53][54][55][56][57][58][59][60][61][62][63][64][65][66][67][98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117]. Augmented reality (AR) enables customers to improve their experience while buying/finding a product from real stores. This algorithm is frequently used by researchers in the field. presents the notable studies.
Table 89. Application of deep learning in retail markets.
2.1.9. Deep Learning in Business (Intelligence)
Nowadays, big data solutions play the key role in business services and productivities to efficiently reinforce the market. To solve the complex business intelligence (BI) problems dealing with market data, DL techniques are useful (see ). presents the notable studies for the application of deep learning in business intelligence.
Table 910. Application of deep learning in business intelligence.
| Reference |
Methods |
Application |
| [52] | [102] |
MLP |
BI with client data |
| [53] | [103] |
MLS and SAE |
Feature selection in market data |
| [54] | [104] |
RNN |
Information detection in business data |
| [55] | [105] |
RNN |
Predicting procedure in business |
2.2. Deep Reinforcement Learning Application in Economics
Despite the traditional approaches, DRL has the important capability of capturing substantial market conditions to provide the best strategy in economics, which also provides the potential of scalability and efficient handling of high-dimensional problems. Thus, we are motivated to consider the recent advance of deep RL applications in economics and the financial market.
2.3. Deep Reinforcement Learning in Stock Trading
Financial companies need to detect the optimal strategy while dealing with stock trading in the dynamic and complicated environment in order to maximize their revenue. Traditional methods applied to stock market trading are quite difficult for experimentation when the practitioner wants to consider transaction costs. RL approaches are not efficient enough to find the best strategy due to the lack of scalability of the models to handle high-dimensional problems
[58][108]. presents the most notable studies developed by deep RLs in the stock market.
Table 101. Application of deep RLs in the stock market.
| Reference |
Methods |
Application |
| [59] | [109] |
DDPG |
Dynamic stock market |
| [60] | [110] |
Adaptive DDPG |
Stock portfolio strategy |
| [61] | [DQN methods |
Efficient market strategy |
| [62] | [112] |
RCNN |
Automated trading |
2.3.1. Deep Reinforcement Learning in Portfolio Management
Algorithmic trading area is currently using deep RL techniques for portfolio management with fixed allocation of capital into various financial products (see ). presents the notable studies in the application of deep reinforcement learning in portfolio management.
Table 112. Application of deep reinforcement learning in portfolio management.
| [66] |
Actor–critic method |
Recommendation architecture |
| [116] |
Actor–critic method |
Recommendation architecture |
| [67] |
SS-RTB method |
Bidding optimization in advertising |
| [ |
| References |
Methods |
Application |
| [59][63] | [109,113] |
DDPG |
Algorithmic trading |
| [64] | [114] |
Model-less CNN |
Financial portfolio algorithm |
| 111] |
| [15] |
Model-free |
Advanced strategy in portfolio trading |
| [65] | [115] |
Model-based |
Dynamic portfolio optimization |
| [ |
| 41 |
| ] |
Feature selection |
Credit risk analysis |
| [90] |
Feature selection |
Credit risk analysis |
| [42] |
AE |
Portfolio management |
| [91] |
AE |
Portfolio management |
| [43] |
Likelihood Esrtimation |
Mortgage risk |
| [92] |
Likelihood Esrtimation |
Mortgage risk |
2.1.7. Deep Learning in Investment
Financial problems generally need to be analyzed in terms of datasets from multiple sources. Thus, it is substantial to construct a reliable model for handling unusual interactions and features from the data for efficient forecasting.
| Reference |
Methods |
Application |
| [48] |
Augmented reality and image classification |
Improving shopping in retail markets |
| [98] |
Augmented reality and image classification |
Improving shopping in retail markets |
| [49] |
DNN methods |
Sale prediction |
| [99] |
DNN methods |
Sale prediction |
| [50] |
CNN |
Investigation in retail stores |
| [100] |
CNN |
Investigation in retail stores |
| [51] |
Adaptable CNN |
Validation in the food industry |
| [101] |
Adaptable CNN |
Validation in the food industry |
2.3.2. Deep Reinforcement Learning in Online Services
In current development of online services, the users face the challenge of detecting their interested items efficiently where recommendation techniques enable us to give the right solutions to this problem. Various recommendation methods are presented such as content-based collaborative filtering, factorization machines, multi-armed bandits, to name a few. These proposed approaches are mostly limited to where the users and recommender systems interact statically and focus on short-term rewards. presents the notable studies in the application of deep reinforcement learning in online services.
Table 123. Application of deep reinforcement learning in online services.
3. Conclusions
In the current fast economics and market growth, there is a high demand for the appropriate mechanisms in order to considerably enhance the productivity and quality of the product. Thus, DL can contribute to effectively forecast and detect complex market trends, as compared to the traditional ML algorithms, with the major advantage of a high-level feature extraction property and proficiency of the problem solver methods. Furthermore, reinforcement learning enables us to construct more efficient frameworks regarding the integration of the prediction problem with the portfolio structure task, considering crucial market constraints and better performance, while using deep reinforcement learning architecture and the combination of both DL and RL approaches, for RL to resolve the problem of scalability and to be applied to the high-dimensional problems as desired in real-world market settings. Several DL and deep RL approaches, such as DNN, Autoencoder, RBM, LSTM-SVR, CNN, RNN, DDPG, DQN, and a few others, were reviewed in the various application of economic and market domains, where the advanced models improved prediction to extract better information and to find the optimal strategy mostly in complicated and dynamic market conditions. This brief work represents the basic issue that all proposed approaches are mainly to fairly deal with the model complexity, robustness, accuracy, performance, computational tasks, risk constraints, and profitability. Practitioners can employ a variety of both DL and deep RL techniques, with the relevant strengths and weaknesses, that serve in economic problems to enable the machine to detect the optimal strategy associated with the market. Recent works showed that the novel techniques in DNN, and recent interaction with reinforcement learning, so-called deep RL, have the potential to considerably enhance the model performance and accuracy while handling real-world economic problems. We mention that our work indicates the recent approaches in both DL and deep RL perform better than the classical ML approaches. Significant progress can be obtained by designing more efficient novel algorithms using deep neural architectures in conjunction with reinforcement learning concepts to detect the optimal strategy, namely, optimize the profit and minimize the loss while considering the risk parameters in a highly competitive market. For future research, a survey of machine learning and deep learning in 5G, cloud computing, cellular network, and COVID-19 outbreak is suggested.
Encyclopedia
