Currently, 5G wireless networks are fully commercialized worldwide and B5G networks are in the process of development and are supposed to be deployed within the next few years. With the fast adoption of 5G technology, the number of users who utilize wireless mobile networks has increased exponentially over the last few years. It is expected that this rapid growth will continue to increase enormously due to the deployment of more smart applications and new enabling technologies within the upcoming B5G networks. Future wireless networks will not only provide much higher spectral efficiency/data rates and lower latency but will also provide new services and technologies that can be applied in various vertical industries [1]. The presence of ubiquitous connections and the growing number of devices that are connected to the internet are expected to cause a challenge for efficient and reliable resource management systems [2,3,4]. Moreover, the rapid growth of the massive number of new devices that are connected to the internet and the Internet of Things (IoT) is expected to cause a serious risk to network security if not manipulated properly [5]. Therefore, when considering all of these capabilities, it can be seen that there is a need for robust security mechanisms across all segments of 5G and B5G networks.

Conventionally, high-layer cryptography-based techniques have been widely adopted to deal with any discrepancies that are associated with information confidentiality, which include data authentication, secret key establishment and secret dissemination [6]. However, with the advancement in the computing capacities of eavesdropping devices, the above-mentioned techniques may not be sufficient or may even become unsuitable when an extra secure channel is required for secret key generation [7]. Recently, physical layer security (PLS) has emerged as a promising means of addressing the eavesdropping computational capability of secure transmission problems [8,9,10,11,12]. Compared to complex cryptography techniques, PLS does not depend on the computational capacity of the eavesdropping devices and, therefore, it has the benefit of reducing computational costs and resource consumption. From the perspective of information-theoretic fundamentals, it has been found that PLS can achieve secure and reliable communication even when network intruders have very strong computing capabilities [13]. This approach to information security is especially effective since it does not rely on underlying computational capabilities, but rather on the characteristics of the transmission media, such as noise, fading and interference, and it provides security guarantees that are independent from the computing power of the eavesdropper. In general, the PLS approach presents distinct advantages and is well suited for distributed processing systems and dynamic network configurations. Therefore, the PLS approach can be used as an alternative supplement for computationally demanding high-layer technologies to further ensure data security.

Although the PLS can be precisely evaluated using popular performance metrics, such as secrecy capacity, secrecy rate, secrecy throughput, etc., which are discussed in detail in the literature, security performance is quantified by maximizing the performance difference between legitimate channels and wiretap channels [14,15,16,17]. This is intuitive since PLS aims to enhance the received signal quality at the intended receiver or reduce the performance of the wiretap channel relative to the legitimate channel. In this circumstance, there is a need to allocate transmission power based on the states of the legitimate and eavesdropper channels in order to improve the PLS, as transmission power affects the signal quality at the intended receiver and eavesdropper. However, the allocation of power in PLS is a challenging task. It relies heavily on the prior information that the transmitter has on the channel state information (CSI) of the intended receiver and the eavesdropper. Most of the optimization functions in PLS are non-convex because of the characteristics of the logarithmic subtraction in security performance metrics. For instance, when the transmission power increases, the capacity and reliability of the main channel improve [13]. On the other hand, the capacity of the eavesdropper channel may also improve and the probability of eavesdropping increases. Therefore, there is no universal approach to achieving a global optimization for non-convex power allocation. Several research works have been conducted to formulate and solve these optimization problems in order to obtain stronger security [18]. In [19], instead of maximizing the secrecy capacity of the main channel, suboptimal power allocation was presented to minimize the SINR at an unintended receiver. However, the minimization of the SINR at an unintended receiver is not assessed by direct performance metrics and the security measure cannot ensure a non-negative rate of transmission. Moreover, a joint subcarrier and power allocation mechanism were proposed in [20] for maximizing the secrecy capacity of OFDMA-based wireless networks. Nevertheless, the performance of secrecy gain can be enhanced by limited power allocation. Consequently, it is hard to achieve global quality of service (QoS) constraints for secure transmission systems.

The mainstream studies on PLS as a method for characterizing an achievable security performance against eavesdropping have been extensively investigated from the fundamental viewpoints of information theory for different communication scenarios and channel types and under different assumptions on the knowledge of CSI. These studies have inspired the development of many signal processing design techniques [21,22,23,24]. In this context, a large number of research works have been conducted, which have contributed insightful thoughts and opportunities to the understanding of practical security designs, optimization techniques, technology status, etc. For example, in [25,26,27], key technologies, technical challenges, and countermeasures were reviewed from the fundamental viewpoints of design strategies that involve physical-layer authentication, secret key generation, wiretap coding, and multi-antenna techniques, and relay cooperation. Moreover, the authors in [28,29] presented an extensive investigation of multi-antenna techniques in multi-user wireless networks using different assumptions on the availability of CSI. Providing security for multi-antenna techniques is an effective and powerful approach in PLS that can offer higher spatial degrees of freedom. The survey paper in [30] also provided a comprehensive overview of secure transmission designs from the viewpoints of information theory and optimization problems using security performance metrics. Furthermore, a comprehensive overview of fundamental classification and applications of existing PLS techniques was presented by [31]. On the other hand, the challenges that face PLS were reviewed in [32]. It can be seen that the hurdles facing PLS are issued from different assumptions regarding the characteristics of wireless channels and eavesdropper models.

Naturally, the concept of optimization techniques in PLS plays a pivotal role in practical security design and thus, has received considerable attention from the research community. In this review paper, due to the importance of secure transmission design in most practical scenarios, we were motivated to conduct a systematic overview of this research direction. It has to be noted that these studies have been extensively investigated and have been published in many PLS research works. Nevertheless, we outline a summary of some of the interesting studies in Table 1. In contrast to the aforementioned works, our review paper provides a brief overview of recent results and technical challenges for the system design and optimization techniques for 5G wireless networks. The main focus of this review paper is the existing techniques and design strategies for PLS optimization, optimization problems, and the solutions that are related to wireless PLS. Moreover, it inclusively discusses the applications of several enabling and computing technologies that could improve the corresponding research challenges. In order to address the limitations of existing optimization challenges, ML and AI technologies need to be efficiently integrated into 5G networks in order to produce better security and resource management. The use of ML and AI technologies within the field of mobile communication infrastructure has made significant progress in ensuring security, reliability, and resource allocations in a dynamic, robust and trustworthy way [33,34,35,36].