Submitted Successfully!
To reward your contribution, here is a gift for you: A free trial for our video production service.
Thank you for your contribution! You can also upload a video entry or images related to this topic.
Version Summary Created by Modification Content Size Created at Operation
1
Ammar AlDallal
 -- 2056 2022-09-27 10:40:42 |
2 update references and layout
Amina Yu
 -4 word(s) 2052 2022-09-30 09:31:26 |

Video Upload Options

We provide professional Video Production Services to translate complex research into visually appealing presentations. Would you like to try it?
No, upload directly Yes

Confirm

Are you sure to Delete?
Yes No
Cite
If you have any further questions, please contact Encyclopedia Editorial Office.
Aldallal, A. Intrusion Detection System. Encyclopedia. Available online: https://encyclopedia.pub/entry/27663 (accessed on 22 December 2024).
Aldallal A. Intrusion Detection System. Encyclopedia. Available at: https://encyclopedia.pub/entry/27663. Accessed December 22, 2024.
Aldallal, Ammar. "Intrusion Detection System" Encyclopedia, https://encyclopedia.pub/entry/27663 (accessed December 22, 2024).
Aldallal, A. (2022, September 27). Intrusion Detection System. In Encyclopedia. https://encyclopedia.pub/entry/27663
Aldallal, Ammar. "Intrusion Detection System." Encyclopedia. Web. 27 September, 2022.
Intrusion Detection System
Edit
This entry is adapted from the peer-reviewed paper 10.3390/sym14091916

The increased adoption of cloud computing resources produces major loopholes in cloud computing for cybersecurity attacks. An intrusion detection system (IDS) is one of the vital defenses against threats and attacks to cloud computing. IDSs encounter two challenges, namely, low accuracy and a high false alarm rate. Due to these challenges, additional efforts are required by network experts to respond to abnormal traffic alerts. To improve IDS efficiency in detecting abnormal network traffic, an IDS using a recurrent neural network based on gated recurrent units (GRUs) was developed and long short-term memory (LSTM) through a computing unit to form Cu-LSTMGRU was improved. 

deep learning LSTM GRU feature selection

1. Introduction

The ability to enact cloud-based threats and attacks has enabled a high-quality strategy for cyber intruders, attackers, and hackers worldwide, meaning that they can drastically affect the quality of the cloud environment. Cloud computing is vulnerable to several types of attacks. These include data loss, data breaches, insecure interfaces and APIs, malicious insiders, unknown risk profiles, and identity theft [1]. Cloud-based threats, such as DoS/DDoS, can rapidly deactivate a victim and initiate huge income losses. Regardless of the huge presence of available traditional solutions for threat detection, there remains significant and continuous growth in threats and attacks, with an extended volume and criticality. In cybersecurity, an intruder is an entity that seeks to exploit system vulnerabilities. Intrusion can be detected using signature-based or anomaly-based techniques. Outdated signature-based intrusion detection systems cannot respond to novel attacks, whereas the anomaly-based technique, which compares user patterns against known patterns, suffers from a high false positive rate of detection. However, this can be solved using an effective classification method. In many cases, it is not viable to test the efficiency of the developed IDS on a live dataset; hence, a predefined dataset that consists of real-time network traffic is used to examine IDS performance. The most well-known dataset of this kind is the KDD CUP 99 dataset, which has been considered by many researchers [1][2][3][4]. The optimized version of it is the NSL-KDD dataset, which has been employed by [5][6][7][8][9][10][11], among others. However, these datasets are vulnerable to a few types of attacks. In addition, these two datasets suffer from a limited number of features, which makes them unreliable when it comes to testing an IDS with new and emerging security threats and strategies used by attackers and intruders. From this point of view, it is crucial to apply IDSs on more recent datasets with a bigger number of features and more types of attacks, such as CICIDS2018.
Cybersecurity issues put users’ data security and data privacy at major risk. The openness of cloud environments needs more effective and intelligent solutions to tackle emerging security threats and attacks. Outdated signature-based intrusion detection systems cannot respond to novel attacks.
Network intrusion detection systems (NIDSs) expand machine learning procedures and are being increasingly used to address the restrictions of current interpretations. In this classification, machine learning (ML)– and deep learning (DL)–driven schemes have been developed to be highly effective in the detection of evolving cyberattacks. DL in particular is a subclass of machine learning. According to [12], three main reasons are behind the superiority of DL over other approaches. The first is that its processing abilities have increased sharply. The second reason is that the computing hardware is becoming more affordable. The third reason is the breakthrough in ML research.
DL has an important influence in several applications, such as language, audio, image, video, graphical modelling, pattern recognition, speech recognition [13], energy prediction [14], diagnosis of chest radiography [15], natural language, and signal processing [12]. Among the various neural network models that have been developed, recurrent neural networks are characterized by the possibility of transmitting information between neurons in the same layer, unlike in traditional neural networks; therefore, RNNs are considered superior and are used as the basis for the proposed model. The advancements in deep learning algorithms have been applied to IDSs to improve the detection rate and lower the false alarm rate.
Cloud-based threats and attacks have enabled a high-quality strategy for cyber intruders, a number of attackers, and hackers worldwide; therefore, they can drastically affect the quality of the cloud environment. Cloud computing is vulnerable to several types of attacks. These include data loss, data breaches, insecure interfaces and APIs, malicious insiders, unknown risk profiles, and identity theft [16].
Regardless of the huge presence of available traditional solutions for threat detection, there is still a significant continuous growth in frequent threats and attacks, with an extended volume and criticality [17].
According to [16][18], the detection of evolving cyber threats to the cloud computing environment has become a huge motivation of researchers’ studies and works. On that basis, herein, it is to develop an efficient IDS system that is proficient in the detection of evolving cloud-based cyberattacks and is also an innovative state-of-the-art deep-learning-enabled architecture for the effective identification of multiclass threats in cloud computing.

2. Deep-Learning-Based Intrusion Detection System

Due to wide usage of cloud services, detecting attacks and malicious traffic has attracted researchers to develop a highly efficient mechanism for intrusions detection.
Researchers have developed and adopted various methods and techniques derived from deep learning. In recent years, different DL algorithms have been applied or combined to produce high-performance IDSs. Examples include auto-encoder-based IDS schemes [19][20][21], RBM-based IDS schemes [22][23], DBM-based IDSs [24][25], DNN-based IDS schemes [26][27], CNN-based IDS schemes [8][28], and LSTM-based IDS schemes [16][29]. Hybrid IDS schemes include the AE and CNN hybrid [30], the AE and DBN hybrid [31], the CNN and LSTM hybrid [32], the DNN and RNN hybrid [33], the AE and GAN hybrid [34], the AE and LSTM hybrid [35], the CNN and LSTM hybrid [36], and finally, the Variational Laplace Auto-encoder and DNN. hybrid [37].
In addition, these approaches are applied to well-known datasets, such as KDD CUP 99, NLS KDD, ISCX 2012, and CICIDS2017, and the most recent dataset, CICIDS2018.
Examples of researchers who adopted GRUs and LSTM in RNNs are as follows:
Xu et al. in [2] proposed a deep-learning-based IDS. Their model consists of four layers: a GRU, LSTM, multilayer perceptron, and SoftMax regression. The model was evaluated using two well-known datasets: KDD 99 and NSL-KDD. The achieved detection rate was 99.42% for the first dataset and 99.31% for the second dataset with false positive rates of 0.05% and 0.84%, respectively.
A machine-learning-based cooperative IDS is proposed in [1]. Among the building blocks of the deep neural network model is a denoising autoencoder. In the aforementioned study, the maximum number of hidden unites was 350. The results when applying this model to the manipulated KDD Cup 99 dataset showed that the model achieved a detection accuracy of 95%.
The IDS proposed in [4] extracted features from network data using a deep confidence neural network. Intrusion types were classified using the back propagation neural network as the top level. The KDD CUP′99 dataset was used to validate this model, and the results showed improvement over the traditional machine learning accuracy.
Riyaz and Ganapathy in [8] used conditional random fields and a linear-correlation-coefficient-based feature selection algorithm in their proposed IDS to classify features using a convolutional neural network. They reported a 98.88% accuracy using the KDD CUP′99 dataset.
Li et at. [11] proposed an IDS using a multiconvolutional neural network in which feature data were classified into four parts according to the correlation. The results showed that this model outperformed traditional machine learning and recent deep learning methods when using the NSL-KDD dataset.
The approach proposed in [38] combined both ML and DL, where both types of ML were applied: supervised learning (naive Bayes) and unsupervised learning (self-organizing maps). DL is represented by the CNN and used for feature extraction. The best DR achieved was 93%.
The authors of [39] used deep learning in their developed IDS architecture. This model was used to classify both partitioned and user-defined multiclasses. This system obtained an accuracy of 95% using the UNSW-NB15 dataset.
Tang et al. [5] proposed the gated recurrent unit recurrent neural network as an IDS for a software-defined network. The proposed model was evaluated using the NSL-KDD dataset with six features and produced an accuracy of 89%.
The IDS model developed in [10] started with a feature extraction stage. In the feature extraction process, the sequential forward selection (SFS) algorithm and decision tree (DT) model hybrid was applied. Then, a deep learning model based on both LSTM and GRUs was applied to identify two types of attacks, namely, remote-to-local (R2L) and user-to-root (U2R). The results were evaluated using the NSL-KDD 2010 and ISCX 2012 and showed an improved accuracy and detection rate over other DL models.
Few researchers have considered recent datasets, CICIDS2017 and CICIDS2018, in evaluating their DL-based IDSs. Among them, in [29], the authors implemented three DL models in their developed IDS, which are deep neural networks (DNNs), long short-term memory recurrent neural networks (LSTM-RNNs), and deep belief networks (DBNs). The system was examined using both NSL-KDD and CICIDS2017 datasets. The best accuracy for multiclass classification reported for DBN was 98.95% using the CICIDS2017 dataset and 98.77% using the NSL-KDD dataset.
Fernandez and Xu [40] applied a DNN to detect anomaly transactions in both ISCX IDS 2012 and CICIDS2017. The model was compared with other machine learning techniques, such as naive Bayes, a hybrid decision tree, and rule-based IDS and random forest. It outperformed these models since the true positive rate (TPR) was 99.93% when including the IP feature and 96.77% without using the IP address.
Choraś and Pawlicki [41] investigated different hyperparameters of artificial neural networks when applied to common datasets, NSL-KDD and CICIDS2017. These parameters include activation, optimizers, batch size, epochs, layers, and neurons. The best accuracy achieved was 99.9% when the parameter values were tanh, Adam, 100, 300, 1, and 25, respectively. However, the accuracy decreased drastically down to 5.64% for the other parameters, indicating that the ANN model is very sensitive to the parameter values.
The NIDS proposed by Chen et al. [42] was based on a convolutional neural network. The analysis was conducted on a featured dataset and raw traffic dataset extracted from CICIDS2017. The results outperformed both the support vector machine (which is a type of machine learning technique) and the deep belief network (DBN) (which is a type of deep learning technique). Their proposed approach achieved a high accuracy of 99.56% for the row dataset but 96.55% for the featured dataset.
Nayyar et al. [43] built an intrusion detection system that aimed to strike a balance between enhancing accuracy and prediction time. The model was built using an LSTM mechanism. Their system consists of three types of activation for the three layers: tanh for the LSTM layer, ReLU for the hidden layers, and sigmoid for the output layer. This model was examined using CICIDS2017, and the lowest obtained accuracy was 96.7%.
In their DL-IDS model, Bharati and Tamane [44] used a multilayer perceptron in the neural system for feed forward with a minimum of one layer between data. Their model was implemented using 100 neurons in a particular layer. The test accuracy achieved using the CICIDS2018 dataset was 95%.
The main objective of the IDS model of [45] was to detect DDoS while maintaining both speed of action and robustness against adversarial examples. The proposed model used the fast gradient sign method (FGSM) of a neural network to generate an adversarial example and to compute the gradients of a loss function with respect to the input data. The performance of the system was evaluated in terms of robustness score and recall, where the achieved recall was 98.2%.
Feature selection is one of the important stages when the data have high dimensionality. It reduces the model complexity, thus minimizing the computational cost. In addition, it simplifies the model debugging, thus enhancing the model interpretation of the learning results. Amjad et al. [13] applied the openSMILE toolkit to extract the low-level acoustic characteristics that are more suitable for the suggested speech recognition based on a deep neural network.
From the above literature of DL-based IDSs, several observations can be made. First, less importance is assigned to feature selection, since it helps reduce the complexity of the classification and identification of attacks in real time [1][2][4][7][39][40][42][43][44]. Second, although the false alarm rate (FAR) is one of the typical indicators of IDS efficiency [46], most similar works do not include it as a measure. On the other hand, those who considered it produce a high FAR [5][6][9][11][29][40][47], where the FAR ranges from 0.5% to 1.73%. Additionally, the conducted studies basically limit the evaluation to the accuracy and detection rate. Improving the accuracy, among other measures, and reducing the false discovery rate (FDR) result in reducing the workload of network experts and making the system practical and reliable for real-life implementation.

References

  1. Abusitta, A.; Bellaiche, M.; Dagenais, M.; Halabi, T. A deep learning approach for proactive multi-cloud cooperative intrusion detection system. Future Gener. Comput. Syst. 2019, 98, 308–318.
  2. Xu, C.; Shen, J.; Du, X.; Zhang, F. An Intrusion Detection System Using a Deep Neural Network with Gated Recurrent Units. IEEE Access 2018, 6, 48697–48707.
  3. Khan, M.A.; Ghazal, T.M.; Lee, S.-W.; Rehman, A. Data Fusion-Based Machine Learning Architecture for Intrusion Detection. Comput. Mater. Contin. 2021, 70, 3399–3413.
  4. Peng, W.; Kong, X.; Peng, G.; Li, X.; Wang, Z. Network Intrusion Detection Based on Deep Learning. In Proceedings of the 2019 International Conference on Communications, Information System and Computer Engineering (CISCE), Haikou, China, 5–7 July 2019; pp. 431–435.
  5. Tang, T.A.; Mhamdi, L.; McLernon, D.; Zaidi, S.A.R.; Ghogho, M. Deep Recurrent Neural Network for Intrusion Detection in SDN-based Networks. In Proceedings of the 2018 4th IEEE Conference on Network Softwarization and Workshops (NetSoft), Montreal, QC, Canada, 25–29 June 2018; pp. 202–206.
  6. Elsherif, A. Automatic Intrusion Detection System Using Deep Recurrent Neural Network Paradigm. J. Inf. Secur. Cybercrimes Res. 2018, 1, 21–31.
  7. Ambusaidi, M.A.; He, X.; Nanda, P.; Tan, Z. Building an intrusion detection system using a filter-based feature selection algorithm. IEEE Trans. Comput. 2016, 65, 2986–2998.
  8. Riyaz, B.; Ganapathy, S. A deep learning approach for effective intrusion detection in wireless networks using CNN. Soft Comput. 2020, 24, 17265–17278.
  9. Almiani, M.; AbuGhazleh, A.; Al-Rahayfeh, A.; Atiewi, S.; Razaque, A. Deep recurrent neural network for IoT intrusion detection system. Simul. Model. Pract. Theory 2020, 101, 102031.
  10. Le, T.-T.-H.; Kim, Y.; Kim, H. Network Intrusion Detection Based on Novel Feature Selection Model and Various Recurrent Neural Networks. Appl. Sci. 2019, 9, 1392.
  11. Li, Y.; Xu, Y.; Liu, Z.; Hou, H.; Zheng, Y.; Xin, Y.; Zhao, Y.; Cui, L. Robust detection for network intrusion of industrial IoT based on multi-CNN fusion. Measurement 2020, 154, 107450.
  12. Kim, A.; Park, M.; Lee, D.H. AI-IDS: Application of Deep Learning to Real-Time Web Intrusion Detection. IEEE Access 2020, 8, 70245–70261.
  13. Amjad, A.; Khan, L.; Chang, H.-T. Semi-Natural and Spontaneous Speech Recognition Using Deep Neural Networks with Hybrid Features Unification. Processes 2021, 9, 2286.
  14. Phyo, P.P.; Byun, Y.-C. Hybrid Ensemble Deep Learning-Based Approach for Time Series Energy Prediction. Symmetry 2021, 13, 1942.
  15. Sahlol, A.; Elaziz, M.A.; Jamal, A.T.; Damaševičius, R.; Hassan, O.F. A Novel Method for Detection of Tuberculosis in Chest Radiographs Using Artificial Ecosystem-Based Optimisation of Deep Neural Network Features. Symmetry 2020, 12, 1146.
  16. Amara, N.; Zhiqui, H.; Ali, A. Cloud Computing Security Threats and Attacks with Their Mitigation Techniques. In Proceedings of the 2017 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC), Nanjing, China, 12–14 October 2017; pp. 244–251.
  17. Maeda, S.; Kanai, A.; Tanimoto, S.; Hatashima, T.; Ohkubo, O. A Botnet Detection Method on SDN using Deep Learning. In Proceedings of the 2019 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA, 11–13 January 2019; pp. 1–6.
  18. Ashraf, N.; Ahmad, W.; Ashraf, R. A Comparative Study of Data Mining Algorithms for High Detection Rate in Intrusion Detection System. Ann. Emerg. Technol. Comput. 2018, 2, 49–57.
  19. Sadaf, K.; Sultana, J. Intrusion Detection Based on Autoencoder and Isolation Forest in Fog Computing. IEEE Access 2020, 8, 167059–167068.
  20. Louati, F.; Ktata, F.B. A deep learning-based multi-agent system for intrusion detection. SN Appl. Sci. 2020, 2, 675.
  21. Mighan, S.N.; Kahani, M. A novel scalable intrusion detection system based on deep learning. Int. J. Inf. Secur. 2020, 20, 387–403.
  22. Mayuranathan, M.; Murugan, M.; Dhanakoti, V. Best features based intrusion detection system by RBM model for detecting DDoS in cloud environment. J. Ambient. Intell. Humaniz. Comput. 2019, 2, 3609–3619.
  23. Masdari, M.; Khezri, H. Efficient VM migrations using forecasting techniques in cloud computing: A comprehensive review. Clust. Comput. 2020, 23, 2629–2658.
  24. Yang, H.; Qin, G.; Ye, L. Combined Wireless Network Intrusion Detection Model Based on Deep Learning. IEEE Access 2019, 7, 82624–82632.
  25. Wang, Z.; Zeng, Y.; Liu, Y.; Li, D. Deep Belief Network Integrating Improved Kernel-Based Extreme Learning Machine for Network Intrusion Detection. IEEE Access 2021, 9, 16062–16091.
  26. Thamilarasu, G.; Chawla, S. Towards Deep-Learning-Driven Intrusion Detection for the Internet of Things. Sensors 2019, 19, 1977.
  27. Zhang, J.; Li, F.; Zhang, H.; Li, R.; Li, Y. Intrusion detection system using deep learning for in-vehicle security. Ad Hoc Netw. 2019, 95, 101974.
  28. Hu, Z.; Wang, L.; Qi, L.; Li, Y.; Yang, W. A Novel Wireless Network Intrusion Detection Method Based on Adaptive Synthetic Sampling and an Improved Convolutional Neural Network. IEEE Access 2020, 8, 195741–195751.
  29. Elmasry, W.; Akbulut, A.; Zaim, A.H. Evolving deep learning architectures for network intrusion detection using a double PSO metaheuristic. Comput. Netw. 2020, 168, 107042.
  30. Xu, X.; Li, J.; Yang, Y.; Shen, F. Towards Effective Intrusion Detection Using Log-Cosh Conditional Variational Autoencoder. IEEE Internet Things J. 2021, 8, 6187–6196.
  31. Yang, L.; Li, J.; Yin, L.; Sun, Z.; Zhao, Y.; Li, Z. Real-Time Intrusion Detection in Wireless Network: A Deep Learning-Based Intelligent Mechanism. IEEE Access 2020, 8, 170128–170139.
  32. Zhang, G.; Wang, X.; Li, R.; Song, Y.; He, J.; Lai, J. Network Intrusion Detection Based on Conditional Wasserstein Generative Adversarial Network and Cost-Sensitive Stacked Autoencoder. IEEE Access 2020, 8, 190431–190447.
  33. Tang, T.A.; Mhamdi, L.; McLernon, D.; Zaidi, S.A.R.; Ghogho, M.; El Moussa, F. DeepIDS: Deep Learning Approach for Intrusion Detection in Software Defined Networking. Electronics 2020, 9, 1533.
  34. Hara, K.; Shiomoto, K. Intrusion Detection System using Semi-Supervised Learning with Adversarial Auto-encoder. In Proceedings of the NOMS 2020-2020 IEEE/IFIP Network Operations and Management Symposium, Budapest, Hungary, 20–24 April 2020; pp. 1–8.
  35. Zhang, Y.; Zhang, Y.; Zhang, N.; Xiao, M. A network intrusion detection method based on deep learning with higher accuracy. Procedia Comput. Sci. 2020, 174, 50–54.
  36. Zhang, C.; Costa-P’erez, X.; Patras, P. Tiki-taka: Attacking and defending deep learning-based intrusion detection systems. In Proceedings of the 2020 ACM SIGSAC Conference on Cloud Computing Security Workshop, Virtual Event, USA, 9 November 2020; pp. 27–39.
  37. Azmin, S.; Islam, A.M.A.A. Network intrusion detection system based on conditional variational Laplace AutoEncoder. In Proceedings of the 7th International Conference on Networking, Systems and Security, Dhaka, Bangladesh, 22–24 December 2020; pp. 82–88.
  38. Kumar, P.; Kumar, A.A.; Sahayakingsly, C.; Udayakumar, A. Analysis of intrusion detection in cyber attacks using DEEP learning neural networks. Peer-To-Peer Netw. Appl. 2021, 14, 2565–2584.
  39. Ashiku, L.; Dagli, C. Network Intrusion Detection System using Deep Learning. Procedia Comput. Sci. 2021, 185, 239–247.
  40. Fernandez, G.C.; Xu, S. A Case Study on using Deep Learning for Network Intrusion Detection. In Proceedings of the MILCOM 2019—2019 IEEE Military Communications Conference (MILCOM), Norfolk, VA, USA, 12–14 November 2019; pp. 1–6.
  41. Choraś, M.; Pawlicki, M. Intrusion detection approach based on optimised artificial neural network. Neurocomputing 2021, 452, 705–715.
  42. Chen, L.; Kuang, X.; Xu, A.; Suo, S.; Yang, Y. A Novel Network Intrusion Detection System Based on CNN. In Proceedings of the 2020 Eighth International Conference on Advanced Cloud and Big Data (CBD), Taiyuan, China, 5–6 December 2020; pp. 243–247.
  43. Nayyar, S.; Arora, S.; Singh, M. Recurrent Neural Network Based Intrusion Detection System. In Proceedings of the 2020 International Conference on Communication and Signal Processing (ICCSP), Chennai, India, 28–30 July 2020; pp. 0136–0140.
  44. Bharati, M.P.; Tamane, S. NIDS-Network Intrusion Detection System Based on Deep and Machine Learning Frameworks with CICIDS2018 using Cloud Computing. In Proceedings of the 2020 International Conference on Smart Innovations in Design, Environment, Management, Planning and Computing (ICSIDEMPC), Aurangabad, India, 30–31 October 2020; pp. 27–30.
  45. Meamarian, M.; Yazdani, N. A Robust, Lightweight Deep Learning Approach for Detection and Mitigation of DDoS Attacks in SDN. In Proceedings of the 2022 27th International Computer Conference, Computer Society of Iran (CSICC), Tehran, Iran, 23–24 February 2022; pp. 1–7.
  46. Pendleton, M.; Garcia-Lebron, R.; Cho, J.-H.; Xu, S. A Survey on Systems Security Metrics. ACM Comput. Surv. 2017, 49, 62.
  47. Catillo, M.; Rak, M.; Villano, U. 2L-ZED-IDS: A Two-Level Anomaly Detector for Multiple Attack Classes. In Web, Artificial Intelligence and Network Applications. WAINA 2020; Advances in Intelligent Systems and Computing; Barolli, L., Amato, F., Moscato, F., Enokido, T., Takizawa, M., Eds.; Springer: Cham, Switzerland, 2020; Volume 1150.
More
© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license; additional terms may apply. By using this site, you agree to the Terms and Conditions and Privacy Policy.
Upload a video for this entry
Information
Subjects: Computer Science, Artificial Intelligence
Contributor MDPI registered users' name will be linked to their SciProfiles pages. To register with us, please refer to https://encyclopedia.pub/register :
Ammar Aldallal
View Times: 578
Revisions: 2 times (View History)
Update Date: 30 Sep 2022
Table of Contents
    1000/1000
    Hot Most Recent
    Video Production Service
    Feedback