Sequence-Based Recommendation System: History
View Latest Version
Please note this is an old version of this entry, which may differ significantly from the current revision.
Contributor:
Xulin Ma
,
Jiajia Tan
,
Linan Zhu
,
Xiaoran Yan
,
Xiangjie Kong

Sequence-based models have various applications in recommendation systems; these models recommend the interested items of the user according to the user’s behavioral sequence. However, sequence-based models have a limitation of length. When the length of the user’s behavioral sequence exceeds the limitation of the model, the model cannot take advantage of the complete behavioral sequence of the user and cannot know the user’s holistic interests. The accuracy of the model then goes down. Meanwhile, sequence-based models only pay attention to the sequential signals of the data but do not pay attention to the spatial signals of the data, which will also affect the model’s accuracy.

  • graph neural networks (GNN)
  • recommendation
  • sequence
  • Graph Convolutional Network (GCN)
  • hybrid

1. Introduction

In the past few decades, because of the rapid development of the Internet, individuals can collect various information simply. However, massive amounts of information often make individuals unable to find items they are interested in; as a consequence, the personalized recommendation system is proposed. The personalized recommendation system can provide information filtering for users and screen out items that users need.
Many experiments [1][2][3][4][5] show that the user’s current interest is dynamic in nature, and the user’s behavioral sequence influences this interest. For example, if a user buys a phone, he is more likely to buy a phone shell, even though he would not usually buy one. In order to capture this dynamic interest, many models [6][7][8] based on users’ behavioral sequences have been proposed. These models learn the sequential signals between items and predict which item is most likely to be purchased based on the user’s behavioral sequence.
Although sequence-based models have achieved great success in recommendation systems, sequence-based models have the following disadvantages. First, the length of the sequence is limited. When the length of the user’s behavioral sequence exceeds the limit of the model, the model cannot fully capture the user’s interest, thus reducing the model’s accuracy. Furthermore, if the length is too long, it will inevitably increase the model prediction time. For example, in a scenario where the sequence length is 5, and the model predicts that the user is likely to buy a computer based on the last five items purchased by the user, due to the length limit of the model, the model cannot observe the complete purchase sequence of the user and, therefore, cannot find that the item had been purchased long ago. Second, sequence-based models only pay attention to the sequential signals between items but do not pay attention to the spatial signals of items and users, which will also affect the effectiveness of the model. This is because, without the spatial signals, the model cannot learn a user’s interest from other users.
Overall, the sequence-based models have the following issues: (1) Due to the limitation of sequence length, the sequence-based models may discard the long-term behavior of users and cannot fully capture the user’s interest. (2) Sequence-based models focus on the user’s recent behavior and ignore their complete behavior. (3) Sequence-based models only focus on the user’s behavior sequence and ignore the spatial signal between users and items.

2. Sequence-Based Recommendation

The task of sequence-based recommendation is to predict the next item the user will buy based on the user’s behavioral sequence. The Markov chain-based model is a classic sequence-based recommendation model [9][10]. Because of the development of deep learning, RNN-based models were introduced into sequence-based recommendations. Bal et al. used the RNN-based model GRU4Rec [11] for the first time to predict the next item that the user might be interested in. However, because Markov assumes that the current interaction only depends on one or several recent interactions, the results predicted by models would only be dependent on the user’s most recent behavioral sequence. In addition, CNN-based models were also introduced into sequential recommendations, such as Caser [12]. Since CNN-based models do not have a strong ordering assumption for the interaction in the sequence, the CNN-based recommendations can make up for the disadvantage based on RNN to a certain extent. Transformer [13], as a sequence-based model, achieved state-of-the-art performance and efficiency for machine translation tasks because of ’self-attention’. Therefore, an attention mechanism was introduced into sequence-based recommendations [14], and Wang et al. [15] proposed SASRec, which focuses more on the whole sequence instead of the most recent behavioral sequence. In addition, Li et al. [16] proposed TiSASRec, which is based on SASRec and uses the temporal factor, and Sun et al. [7] proposed Bert4Rec using Bidirectional Encoder Representations from Transformers (BERT). Yukuo et al. [17] introduced multiple interests into sequence-based recommendations to improve the model’s performance. Although the above models are efficient, they require a fixed length, which means that if the user’s sequence exceeds this length these models will not be able to express the user’s interests fully. In order to solve this problem, Bai et al. proposed LSDA [18], which uses multiple LSTM to learn the whole sequences of users, and Bo et al. proposed HAM [19]; this model uses MLP to learn the whole sequences of users. Nevertheless, these models only pay attention to the sequence and do not pay attention to the spatial signals of data.

3. Graph Convolutional Network (GCN)-Based Recommendation

Initially, traditional matrix factorization models [20][21] were used for the recommendation system. With the rise in deep learning methods, traditional matrix factorization models have been gradually replaced by deep neural networks, especially graph neural networks (GNN). Berg et al. [22] utilized a graph convolutional neural network in a recommendation system, and then Wang et al. [23] improved it and proposed NGCF. However, these models use the Laplacian matrix, and the matrix operation is troublesome; hence, He et al. [24] simplified the matrix operation and proposed LightGCN, which improved the efficiency of the operation in sparse datasets. Fan et al. [25] improved LightGCN by changing the global graph into a subgraph to improve the generalization of the model. In addition, Liang et al. [26] used auto-encoders for recommendations, which can effectively provide feedback on the implicit data of users. At the same time, Wang et al. [27] combined the attention mechanism with the graph neural network, proposed GAT, and applied it to recommendation systems [28]. Some scholars also use dynamic graphs to improve the recommendation ability of models [29][30][31][32]. GCN-based models can fully use the user’s behaviors and can also use the spatial signal of the data to learn the user’s holistic interests. However, GCN-based models cannot utilize the user’s temporal signals and capture the recent interests effectively.

4. Hybrid Recommendation

Since graph convolutional neural networks can handle graph-structured data well, some scholars combine GNN with other models and apply them to sequence-based recommendation systems. For example, SR-GNN [33], MA-GNN [34], and DGCN [35] combine GNN and GRU, which improves the generalization ability of the model. APGNN [36] combines GNN and attention mechanisms. However, these models that use GRU will focus too much on the recent behavior of users and ignore their complete behavior, and GRU’s computational complexity is high. Some scholars have also introduced a memory network into the recommendation system; thereby, the matrix can more explicitly and dynamically store and update the historical interactions in the sequence to improve the expressive ability of the model. For example, Chen et al. [37] and Huang et al. [38] added a memory network to the GRU to improve the expression ability of the model. Yuan et al. [39] used a memory network for session-based recommendations, and Wang et al. [40] used an attention network for the next location recommendation. Tan et al. [41] and Hsu et al. [42] used an attention graph for sequential recommendations. Meanwhile, some scholars introduce models that pay more attention to sequence. Zhu et al. proposed GES [43], combining GNN and Transformer, but this model focuses too much on the user’s recent behavioral sequence. These hybrid models combine GCN-based models and sequence-based models, but the above models still focus on the recent behavior sequence of users and lack attention to the complete behavior of users.

This entry is adapted from the peer-reviewed paper 10.3390/math12010164

References

  1. Sun, K.; Qian, T.; Zhong, M.; Li, X. Towards more effective encoders in pre-training for sequential recommendation. World Wide Web 2023, 26, 2801–2832.
  2. Jiang, N.; Hu, Z.; Wen, J.; Zhao, J.; Gu, W.; Tu, Z.; Liu, X.; Li, Y.; Gong, J.; Lin, F. NAH: Neighbor-aware attention-based heterogeneous relation network model in E-commerce recommendation. World Wide Web 2023, 26, 2373–2394.
  3. Duan, J.; Zhang, P.F.; Qiu, R.; Huang, Z. Long short-term enhanced memory for sequential recommendation. World Wide Web 2023, 26, 561–583.
  4. Zhu, L.; Zhu, Z.; Zhang, C.; Xu, Y.; Kong, X. Multimodal sentiment analysis based on fusion methods: A survey. Inf. Fusion 2023, 95, 306–325.
  5. Kim, Y.E.; Choi, S.M.; Lee, D.; Seo, Y.G.; Lee, S. A Reliable Prediction Algorithm Based on Genre2Vec for Item-Side Cold-Start Problems in Recommender Systems with Smart Contracts. Mathematics 2023, 11, 2962.
  6. Chen, Y.; Liu, Z.; Li, J.; McAuley, J.; Xiong, C. Intent contrastive learning for sequential recommendation. In Proceedings of the ACM Web Conference 2022, Singapore, 13–17 May 2024; pp. 2172–2182.
  7. Sun, F.; Liu, J.; Wu, J.; Pei, C.; Lin, X.; Ou, W.; Jiang, P. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, Beijing, China, 3–7 November 2019; pp. 1441–1450.
  8. Zheng, Y.; Gao, C.; Chang, J.; Niu, Y.; Song, Y.; Jin, D.; Li, Y. Disentangling long and short-term interests for recommendation. In Proceedings of the ACM Web Conference 2022, Virtual, 25–29 April 2022; pp. 2256–2267.
  9. Rendle, S.; Freudenthaler, C.; Schmidt-Thieme, L. Factorizing personalized markov chains for next-basket recommendation. In Proceedings of the 19th International Conference on World Wide Web, Raleigh, NC, USA, 26–30 April 2010; pp. 811–820.
  10. He, R.; McAuley, J. Fusing similarity models with markov chains for sparse sequential recommendation. In Proceedings of the 2016 IEEE 16th International Conference on Data Mining (ICDM), Barcelona, Spain, 12–15 December 2016; pp. 191–200.
  11. Hidasi, B.; Karatzoglou, A.; Baltrunas, L.; Tikk, D. Session-based Recommendations with Recurrent Neural Networks. In Proceedings of the 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, 2–4 May 2016.
  12. Tang, J.; Wang, K. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, Marina Del Rey, CA, USA, 5–9 February 2018; pp. 565–573.
  13. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. Adv. Neural Inf. Process. Syst. 2017, 30, 5998–6008.
  14. Liu, Q.; Zeng, Y.; Mokhosi, R.; Zhang, H. STAMP: Short-term attention/memory priority model for session-based recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK, 19–23 August 2018; pp. 1831–1839.
  15. Kang, W.C.; McAuley, J. Self-attentive sequential recommendation. In Proceedings of the 2018 IEEE International Conference on Data Mining (ICDM), Singapore, 17–20 November 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 197–206.
  16. Li, J.; Wang, Y.; McAuley, J. Time interval aware self-attention for sequential recommendation. In Proceedings of the 13th International Conference on Web Search and Data Mining, Houston, TX, USA, 3–7 February 2020; pp. 322–330.
  17. Cen, Y.; Zhang, J.; Zou, X.; Zhou, C.; Yang, H.; Tang, J. Controllable Multi-Interest Framework for Recommendation. In Proceedings of the KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual, 23–27 August 2020; Gupta, R., Liu, Y., Tang, J., Prakash, B.A., Eds.; Association for Computing Machinery: New York, NY, USA, 2020.
  18. Bai, T.; Du, P.; Zhao, W.X.; Wen, J.; Nie, J. A Long-Short Demands-Aware Model for Next-Item Recommendation. arXiv 2019, arXiv:1903.00066.
  19. Peng, B.; Ren, Z.; Parthasarathy, S.; Ning, X. HAM: Hybrid Associations Model with Pooling for Sequential Recommendation. IEEE Trans. Knowl. Data Eng. 2020, 34, 4838–4853.
  20. Lee, D.D.; Seung, H.S. Algorithms for Non-Negative Matrix Factorization. In Proceedings of the NIPS’00: 13th International Conference on Neural Information Processing Systems, Denver, CO, USA, 1 January 2000; pp. 535–541.
  21. Rendle, S.; Freudenthaler, C.; Gantner, Z.; Schmidt-Thieme, L.B. Bayesian personalized ranking from implicit feedback. In Proceedings of the 25 Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, 18–21 June 2009; pp. 452–461.
  22. Li, X.; She, J. Collaborative variational autoencoder for recommender systems. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, 13–17 August 2017; pp. 305–314.
  23. Wang, X.; He, X.; Wang, M.; Feng, F.; Chua, T.S. Neural graph collaborative filtering. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, Paris, France, 21–25 July 2019; pp. 165–174.
  24. He, X.; Deng, K.; Wang, X.; Li, Y.; Zhang, Y.; Wang, M. Lightgcn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual, 25–30 July 2020; pp. 639–648.
  25. Liu, F.; Cheng, Z.; Zhu, L.; Gao, Z.; Nie, L. Interest-aware Message-Passing GCN for Recommendation. In Proceedings of the WWW ’21: The Web Conference 2021, Virtual, 19–23 April 2021; Leskovec, J., Grobelnik, M., Najork, M., Tang, J., Zia, L., Eds.; Association for Computing Machinery: New York, NY, USA, 2021; pp. 1296–1305.
  26. Liang, D.; Krishnan, R.G.; Hoffman, M.D.; Jebara, T. Variational autoencoders for collaborative filtering. In Proceedings of the 2018 World Wide Web Conference, Lyon, France, 23–27 April 2018; pp. 689–698.
  27. Wang, X.; Wang, R.; Shi, C.; Song, G.; Li, Q. Multi-component graph convolutional collaborative filtering. Proc. AAAI Conf. Artif. Intell. 2020, 34, 6267–6274.
  28. Xia, F.; Sun, K.; Yu, S.; Aziz, A.; Wan, L.; Pan, S.; Liu, H. Graph learning: A survey. IEEE Trans. Artif. Intell. 2021, 2, 109–127.
  29. Zhang, Y.; Shen, G.; Han, X.; Wang, W.; Kong, X. Spatio-Temporal Digraph Convolutional Network Based Taxi Pick-Up Location Recommendation. IEEE Trans. Ind. Inform. 2022, 19, 394–403.
  30. Shen, G.; Tan, J.; Liu, Z.; Kong, X. Enhancing interactive graph representation learning for review-based item recommendation. Comput. Sci. Inf. Syst. 2022, 19, 573–593.
  31. Liu, T.; Lou, S.; Liao, J.; Feng, H. Dynamic and Static Representation Learning Network for Recommendation. IEEE Trans. Neural Netw. Learn. Syst. 2022.
  32. Zhang, X.; Wang, Z.; Du, B. Graph-aware collaborative reasoning for click-through rate prediction. World Wide Web 2023, 26, 967–987.
  33. Xu, D.; Ruan, C.; Korpeoglu, E.; Kumar, S.; Achan, K. Inductive representation learning on temporal graphs. arXiv 2020, arXiv:2002.07962.
  34. Ma, C.; Ma, L.; Zhang, Y.; Sun, J.; Liu, X.; Coates, M. Memory augmented graph neural networks for sequential recommendation. Proc. AAAI Conf. Artif. Intell. 2020, 34, 5045–5052.
  35. Zheng, Y.; Gao, C.; Chen, L.; Jin, D.; Li, Y. DGCN: Diversified Recommendation with Graph Convolutional Networks. In Proceedings of the Web Conference 2021, Ljubljana, Slovenia, 19–23 April 2021; pp. 401–412.
  36. Zhang, M.; Wu, S.; Gao, M.; Jiang, X.; Xu, K.; Wang, L. Personalized graph neural networks with attention mechanism for session-aware recommendation. IEEE Trans. Knowl. Data Eng. 2020, 34, 3946–3957.
  37. Qu, S.; Yuan, F.; Guo, G.; Zhang, L.; Wei, W. CmnRec: Sequential Recommendations with Chunk-accelerated Memory Network. IEEE Trans. Knowl. Data Eng. 2022, 35, 3540–3550.
  38. Huang, J.; Zhao, W.X.; Dou, H.; Wen, J.R.; Chang, E.Y. Improving sequential recommendation with knowledge-enhanced memory networks. In Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, Ann Arbor, MI, USA, 8–12 July 2018; pp. 505–514.
  39. Yuan, J.; Song, Z.; Sun, M.; Wang, X.; Zhao, W.X. Dual Sparse Attention Network For Session-based Recommendation. Proc. AAAI Conf. Artif. Intell. 2021, 35, 4635–4643.
  40. Wang, R.; Wu, Z.; Lou, J.; Jiang, Y. Attention-based dynamic user modeling and deep collaborative filtering recommendation. Expert Syst. Appl. 2022, 188, 116036.
  41. Tan, Q.; Zhang, J.; Liu, N.; Huang, X.; Yang, H.; Zhou, J.; Hu, X. Dynamic Memory based Attention Network for Sequential Recommendation. Proc. AAAI Conf. Artif. Intell. 2021, 35, 4384–4392.
  42. Hsu, C.; Li, C. RetaGNN: Relational Temporal Attentive Graph Neural Networks for Holistic Sequential Recommendation. In Proceedings of the WWW ’21: The Web Conference 2021, Virtual, 19–23 April 2021; Leskovec, J., Grobelnik, M., Najork, M., Tang, J., Zia, L., Eds.; Association for Computing Machinery: New York, NY, USA, 2021; pp. 2968–2979.
  43. Zhu, T.; Sun, L.; Chen, G. Graph-based Embedding Smoothing for Sequential Recommendation. IEEE Trans. Knowl. Data Eng. 2021, 35, 496–508.
More
© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license; additional terms may apply. By using this site, you agree to the Terms and Conditions and Privacy Policy.
This entry is offline, you can click here to edit this entry!
Video Production Service
Feedback