The efficiency of (chemo-)radiotherapy for rectal cancer is not only determined by the impact on the tumor cells themselves, but also by the highly individual surrounding tumor microenvironment, including immune cells. However, many aspects of the radiation-induced immune response remain to be fully understood.  This review summarizes existing literature about the effects of (chemo-)radiotherapy on the rectal cancer immune microenvironment, which can be both tumor- suppressive or pro-tumorigenic, by either promoting an effective anti-tumor immune response or mediating resistance. We further aim to highlight potential immune-modulating combination therapies, such as immune checkpoint inhibitors, that offer individualized approaches to target the heterogeneous tumor immune microenvironment.