Because of the recent advances in drones or Unmanned Aerial Vehicle (UAV) platforms, sensors and software, UAVs have gained popularity among precision agriculture researchers and stakeholders for estimating traits such as crop yield and diseases. Early detection of crop disease is essential to prevent possible losses on crop yield and ultimately increasing the benefits. However, accurate estimation of crop disease requires modern data analysis techniques such as machine learning and deep learning. This work aims to review the actual progress in crop disease detection, with an emphasis on machine learning and deep learning techniques using UAV-based remote sensing. First, we present the importance of different sensors and image-processing techniques for improving crop disease estimation with UAV imagery. Second, we propose a taxonomy to accumulate and categorize the existing works on crop disease detection with UAV imagery. Third, we analyze and summarize the performance of various machine learning and deep learning methods for crop disease detection. Finally, we underscore the challenges, opportunities and research directions of UAV-based remote sensing for crop disease detection.