The concept of a smart house or smart living environment is more current than ever, as more and more everyday devices are equipped with smart capabilities. The importance of activity recognition research lies in the advantages of being able to monitor and assist a person who uses smart sensors. Internet of Things (IoT) technology is applied in multiple domains such as medicine, manufacturing, emergency response, logistics, and daily life activity recognition. The smartphone is one of the most used devices for HAR as it can record and process information itself. The major downside of using a smartphone for detecting the user's activity is data downtime for not wearing the device. If the smartphone is not worn or directly used by the user, the system does not receive any relevant data regarding the current activity and, thus, the HAR precision decreases. Smartphones are not necessarily worn consistently, as the wear position will greatly vary depending on the person and situation. A watch has a more stable wear pattern, as it is primarily worn on the user's wrist and is usually worn extensively for long periods of time. A smartwatch is a small device that can be easily and non-intrusively worn for long periods of time, making it ideal for data acquisition.

For HAR based on accelerometer and gyroscope data, provided by sensors also found in a smartwatch, the classic approach is to use the raw sensor data and preprocess it. Features are then extracted and used to train a neural network for activity recognition. In this scenario, the neural network input is represented by a series of numeric values that try to capture the essence of that particular activity. Based on the raw sensor data or even extracted features, plot images can be generated and fed to the neural network as input data instead of numerical values. In this scenario, the human activity recognition task becomes an image classification task, trying, in essence, to identify the activity based on the plot image using specific image classification neural networks. The numerical data that are turned into a plot image can be graphically represented in multiple ways depending on the type of plot image and the input raw data structure; these variations can have a significant impact on the activity recognition rate.

The `WISDM Smartphone and Smartwatch Activity and Biometrics Dataset' dataset was chosen for this implementation. This dataset is extensively used for human activity recognition and was chosen as it is one of the most important and used datasets for human activity recognition based on wearable devices.

We describe a HAR system able to perform real-time activity recognition based on internally generated plot images. The proposed system has multiple components:a data preprocessing app, a machine learning core processor, a machine learning processor Web API and a real-time Cloud human activity recognition system. The 'Data preprocessing app' handles the conversion of raw movement data from an accelerometer and gyroscope to a plot image. This conversion transforms a movement data window to a single image that will be used as input for the machine learning algorithm. The 'Machine learning core processor' represents the main computing logic that is a machine learning implementation trained using the previously generated movement images. After training, this component is capable of receiving an image and predicting its source activity. This behavior is supported only locally, as~this component does not support network interactions or advanced conversions from raw data to images. The 'Machine learning processor Web Api' is the component that incorporates the 'Machine learning core processor' and is able to support network connections and recognize human activity. The recognition process can be based on a generated image but this component is also able to generate the image itself based on raw accelerometer and gyroscope data. So, in order to recognize what activity a series of movement data is part of, a simple API call is sufficient. The 'Real-time Cloud human activity recognition system' represents the 'Machine learning processor Web Api' cloud correspondent, which is able to handle requests from multiple computer networks via the Internet. This component is not limited to a single local network and can be easily scaled and enhanced for the human activity recognition process. In this way, we can easily recognize real-time human activity from any source application that can make a web API request that contains movement data.

The main features developed are:
- the implementation of a real-time system for human activity recognition that can operate locally and in the cloud via rest API calls based on image plot recognition.
- the implementation and usage of a .NET C# console application to generate label images based on raw accelerometer and gyroscope sensor data.
- the creation of a .NET C\# application that contains a deep neural network that was created based on a pre-trained TensorFlow DNN model and trained for HAR using plot images.
- the integration of the created and trained neural network in a .NET Web API application capable of real-time activity recognition based on rest API calls.
- the further extension of the HAR Web API application capabilities to allow cloud-based activity recognition.
- the analysis of multiple scenarios for plot image generation configuration and plot types and the evaluation of the obtained activity recognition precision results.
- concluding that a real-time HAR system, based on plot image recognition and REST requests, can be a good system architecture for real-time activity recognition.

We used the WISDM dataset to train a real-time human activity recognition system that is based on a Resnet50 neural network that achieved the best precision of 94.89% using scatter plot images with overlapping scatter plots. Raw accelerometer and gyroscope data from the previously mentioned WISDM dataset are both used to generate the plot images that consist of the input data for the neural network training process.

For the obtained results, the following activities were used: Walking, Jogging, Stairs, Sitting, Standing, Typing, and Brushing Teeth, for five selected users. The usage of a reduced dataset is due to the large size and number of the generated image data sets, as we have reduced the available total number of 18 activities to 7 activities and the total number of 30 users to a smaller one of 5. Reducing the number of analyzed users and activities provided a decent working dataset and the decrease of the generated plot image dimensions further decreased the side of the generated plot image dataset.

We can clearly notice that Scatter plot images with the overlapping scatter plots method obtained the best result. Another method that obtained decent results is using population images with the 'BarMeanStDev' option. We managed to obtain a decent precision with a maximum value of 94.89% when using scatter plot images with overlapping scatter~plots.

In order to further improve the accuracy of the implemented system, additional plot-type images can be analyzed to see if we can gain a performance boost. Other types of neural networks can also be analyzed, and other custom TensorFlow models may even provide a better implementation to further expand the system. The system could be expanded to use more data from the initial dataset, by increasing the number of analyzed activities and users and trying to optimize the training time by using a more powerful training machine, and trying to lower the training time using other pre-trained deep neural networks.

This entry is adapted from the peer-reviewed paper 10.3390/app14010329