Light curves represent the brightness of objects (stars, RSOs and planets) as a function of time. They are commonly used in various space applications (astronomy, planetary studies, etc.) to understand the characteristics of celestial objects. Various studies on SSA, such as [1,2,3,4,5], rely on light curve information to extract the properties of RSOs in the trends of this time series data; as such, extracting RSO features from these data are proven to be very valuable. Light curves have also been used for estimating ballistic coefficients for LEO RSOs, as proven in [6]. Additionally, light curves can be simulated as was performed by [7,8,9,10], which allows researchers to obtain large quantities of data on which to perform analyses.

Assessing an RSO’s stability (spin rate or pointing stability) is useful as researchers identify the characteristics of RSOs through light curves even when researchers don’t know the identity of the RSO researchers are observing. Once the stability of the RSO has been determined, researchers classify the RSO into various categories. Figure 1 illustrates the commonly accepted RSO classes.

There have been multiple studies on the use of ontology for RSO characterization and classification using the classes listed in Figure 1 [11,12,13]. The most basic classification is by object type. RSOs include satellites, rocket bodies, or other debris. Satellites are further classified by the presence or absence of an active control system which generates two subcategories: stable and tumbling satellites. Another common label for RSOs is the orbit type, which is determined from the altitude of the orbit. There are three main orbit types, LEO, medium Earth orbit (MEO), and high Earth orbit (HEO). A very common HEO orbit is a geostationary orbit (GEO), which is typically placed as a separate category. GEO orbits are unique as they have an altitude of 35,786 km and an orbital period that equals Earth’s rotation around itself. RSOs are also classified by their geometry and the direction they are pointing at. When considering observations of space objects, there are more categories that are useful in SSA, the most important of which is the material an RSO is made of, which impacts its reflection patterns. Another important parameter is the orientation, as the shape of the RSO as perceived by the observer appears different depending on its attitude. This also affects studies of brightness, as the reflection from a solar panel is different from that of the bus.

To classify light curves, researchers first implemented an SVM with two different methods of preparing the inputs to study the difference in performance. Then, researchers trained an LSTM RNN to assess the difference between a simple machine learning approach and a neural network. To compare the results, researchers computed some performance metrics such as accuracy, precision, recall, and F1 score. Since researchers have a multi-class classification problem with a different number of elements per class, weighted averaging was used. The definition and equations for each of these terms are defined in detail in [46,47].

When implementing an SVM, researchers utilized two different approaches for generating the model inputs. researchers attempted a mean features approach and a window-based approach. For both types of inputs, researchers defined the same training parameters. To obtain higher accuracy results, researchers implemented a second order polynomial kernel as opposed to using a linear function. The next factor to choose was the type of multi-class classifier. When there are more than two output classes, there are two main ways to specify how the computation happens. The first approach is a one vs. all classification, where each class is compared with all other classes [48]. The second approach, called one vs. one, breaks the multi-class classification problem into a collection of binary classifications, where each class is compared to one class at a time forming pairs. A deeper comparison of the two techniques is in [49]. 

The first method of preparing inputs for SVM researchers attempted is by using a window-based approach. To perform this, researchers split each scattering time window as a separate input to the model and trained the SVM. When computing the performance metrics, researchers combined the signals back to the initial input size and took the label featured most for each RSO as the predicted label. The overall accuracy of this method was 60%, which is unacceptable for classification. Table 1 summarizes the performance metrics. The performance of the algorithm is poor specifically when considering precision, recall, specificity, and F1 score.

The second type of feature pre-processing involves using the mean features from WST as inputs to the SVM. This allows us to obtain a 2D set of features from a 3D array. The average of the columns, which consist of the second dimension of the matrix, is taken such that an m by n by o sized-matrix becomes an m by o sized array. By reducing the dimensional of the input, researchers reduce computation time significantly. The resulting test accuracy is 87%.

The overall performance of the classifier when using mean features has improved significantly as compared to the window-based approach (see Table 1), and the algorithm runs about 86% quicker as it only takes 7 s to train the network. Given that the values of all metrics are relatively similar to each other, researchers conclude that this approach results in more consistent classification regarding false positives and false negatives.

To study whether a neural network performs better LEO light curve classification as compared to a simpler approach like SVM, researchers trained an LSTM. This method was chosen specifically due to its wide use in the literature for multi-class classification. As opposed to SVM, which requires a few options to choose from before training, neural networks require the user to supply many hyperparameters, and the sheer number of potential options require some form of optimization technique to determine the best values to choose from. As a result, researchers implemented Bayesian optimization to select the best values for the hyperparameters.

There are many hyperparameters that researchers could optimize but, for this analysis, researchers chose to prioritize the number of hidden units, the maximum number of epochs, and the mini batch size, as well as the initial learning rate. The number of hidden or recurrent units refers to the number of internal layers within the LSTM network. As indicated in [50], if this number is too large, the network overfits the data quickly, which deteriorates the performance on a test set, but if the number is too small, then the network does not store enough details to efficiently solve the problem. The number of epochs refers to the number of times the model passes through the complete training set. Similar to the number of hidden units, too small a value misses essential features of the data, while too large a value risks overfitting [51]. Mini batch size refers to the number of inputs to use for each iteration. Thus, each epoch has a collection of batches. Finally, the learning rate indicates the change in weight of the model. A very small value takes too long to train and a very large value risks overshooting and diverging from the expected result.

After obtaining the hyperparameter values using Bayesian optimization, which took over a week to optimize 30 epochs, the LSTM network was trained, which resulted in 92% accuracy. Additional performance metrics are provided in Table 1. Training the LSTM was completed in 3.5 min, which is 30 times longer than SVM mean features. Although this algorithm is slower than the mean features SVM approach, LSTM has the best performance as compared to both SVM approaches with regards to all five performance metrics. This means that, although simpler solutions like SVM work relatively well for light curve classification, higher accuracy demands more complex methods such as deep learning with LSTM used as an example.