Light Curve Classification: Comparison
View Latest Version
Please note this is a comparison between Version 2 by Fanny Huang and Version 1 by Randa Qashoa.

Light curves are plots of brightness measured over time. In the field of Space Situational Awareness (SSA), light curves of Resident Space Objects (RSOs) can be utilized to infer information about an RSO such as the type of object, its attitude, and its shape.

  • light curve
  • low Earth orbit

1. RSO Light Curves

Light curves represent the brightness of objects (stars, RSOs and planets) as a function of time. They are commonly used in various space applications (astronomy, planetary studies, etc.) to understand the characteristics of celestial objects. Various studies on SSA, such as [1[1][2][3][4][5],2,3,4,5], rely on light curve information to extract the properties of RSOs in the trends of this time series data; as such, extracting RSO features from these data are proven to be very valuable. Light curves have also been used for estimating ballistic coefficients for LEO RSOs, as proven in [6]. Additionally, light curves can be simulated as was performed by [7[7][8][9][10],8,9,10], which allows researchers to obtain large quantities of data on which to perform analyses.
Assessing an RSO’s stability (spin rate or pointing stability) is useful as researchers identify the characteristics of RSOs through light curves even when researchers don’t know the identity of the RSO researchers are observing. Once the stability of the RSO has been determined, researchers classify the RSO into various categories. Figure 1 illustrates the commonly accepted RSO classes.
Figure 1. RSO classes.
There have been multiple studies on the use of ontology for RSO characterization and classification using the classes listed in Figure 1 [11,12,13][11][12][13]. The most basic classification is by object type. RSOs include satellites, rocket bodies, or other debris. Satellites are further classified by the presence or absence of an active control system which generates two subcategories: stable and tumbling satellites. Another common label for RSOs is the orbit type, which is determined from the altitude of the orbit. There are three main orbit types, LEO, medium Earth orbit (MEO), and high Earth orbit (HEO). A very common HEO orbit is a geostationary orbit (GEO), which is typically placed as a separate category. GEO orbits are unique as they have an altitude of 35,786 km and an orbital period that equals Earth’s rotation around itself. RSOs are also classified by their geometry and the direction they are pointing at. When considering observations of space objects, there are more categories that are useful in SSA, the most important of which is the material an RSO is made of, which impacts its reflection patterns. Another important parameter is the orientation, as the shape of the RSO as perceived by the observer appears different depending on its attitude. This also affects studies of brightness, as the reflection from a solar panel is different from that of the bus.

2. Light Curve Classification

To classify light curves, researchers first implemented an SVM with two different methods of preparing the inputs to study the difference in performance. Then, researchers trained an LSTM RNN to assess the difference between a simple machine learning approach and a neural network. To compare the results, researchers computed some performance metrics such as accuracy, precision, recall, and F1 score. Since researchers have a multi-class classification problem with a different number of elements per class, weighted averaging was used. The definition and equations for each of these terms are defined in detail in [46,47][14][15].

2.1. SVM Results

When implementing an SVM, researchers utilized two different approaches for generating the model inputs. researchers attempted a mean features approach and a window-based approach. For both types of inputs, researchers defined the same training parameters. To obtain higher accuracy results, researchers implemented a second order polynomial kernel as opposed to using a linear function. The next factor to choose was the type of multi-class classifier. When there are more than two output classes, there are two main ways to specify how the computation happens. The first approach is a one vs. all classification, where each class is compared with all other classes [48][16]. The second approach, called one vs. one, breaks the multi-class classification problem into a collection of binary classifications, where each class is compared to one class at a time forming pairs. A deeper comparison of the two techniques is in [49][17]. 

2.1.1. Window Based Approach

The first method of preparing inputs for SVM researchers attempted is by using a window-based approach. To perform this, researchers split each scattering time window as a separate input to the model and trained the SVM. When computing the performance metrics, researchers combined the signals back to the initial input size and took the label featured most for each RSO as the predicted label. The overall accuracy of this method was 60%, which is unacceptable for classification. Table 1 summarizes the performance metrics. The performance of the algorithm is poor specifically when considering precision, recall, specificity, and F1 score.
Table 1. Performance metrics of different light curve classifiers.
Method Accuracy Precision Recall Specificity F1 Score
SVM Window-Based Approach 60% 40% 48% 46% 40%
SVM Mean Features 87% 86% 82% 84% 81%
LSTM 92% 90% 89% 95% 89%

2.1.2. Mean Features

The second type of feature pre-processing involves using the mean features from WST as inputs to the SVM. This allows us to obtain a 2D set of features from a 3D array. The average of the columns, which consist of the second dimension of the matrix, is taken such that an m by n by o sized-matrix becomes an m by o sized array. By reducing the dimensional of the input, researchers reduce computation time significantly. The resulting test accuracy is 87%.
The overall performance of the classifier when using mean features has improved significantly as compared to the window-based approach (see Table 1), and the algorithm runs about 86% quicker as it only takes 7 s to train the network. Given that the values of all metrics are relatively similar to each other, researchers conclude that this approach results in more consistent classification regarding false positives and false negatives.

2.2. LSTM Implementation

To study whether a neural network performs better LEO light curve classification as compared to a simpler approach like SVM, researchers trained an LSTM. This method was chosen specifically due to its wide use in the literature for multi-class classification. As opposed to SVM, which requires a few options to choose from before training, neural networks require the user to supply many hyperparameters, and the sheer number of potential options require some form of optimization technique to determine the best values to choose from. As a result, researchers implemented Bayesian optimization to select the best values for the hyperparameters.

2.2.1. Applying Bayesian Optimization

There are many hyperparameters that researchers could optimize but, for this analysis, researchers chose to prioritize the number of hidden units, the maximum number of epochs, and the mini batch size, as well as the initial learning rate. The number of hidden or recurrent units refers to the number of internal layers within the LSTM network. As indicated in [50][18], if this number is too large, the network overfits the data quickly, which deteriorates the performance on a test set, but if the number is too small, then the network does not store enough details to efficiently solve the problem. The number of epochs refers to the number of times the model passes through the complete training set. Similar to the number of hidden units, too small a value misses essential features of the data, while too large a value risks overfitting [51][19]. Mini batch size refers to the number of inputs to use for each iteration. Thus, each epoch has a collection of batches. Finally, the learning rate indicates the change in weight of the model. A very small value takes too long to train and a very large value risks overshooting and diverging from the expected result.

2.2.2. LSTM Results

After obtaining the hyperparameter values using Bayesian optimization, which took over a week to optimize 30 epochs, the LSTM network was trained, which resulted in 92% accuracy. Additional performance metrics are provided in Table 1. Training the LSTM was completed in 3.5 min, which is 30 times longer than SVM mean features. Although this algorithm is slower than the mean features SVM approach, LSTM has the best performance as compared to both SVM approaches with regards to all five performance metrics. This means that, although simpler solutions like SVM work relatively well for light curve classification, higher accuracy demands more complex methods such as deep learning with LSTM used as an example.

References

  1. Friedman, A.M.; Frueh, C. Observability of Light Curve Inversion for Shape and Feature Determination Exemplified by a Case Analysis. J. Astronaut. Sci. 2022, 69, 537–569.
  2. Linares, R.; Jah, M.K.; Crassidis, J.L.; Nebelecky, C.K. Space object shape characterization and tracking using light curve and angles data. J. Guid. Control. Dyn. 2014, 37, 13–25.
  3. Dianetti, A.D.; Crassidis, J.L. Space object material determination from polarized light curves. In Proceedings of the AIAA Scitech 2019 Forum, San Diego, CA, USA, 7–11 January 2019.
  4. Matsushita, Y.; Arakawa, R.; Yoshimura, Y.; Hanada, T. Light Curve Analysis and Attitude Estimation of Space Objects Focusing on Glint. In Proceedings of the First International Orbital Debris Conference (IOC), Sugar Land, TX, USA, 9–12 December 2019.
  5. Šilha, J.; Zigo, M.; Hrobár, T.; Jevčák, P.; Verešvárska, M. Light curves application to space debris characterization and classification. In Proceedings of the 8th European Conference on Space Debris, Darmstadt, Germany, 20–23 April 2021.
  6. Cimmino, N.; Opromolla, R.; Fasano, G. Machine learning-based approach for ballistic coefficient estimation of resident space objects in LEO. Adv. Space Res. 2023, 71, 5007–5025.
  7. Clark, R.; Fu, Y.; Dave, S.; Lee, R.S.K. Resident Space Object (RSO) attitude and optical property estimation from space-based light curves. Adv. Space Res. 2022, 70, 3271–3280.
  8. Ceniceros, A.; Gaylor, D.E.; Anderson, J.; Pinon, E., III; Dao, P.; Rast, R. Comparison of BRDF-Predicted and Observed Light Curves of GEO Satellites. In Proceedings of the Advanced Maui Optical and Space Surveillance Technologies Conference, Maui, HI, USA, 15–18 September 2015.
  9. Badura, G.P.; Valenta, C.R.; Gunter, B. Convolutional Neural Networks for Inference of Space Object Attitude Status. J. Astronaut. Sci. 2022, 69, 593–626.
  10. Allworth, J.; Windrim, L.; Wardman, J.; Kucharski, D.; Bennett, J.; Bryson, M. Development of a high fidelity simulator for generalised photometric based space object classification using machine learning. In Proceedings of the International Astronautical Congress, IAC, Washington, DC, USA, 21–25 October 2019.
  11. Liu, B.; Yao, L.; Han, D. Harnessing ontology and machine learning for RSO classification. SpringerPlus 2016, 5, 1655.
  12. Furfaro, R.; Linares, R.; Gaylor, D.; Jah, M.; Walls, R. Resident Space Object Characterization and Behavior Understanding via Machine Learning and Ontology-based Bayesian Networks. In Proceedings of the Advanced Maui Optical and Space Surveillance Technologies Conference (AMOS), Maui, HI, USA, 20–23 September 2016.
  13. Liu, B.; Yao, L.; Wu, J.; Hao, Z.; Ding, Z. From Data Silos to Intelligent Web for RSO Recognition. In Proceedings of the IIAE Conference System, the 6th IIAE International Conference on Intelligent Systems and Image Processing (ICISIP), Matsue, Japan, 10–14 September 2018.
  14. Erickson, B.J.; Kitamura, F. Magician’s corner: 9. performance metrics for machine learning models. Radiol. Artif. Intell. 2021, 3, e200126.
  15. Grandini, M.; Bagli, E.; Visani, G. Metrics for Multi-Class Classification: An Overview. arXiv 2020, arXiv:2008.05756.
  16. Pawara, P.; Okafor, E.; Groefsema, M.; He, S.; Schomaker, L.R.; Wiering, M.A. One-vs-One classification for deep neural networks. Pattern Recognit. 2020, 108, 107528.
  17. Galar, M.; Fernández, A.; Barrenechea, E.; Bustince, H.; Herrera, F. An overview of ensemble methods for binary classifiers in multi-class problems: Experimental study on one-vs-one and one-vs-all schemes. Pattern Recognit. 2011, 44, 1761–1776.
  18. Reimers, N.; Gurevych, I. Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks. arXiv 2017, arXiv:1707.06799.
  19. Abbasimehr, H.; Shabani, M.; Yousefi, M. An optimized model using LSTM network for demand forecasting. Comput. Ind. Eng. 2020, 143, 106435.
More
© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/)
Video Production Service
Feedback