We address the problem of estimating the relative 6D pose, e.g., position and orientation, of a target spacecraft from a monocular image, a key capability for future autonomous rendezvous and proximity operations. Due to the difficulty of acquiring large sets of real images, spacecraft pose estimation networks are exclusively trained on synthetic ones. However, because those images do not capture the illumination conditions encountered in orbit, pose estimation networks face a domain gap problem, e.g., they do not generalize to real images. Our work introduces a method that bridges this domain gap. It relies on a novel, end-to-end, neural-based architecture as well as a novel learning strategy. This strategy improves the domain generalization abilities of the network through multitask learning and aggressive data augmentation policies, thereby enforcing the network to learn domain-invariant features. We demonstrate that our method effectively closes the domain gap, achieving state-of-the-art accuracy on the widespread SPEED+ dataset. Finally, ablation studies assess the impact of key components of our method on its generalization abilities.
[1] Forshaw J. L., Aglietti G. S., Navarathinam N., Kadhem H., Salmon T., Pisseloup A., Joffre E., Chabot T., Retat I., Axthelm R. and et al., "RemoveDEBRIS: An In-Orbit Active Debris Removal Demonstration Mission," Acta Astronautica, Vol. 127, Oct.-Nov. 2016, pp. 448-463. https://doi.org/10.1016/j.actaastro.2016.06.018 CrossrefGoogle Scholar
[2] Reed B. B., Smith R. C., Naasz B. J., Pellegrino J. F. and Bacon C. E., "The Restore-L Servicing Mission," AIAA Space 2016, AIAA Paper 2016-5478, 2016. https://doi.org/10.2514/6.2016-5478 LinkGoogle Scholar
[3] Biesbroek R., Aziz S., Wolahan A., Cipolla S.-f., Richard-Noca M. and Piguet L., "The Clearspace-1 Mission: ESA and Clearspace Team up to Remove Debris," Proceedings of the 8th Conference on Space Debris, ESA Space Debris Office, Paris, France, 2021, pp. 1-3. Google Scholar
[4] Henshaw C. G., "The Darpa Phoenix Spacecraft Servicing Program: Overview and Plans for Risk Reduction," International Symposium on Artificial Intelligence, Robotics and Automation in Space (i-SAIRAS), European Space Agency, Paris, France, 2014. Google Scholar
[5] Pyrak M. and Anderson J., "Performance of Northrop Grumman's Mission Extension Vehicle (MEV) RPO Imagers at GEO," Autonomous Systems: Sensors, Processing and Security for Ground, Air, Sea and Space Vehicles and Infrastructure 2022, Vol. 12115, SPIE, 2022, pp. 64-82. https://doi.org/10.1117/12.2631524 Google Scholar
[6] Ventura J., "Autonomous Proximity Operations for Noncooperative Space Targets," Ph.D. Thesis, Technische Universität München, München, Germany, 2016. Google Scholar
[7] Opromolla R., Fasano G., Rufino G. and Grassi M., "A Review of Cooperative and Uncooperative Spacecraft Pose Determination Techniques for Close-Proximity Operations," Progress in Aerospace Sciences, Vol. 93, Aug. 2017, pp. 53-72. https://doi.org/10.1016/j.paerosci.2017.07.001 CrossrefGoogle Scholar
[8] Opromolla R., Fasano G., Rufino G. and Grassi M., "Pose Estimation for Spacecraft Relative Navigation Using Model-Based Algorithms," IEEE Transactions on Aerospace and Electronic Systems, Vol. 53, No. 1, 2017, pp. 431-447. https://doi.org/10.1109/TAES.2017.2650785 CrossrefGoogle Scholar
[9] Christian J. A. and Cryan S., "A Survey of LIDAR Technology and Its Use in Spacecraft Relative Navigation," AIAA Guidance, Navigation, and Control (GNC) Conference, AIAA Paper 2013-4641, 2013. https://doi.org/10.2514/6.2013-4641 LinkGoogle Scholar
[10] Shi J.-F., Ulrich S., Ruel S. and Anctil M., "Uncooperative Spacecraft Pose Estimation Using an Infrared Camera during Proximity Operations," AIAA SPACE 2015 Conference and Exposition, AIAA Paper 2015-4429, 2015. https://doi.org/10.2514/6.2015-4429 LinkGoogle Scholar
[11] Rondao D., Aouf N. and Richardson M. A., "ChiNet: Deep Recurrent Convolutional Learning for Multimodal Spacecraft Pose Estimation," IEEE Transactions on Aerospace and Electronic Systems, Vol. 59, No. 2, 2022, pp. 937-949. https://doi.org/10.1109/TAES.2022.3193085 Google Scholar
[12] Martínez H. G., Giorgi G. and Eissfeller B., "Pose Estimation and Tracking of Non-Cooperative Rocket Bodies Using Time-of-Flight Cameras," Acta Astronautica, Vol. 139, Oct. 2017, pp. 165-175. https://doi.org/10.1016/j.actaastro.2017.07.002 CrossrefGoogle Scholar
[13] Pesce V., Lavagna M. and Bevilacqua R., "Stereovision-Based Pose and Inertia Estimation of Unknown and Uncooperative Space Objects," Advances in Space Research, Vol. 59, No. 1, 2017, pp. 236-251. https://doi.org/10.1016/j.asr.2016.10.002 CrossrefGoogle Scholar
[14] Cosmas K. and Kenichi A., "Utilization of FPGA for Onboard Inference of Landmark Localization in CNN-Based Spacecraft Pose Estimation," Aerospace, Vol. 7, No. 11, 2020, p. 159. https://doi.org/10.3390/aerospace7110159 CrossrefGoogle Scholar
[15] Leon V., Lentaris G., Soudris D., Vellas S. and Bernou M., "Towards Employing FPGA and ASIP Acceleration to Enable Onboard AI/ML in Space Applications," 2022 IFIP/IEEE 30th International Conference on Very Large Scale Integration (VLSI-SoC), Inst. of Electrical and Electronics Engineers, New York, 2022, pp. 1-4. https://doi.org/10.1109/VLSI-SoC54400.2022.9939566 Google Scholar
[16] Chen B., Cao J., Parra A. and Chin T.-J., "Satellite Pose Estimation with Deep Landmark Regression and Nonlinear Pose Refinement," Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, Inst. of Electrical and Electronics Engineers, New York, 2019. https://doi.org/10.1109/ICCVW.2019.00343 Google Scholar
[17] Sharma S. and D'Amico S., "Pose Estimation for Non-Cooperative Rendezvous Using Neural Networks," arXiv preprint arXiv: 1906.09868, 2019. https://doi.org/10.48550/arXiv.1906.09868 Google Scholar
[18] Park T. H., Sharma S. and D'Amico S., "Towards Robust Learning-Based Pose Estimation of Noncooperative Spacecraft," arXiv preprint arXiv: 1909.039, 2019. https://doi.org/10.48550/arXiv.1909.00392 Google Scholar
[19] Park T. H., Märtens M., Jawaid M., Wang Z., Chen B., Chin T.-J., Izzo D. and D'Amico S., "Satellite Pose Estimation Competition 2021: Results and Analyses," Acta Astronautica, Vol. 204, March 2023, pp. 640-665. https://doi.org/10.1016/j.actaastro.2023.01.002 CrossrefGoogle Scholar
[20] Pérez-Villar J. I. B., García-Martín Á. and Bescós J., "Spacecraft Pose Estimation Based on Unsupervised Domain Adaptation and on a 3D-Guided Loss Combination," European Conference on Computer Vision, Vol. 13801, Springer Nature, Cham, Switzerland, 2022, pp. 37-52. https://doi.org/10.1007/978-3-031-25056-9_3 Google Scholar
[21] Wang Z., Chen M., Guo Y., Li Z. and Yu Q., "Bridging the Domain Gap in Satellite Pose Estimation: A Self-Training Approach Based on Geometrical Constraints," IEEE Transactions on Aerospace and Electronic Systems, Vol. 60, No. 3, 2023, pp. 2,500-2,514. https://doi.org/10.1109/TAES.2023.3250385 CrossrefGoogle Scholar
[22] Goodfellow I., Pouget-Abadie J., Mirza M., Xu B., Warde-Farley D., Ozair S., Courville A. and Bengio Y., "Generative Adversarial Nets," Advances in Neural Information Processing Systems, Vol. 27, Neural Information Processing Systems Foundation, San Diego, CA, 2014. Google Scholar
[23] Sharma S., Ventura J. and D'Amico S., "Robust Model-Based Monocular Pose Initialization for Noncooperative Spacecraft Rendezvous," Journal of Spacecraft and Rockets, Vol. 55, No. 6, 2018, pp. 1414-1429. https://doi.org/10.2514/1.A34124 LinkGoogle Scholar
[24] Kisantal M., Sharma S., Park T. H., Izzo D., Märtens M. and D'Amico S., "Satellite Pose Estimation Challenge: Dataset, Competition Design, and Results," IEEE Transactions on Aerospace and Electronic Systems, Vol. 56, No. 5, 2020, pp. 4083-4098. https://doi.org/10.1109/TAES.2020.2989063 CrossrefGoogle Scholar
[25] Sun K., Xiao B., Liu D. and Wang J., "Deep High-Resolution Representation Learning for Human Pose Estimation," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Inst. of Electrical and Electronics Engineers, New York, 2019, pp. 5693-5703. https://doi.org/10.1109/CVPR.2019.00584 Google Scholar
[26] Marquardt D. W., "An Algorithm for Least-Squares Estimation of Nonlinear Parameters," Journal of the Society for Industrial and Applied Mathematics, Vol. 11, No. 2, 1963, pp. 431-441. https://doi.org/10.1137/0111030 CrossrefGoogle Scholar
[27] Sandler M., Howard A., Zhu M., Zhmoginov A. and Chen L.-C., "Mobilenetv2: Inverted Residuals and Linear Bottlenecks," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Inst. of Electrical and Electronics Engineers, New York, 2018, pp. 4510-4520. https://doi.org/10.1109/CVPR.2018.00474 Google Scholar
[28] Gerard K., "Segmentation-Driven Satellite Pose Estimation," Kelvins Day Presentation, Vol. 9, 2019, https://indico.esa.int/event/319/attachments/3561/4754/pose_gerard_segmentation.pdf. Google Scholar
[29] Redmon J. and Farhadi A., "Yolov3: An Incremental Improvement," arXiv preprint arXiv: 1804.02767, 2018. https://doi.org/10.48550/arXiv.1804.02767 Google Scholar
[30] Lepetit V., Moreno-Noguer F. and Fua P., "EPnP: An Accurate O(n) Solution to the PnP Problem," International Journal of Computer Vision, Vol. 81, No. 2, 2009, pp. 155-166. https://doi.org/10.1007/s11263-008-0152-6 CrossrefGoogle Scholar
[31] Fischler M. A. and Bolles R. C., "Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography," Communications of the ACM, Vol. 24, No. 6, 1981, pp. 381-395. https://doi.org/10.1145/358669.358692 CrossrefGoogle Scholar
[32] Hu Y., Speierer S., Jakob W., Fua P. and Salzmann M., "Wide-Depth-Range 6D Object Pose Estimation in Space," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Inst. of Electrical and Electronics Engineers, New York, 2021, pp. 15,870-15,879. https://doi.org/10.1109/CVPR46437.2021.01561 Google Scholar
[33] Proença P. F. and Gao Y., "Deep Learning for Spacecraft Pose Estimation from Photorealistic Rendering," 2020 IEEE International Conference on Robotics and Automation (ICRA), Inst. of Electrical and Electronics Engineers, New York, 2020, pp. 6007-6013. https://doi.org/10.1109/ICRA40945.2020.9197244 Google Scholar
[34] He K., Zhang X., Ren S. and Sun J., "Deep Residual Learning for Image Recognition," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Inst. of Electrical and Electronics Engineers, New York, 2016, pp. 770-778. https://doi.org/10.1109/CVPR.2016.90 Google Scholar
[35] Sonawani S., Alimo R., Detry R., Jeong D., Hess A. and Amor H. B., "Assistive Relative Pose Estimation for On-Orbit Assembly Using Convolutional Neural Networks," arXiv preprint arXiv: 2001.10673. 2020.https://doi.org/10.48550/arXiv.2001.10673, Google Scholar
[36] Legrand A., Detry R. and De Vleeschouwer C., "End-to-End Neural Estimation of Spacecraft Pose with Intermediate Detection of Keypoints," Computer Vision -- ECCV 2022 Workshops, Springer Nature, Cham, Switzerland, 2023, pp. 154-169. https://doi.org/10.1007/978-3-031-25056-9_11 Google Scholar
[37] Black K., Shankar S., Fonseka D., Deutsch J., Dhir A. and Akella M. R., "Real-Time, Flight-Ready, Non-Cooperative Spacecraft Pose Estimation Using Monocular Imagery," arXiv preprint arXiv: 2101.09553. https://doi.org/10.48550/arXiv.2101.095532021 Google Scholar
[38] Cassinis L. P., Menicucci A., Gill E., Ahrns I. and Sanchez-Gestido M., "On-Ground Validation of a CNN-Based Monocular Pose Estimation System for Uncooperative Spacecraft: Bridging Domain Shift in Rendezvous Scenarios," Acta Astronautica, Vol. 196, July 2022, pp. 123-138. https://doi.org/10.1016/j.actaastro.2022.04.002 CrossrefGoogle Scholar
[39] Posso J., Bois G. and Savaria Y., "Mobile-Ursonet: an Embeddable Neural Network for Onboard Spacecraft Pose Estimation," 2022 IEEE International Symposium on Circuits and Systems (ISCAS), Inst. of Electrical and Electronics Engineers, New York, 2022, pp. 794-798. https://doi.org/10.1109/ISCAS48785.2022.9937721 Google Scholar
[40] Carcagnì P., Leo M., Spagnolo P., Mazzeo P. L. and Distante C., "A Lightweight Model for Satellite Pose Estimation," International Conference on Image Analysis and Processing, Vol. 13231, Springer Nature, Cham, Switzerland, 2022, pp. 3-14. https://doi.org/10.1007/978-3-031-06427-2 Google Scholar
[41] Zhu J.-Y., Park T., Isola P. and Efros A. A., "Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks," Proceedings of the IEEE International Conference on Computer Vision, Inst. of Electrical and Electronics Engineers, New York, 2017, pp. 2223-2232. Google Scholar
[42] Legrand A., Detry R. and De Vleeschouwer C., "Leveraging Neural Radiance Fields for Pose Estimation of an Unknown Space Object During Proximity Operations," arXiv preprint arXiv: 2405.12728. https://doi.org/10.48550/arXiv.2405.127282024. Google Scholar
[43] Mildenhall B., Srinivasan P. P., Tancik M., Barron J. T., Ramamoorthi R. and Ng R., "Nerf: Representing Scenes as Neural Radiance Fields for View Synthesis," Communications of the ACM, Vol. 65, No. 1, 2021, pp. 99-106. https://doi.org/10.1145/3503250 CrossrefGoogle Scholar
[44] Park T. H. and D'Amico S., "Robust Multi-Task Learning and Online Refinement for Spacecraft Pose Estimation Across Domain Gap," Advances in Space Research, Vol. 73, No. 11, 2023, pp. 5726-5740. https://doi.org/10.1016/j.asr.2023.03.036 CrossrefGoogle Scholar
[45] Park T. H. and D'Amico S., "Bridging Domain Gap for Flight-Ready Spaceborne Vision," arXiv preprint arXiv: 2409.11661. 2024. https://doi.org/10.48550/arXiv.2409.11661 Google Scholar
[46] Hendrycks D., Basart S., Mu N., Kadavath S., Wang F., Dorundo E., Desai R., Zhu T., Parajuli S., Guo M. and et al., "The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization," Proceedings of the IEEE/CVF International Conference on Computer Vision, Inst. of Electrical and Electronics Engineers, New York, 2021, pp. 8340-8349. https://doi.org/10.1109/ICCV48922.2021.00823 Google Scholar
[47] Xu Z., Liu D., Yang J., Raffel C. and Niethammer M., "Robust and Generalizable Visual Representation Learning via Random Convolutions," Proceedings of the International Conference on Learning Representations, OpenReview.net, 2021, https://openreview.net/forum?id=BVSM0x3EDK6. https://doi.org/10.48550/arXiv.2007.13003 Google Scholar
[48] Xu Y., Zhang J., Zhang Q. and Tao D., "Vitpose: Simple Vision Transformer Baselines for Human Pose Estimation," Advances in Neural Information Processing Systems. Vol. 35, Neural Information Processing Systems Foundation, San Diego, CA, 2022, pp. 38,571-38,584. Google Scholar
[49] Ulmer M., Durner M., Sundermeyer M., Stoiber M. and Triebel R., "6d Object Pose Estimation from Approximate 3d Models for Orbital Robotics," 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Inst. of Electrical and Electronics Engineers, New York, 2023, pp. 10,749-10,756. https://doi.org/10.1109/IROS55552.2023.10341511 Google Scholar
[50] Cassinis L. P., Park T. H., Stacey N., D'Amico S., Menicucci A., Gill E., Ahrns I. and Sanchez-Gestido M., "Leveraging Neural Network Uncertainty in Adaptive Unscented Kalman Filter for Spacecraft Pose Estimation," Advances in Space Research, Vol. 71, No. 12, 2023, pp. 5061-5082. https://doi.org/10.1016/j.asr.2023.02.021 CrossrefGoogle Scholar
[51] Tan M., Pang R. and Le Q. V., "Efficientdet: Scalable and Efficient Object Detection," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Inst. of Electrical and Electronics Engineers, New York, 2020, pp. 10,781-10,790. https://doi.org/10.1109/CVPR42600.2020.01079 Google Scholar
[52] Nibali A., He Z., Morgan S. and Prendergast L., "Numerical Coordinate Regression with Convolutional Neural Networks," arXiv preprint arXiv: 1801.07372, 2018. https://doi.org/10.48550/arXiv.1801.07372 Google Scholar
[53] Zhou Y., Barnes C., Lu J., Yang J. and Li H., "On the Continuity of Rotation Representations in Neural Networks," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Inst. of Electrical and Electronics Engineers, New York, 2019, pp. 5745-5753. https://doi.org/10.1109/CVPR.2019.00589 Google Scholar
[54] Good I. J., "Rational Decisions," Journal of the Royal Statistical Society: Series B (Methodological), Vol. 14, No. 1, 1952, pp. 107-114. https://doi.org/10.1111/j.2517-6161.1952.tb00104.x CrossrefGoogle Scholar
[55] Park T. H., Märtens M., Lecuyer G., Izzo D. and D'Amico S., "SPEED+: Next-Generation Dataset for Spacecraft Pose Estimation Across Domain Gap," 2022 IEEE Aerospace Conference (AERO), Inst. of Electrical and Electronics Engineers, New York, 2022, pp. 1-15. https://doi.org/10.1109/AERO53065.2022.9843439 Google Scholar
[56] Tobin J., Fong R., Ray A., Schneider J., Zaremba W. and Abbeel P., "Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World," 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Inst. of Electrical and Electronics Engineers, New York, 2017, pp. 23-30. https://doi.org/10.1109/IROS.2017.8202133 Google Scholar
[57] Singh K. K., Yu H., Sarmasi A., Pradeep G. and Lee Y. J., "Hide-and-Seek: A Data Augmentation Technique for Weakly-Supervised Localization and Beyond," arXiv preprint arXiv: 1811.02545, 2018. https://doi.org/10.48550/arXiv.1811.02545 Google Scholar
[58] Sakkos D., Shum H. P. and Ho E. S., "Illumination-Based Data Augmentation for Robust Background Subtraction," 2019 13th International Conference on Software, Knowledge, Information Management and Applications (SKIMA), Inst. of Electrical and Electronics Engineers, New York, 2019, pp. 1-8. https://doi.org/10.1109/SKIMA47702.2019.8982527 Google Scholar
[59] Geirhos R., Rubisch P., Michaelis C., Bethge M., Wichmann F. A. and Brendel W., "ImageNet-Trained CNNs are Biased Towards Texture; Increasing Shape Bias Improves Accuracy and Robustness," arXiv preprint arXiv: 1811.12231, 2018. https://doi.org/10.48550/arXiv.1811.12231 Google Scholar
[60] Jackson P. T., Atapour-Abarghouei A., Bonner S., Breckon T. P. and Obara B., "Style Augmentation: Data Augmentation via Style Randomization," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Inst. of Electrical and Electronics Engineers, New York, 2019, pp. 83-92. https://doi.org/10.48550/arXiv.1809.05375 Google Scholar
[61] Xu Q., Zhang R., Zhang Y., Wang Y. and Tian Q., "A Fourier-Based Framework for Domain Generalization," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Inst. of Electrical and Electronics Engineers, New York, 2021, pp. 14,383-14,392. https://doi.org/10.1109/CVPR46437.2021.01415 Google Scholar
[62] Xu Q., Zhang R., Fan Z., Wang Y., Wu Y.-Y. and Zhang Y., "Fourier-Based Augmentation with Applications to Domain Generalization," Pattern Recognition, Vol. 139, July 2023, Paper 109474. https://doi.org/10.1016/j.patcog.2023.109474 CrossrefGoogle Scholar
[63] Cooley J. W. and Tukey J. W., "An Algorithm for the Machine Calculation of Complex Fourier Series," Mathematics of Computation, Vol. 19, No. 90, 1965, pp. 297-301. https://doi.org/10.1090/S0025-5718-1965-0178586-1 CrossrefGoogle Scholar
[64] Cubuk E. D., Zoph B., Shlens J. and Le Q. V., "Randaugment: Practical Automated Data Augmentation with a Reduced Search Space," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Inst. of Electrical and Electronics Engineers, New York, 2020, pp. 702-703. https://doi.org/10.1109/CVPRW50498.2020.00359 Google Scholar
[65] Gill E., D'Amico S. and Montenbruck O., "Autonomous Formation Flying for the PRISMA Mission," Journal of Spacecraft and Rockets, Vol. 44, No. 3, 2007, pp. 671-681. https://doi.org/10.2514/1.23015 LinkGoogle Scholar
[66] Park T. H., Bosse J. and D'Amico S., "Robotic Testbed for Rendezvous and Optical Navigation: Multi-Source Calibration and Machine Learning use Cases," arXiv preprint arXiv: 2108.05529, 2021. https://doi.org/10.48550/arXiv.2108.05529 Google Scholar
[67] Kingma D. P. and Ba J., "Adam: A Method for Stochastic Optimization," arXiv preprint arXiv: 1412.6980, 2014. https://doi.org/10.48550/arXiv.1412.6980 Google Scholar
[68] Loshchilov I. and Hutter F., "Sgdr: Stochastic Gradient Descent with Warm Restarts," arXiv preprint arXiv: 1608.03983, 2016. https://doi.org/10.48550/arXiv.1608.03983 Google Scholar
[69] Yu C., Xiao B., Gao C., Yuan L., Zhang L., Sang N. and Wang J., "Lite-Hrnet: A Lightweight High-Resolution Network," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Inst. of Electrical and Electronics Engineers, New York, 2021, pp. 10,440-10,450. https://doi.org/10.1109/CVPR46437.2021.01030 Google Scholar
[70] Cassinis L. P., "Monocular Vision-Based Pose Estimation of Uncooperative Spacecraft," Ph.D. Thesis, TU Delft, Delft, The Netherlands, 2022. Google Scholar