Ensemble deep learning-enabled single-shot composite structured illumination microscopy (eDL-cSIM)

Jiaming Qian^{1, 2, 3},
Chunyao Wang^{1, 2, 3},
Hongjun Wu^{1, 2, 3},
Qian Chen³,
Chao Zuo^{1, 2, 3}

1.
Smart Computational Imaging (SCI) Laboratory, Nanjing University of Science and Technology, No
2.
Smart Computational Imaging Research Institute (SCIRI) of Nanjing University of Science and Technology, 210094, Nanjing, Jiangsu Province, China
3.
Jiangsu Key Laboratory of Visual Sensing & Intelligent Perception, No

Funds: This work was supported by National Natural Science Foundation of China (62405136, 62275125, 62275121, 12204239, 62175109), China Postdoctoral Science Foundation (BX20240486, 2024M754141), Youth Foundation of Jiangsu Province (BK20241466, BK20220946), Jiangsu Funding Program for Excellent Postdoctoral Talent (2024ZB671), Fundamental Research Funds for the Central Universities (30922010313), Fundamental Research Funds for the Central Universities (2023102001), and Open Research Fund of Jiangsu Key Laboratory of Spectral Imaging & Intelligent Sense (JSGP202201, JSGPCXZNGZ202402).

摘要

- /
- /
- /
- /
- /
- /
- /
- /
Abstract: Structured illumination microscopy (SIM) has emerged as a powerful super-resolution technique for studying protein dynamics in live cells thanks to its wide-field imaging mode and high photon efficiency. However, conventional SIM requires at least nine raw images to achieve super-resolution reconstruction, which limits its imaging speed and increases susceptibility to rapid sample dynamics. Moreover, the reliance of SIM on illumination parameters and algorithmic post-processing renders it vulnerable to reconstruction artifacts, especially at low signal-to-noise ratios. In this work, we propose a single-shot composite structured illumination microscopy method using ensemble deep learning (eDL-cSIM). Without modifying the original SIM setup, eDL-cSIM employs only one composite structured illumination pattern generated by 6-beam interferometry. The resultant composite-coded raw image, which contains multiplexed high-frequency spectral information beyond the diffraction limit, is further processed using ensemble deep learning to predict a high-quality, artifact-free super-resolved image. Experimental results demonstrate that eDL-cSIM integrates the advantages of various state-of-the-art neural networks, enabling robust super-resolution image predictions across different specimen types or structures of interest, and outperforms classical physics-driven methods in terms of imaging speed, reconstruction quality and environmental robustness, while avoiding intricate and specialized algorithmic procedures. These collective advantages make eDL-cSIM a promising tool for fast and robust live-cell super-resolution microscopy with significantly reduced phototoxicity and photobleaching.

HTML全文

参考文献(77)

[1]	Rust MJ, Bates M, Zhuang X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat Methods. 2006;3(10):793–6.
[2]	Hess ST, Girirajan TP, Mason MD. Ultra-high resolution imaging by fluorescence photoactivation localization microscopy. Biophys J. 2006;91(11):4258–72.
[3]	Gustafsson MG. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy. J Microsc. 2000;198(2):82–7.
[4]	Klar TA, Jakobs S, Dyba M, Egner A, Hell SW. Fluorescence microscopy with diffraction resolution barrier broken by stimulated emission. Proc Natl Acad Sci. 2000;97(15):8206–10.
[5]	Zhao K, Xu X, Ren W, Jin D, Xi P. Two-photon MINFLUX with doubled localization precision. eLight. 2022;2(1):5.
[6]	Li D, Shao L, Chen BC, Zhang X, Zhang M, Moses B, et al. Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics. Science. 2015;349(6251):aab3500.
[7]	Shao L, Kner P, Rego EH, Gustafsson MG. Super-resolution 3D microscopy of live whole cells using structured illumination. Nat Methods. 2011;8(12):1044–6.
[8]	Zhanghao K, Chen X, Liu W, Li M, Liu Y, Wang Y, et al. Super-resolution imaging of fluorescent dipoles via polarized structured illumination microscopy. Nat Commun. 2019;10(1):4694.
[9]	Johnson KA, Hagen GM. Artifact-free whole-slide imaging with structured illumination microscopy and Bayesian image reconstruction. Gigascience. 2020;9(4):giaa035.
[10]	Wang Z, Zhao T, Hao H, Cai Y, Feng K, Yun X, et al. High-speed image reconstruction for optically sectioned, super-resolution structured illumination microscopy. Adv Photon. 2022;4(2):026003.
[11]	Huang G, Liu Y, Wang D, Zhu Y, Wen S, Ruan J, et al. Upconversion nanoparticles for super-resolution quantification of single small extracellular vesicles. eLight. 2022;2(1):20.
[12]	Chen X, Zhong S, Hou Y, Cao R, Wang W, Li D, et al. Superresolution structured illumination microscopy reconstruction algorithms: a review. Light Sci Appl. 2023;12(1):172.
[13]	Lai-Tim Y, Mugnier LM, Orieux F, Baena-Gallé R, Paques M, Meimon S. Jointly super-resolved and optically sectioned Bayesian reconstruction method for structured illumination microscopy. Opt Express. 2019;27(23):33251–67.
[14]	Müller M, Mönkemöller V, Hennig S, Hübner W, Huser T. Open-source image reconstruction of super-resolution structured illumination microscopy data in ImageJ. Nat Commun. 2016;7(1):10980.
[15]	Saxena M, Eluru G, Gorthi SS. Structured illumination microscopy. Adv Opt Photon. 2015;7(2):241–75.
[16]	Wang Z, Zhao T, Cai Y, Zhang J, Hao H, Liang Y, et al. Rapid, artifact-reduced, image reconstruction for super-resolution structured illumination microscopy. Innovation. 2023;4(3):100425.
[17]	Descloux A, Müller M, Navikas V, Markwirth A, Van den Eynde R, Lukes T, et al. High-speed multiplane structured illumination microscopy of living cells using an image-splitting prism. Nanophotonics. 2020;9(1):143–8.
[18]	Lyu J, Qian J, Xu K, Huang Y, Zuo C. Motion-resistant structured illumination microscopy based on principal component analysis. Opt Lett. 2023;48(1):175–8.
[19]	Turcotte R, Liang Y, Tanimoto M, Zhang Q, Li Z, Koyama M, et al. Dynamic super-resolution structured illumination imaging in the living brain. Proc Natl Acad Sci. 2019;116(19):9586–91.
[20]	Gustafsson MG, Shao L, Carlton PM, Wang CR, Golubovskaya IN, Cande WZ, et al. Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination. Biophys J. 2008;94(12):4957–70.
[21]	Wicker K, Mandula O, Best G, Fiolka R, Heintzmann R. Phase optimisation for structured illumination microscopy. Opt Express. 2013;21(2):2032–49.
[22]	Wicker K. Non-iterative determination of pattern phase in structured illumination microscopy using auto-correlations in Fourier space. Opt Express. 2013;21(21):24692–701.
[23]	Dan D, Lei M, Yao B, Wang W, Winterhalder M, Zumbusch A, et al. DMD-based LED-illumination super-resolution and optical sectioning microscopy. Sci Rep. 2013;3(1):1116.
[24]	Ball G, Demmerle J, Kaufmann R, Davis I, Dobbie IM, Schermelleh L. SIMcheck: a toolbox for successful super-resolution structured illumination microscopy. Sci Rep. 2015;5(1):15915.
[25]	Qian J, Xu K, Feng S, Liu Y, Ma H, Chen Q, et al. Fast-Structured Illumination Microscopy Based on Dichotomy-Correlation Parameter Estimation (dCOR-SIM). ACS Photonics. 2024;11(5):1887–92.
[26]	Huang X, Fan J, Li L, Liu H, Wu R, Wu Y, et al. Fast, long-term, super-resolution imaging with Hessian structured illumination microscopy. Nat Biotechnol. 2018;36(5):451–9.
[27]	Zhao W, Zhao S, Li L, Huang X, Xing S, Zhang Y, et al. Sparse deconvolution improves the resolution of live-cell super-resolution fluorescence microscopy. Nat Biotechnol. 2022;40(4):606–17.
[28]	Chu K, McMillan PJ, Smith ZJ, Yin J, Atkins J, Goodwin P, et al. Image reconstruction for structured-illumination microscopy with low signal level. Opt Express. 2014;22(7):8687–702.
[29]	Wen G, Li S, Wang L, Chen X, Sun Z, Liang Y, et al. High-fidelity structured illumination microscopy by point-spread-function engineering. Light Sci Appl. 2021;10(1):70.
[30]	Soubies E, Nogueron A, Pelletier F, Mangeat T, Leterrier C, Unser M, et al. Surpassing light inhomogeneities in structured-illumination microscopy with FlexSIM. J Microsc. 2024;296(1):94–106.
[31]	Qian J, Cao Y, Bi Y, Wu H, Liu Y, Chen Q, et al. Structured illumination microscopy based on principal component analysis. eLight. 2023;3(1):4.
[32]	Jin L, Liu B, Zhao F, Hahn S, Dong B, Song R, et al. Deep learning enables structured illumination microscopy with low light levels and enhanced speed. Nat Commun. 2020;11(1):1934.
[33]	Qian J, Cao Y, Xu K, Bi Y, Xia W, Chen Q, et al. Robust frame-reduced structured illumination microscopy with accelerated correlation-enabled parameter estimation. Appl Phys Lett. 2022;121(15):153701.
[34]	Ströhl F, Kaminski CF. Speed limits of structured illumination microscopy. Opt Lett. 2017;42(13):2511–4.
[35]	Orieux F, Sepulveda E, Loriette V, Dubertret B, Olivo-Marin JC. Bayesian estimation for optimized structured illumination microscopy. IEEE Trans Image Process. 2011;21(2):601–14.
[36]	Lal A, Shan C, Zhao K, Liu W, Huang X, Zong W, et al. A frequency domain SIM reconstruction algorithm using reduced number of images. IEEE Trans Image Process. 2018;27(9):4555–70.
[37]	Dong S, Liao J, Guo K, Bian L, Suo J, Zheng G. Resolution doubling with a reduced number of image acquisitions. Biomed Opt Express. 2015;6(8):2946–52.
[38]	Ling C, Zhang C, Wang M, Meng F, Du L, Yuan X. Fast structured illumination microscopy via deep learning. Photon Res. 2020;8(8):1350–9.
[39]	Wang H, Rivenson Y, Jin Y, Wei Z, Gao R, Günaydın H, et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat Methods. 2019;16(1):103–10.
[40]	Shah ZH, Müller M, Wang TC, Scheidig PM, Schneider A, Schüttpelz M, et al. Deep-learning based denoising and reconstruction of super-resolution structured illumination microscopy images. Photon Res. 2021;9(5):B168–81.
[41]	Christensen CN, Ward EN, Lu M, Lio P, Kaminski CF. ML-SIM: universal reconstruction of structured illumination microscopy images using transfer learning. Biomed Opt Express. 2021;12(5):2720–33.
[42]	Qiao C, Li D, Guo Y, Liu C, Jiang T, Dai Q, et al. Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nat Methods. 2021;18(2):194–202.
[43]	Qiao C, Li D, Liu Y, Zhang S, Liu K, Liu C, et al. Rationalized deep learning super-resolution microscopy for sustained live imaging of rapid subcellular processes. Nat Biotechnol. 2023;41(3):367–77.
[44]	Qiao C, Zeng Y, Meng Q, Chen X, Chen H, Jiang T, et al. Zero-shot learning enables instant denoising and super-resolution in optical fluorescence microscopy. Nat Commun. 2024;15(1):4180.
[45]	Chen X, Qiao C, Jiang T, Liu J, Meng Q, Zeng Y, et al. Self-supervised denoising for multimodal structured illumination microscopy enables long-term super-resolution live-cell imaging. PhotoniX. 2024;5(1):4.
[46]	Wu H, Li Y, Sun Y, Yin L, Sun W, Ye Z, et al. Single-frame structured illumination microscopy for fast live-cell imaging. APL Photon. 2024;9(3):036102.
[47]	Saurabh A, Brown PT, Bryan JS 4th, et al. Approaching maximum resolution in structured illumination microscopy via accurate noise modeling. npj Imaging. 2025;3(5).
[48]	Zhang, Qinnan, et al. Deep learning-based single-shot structured illumination microscopy. Opt Lasers Eng. 2022;155:107066.
[49]	Cheng X, Li J, Dai Q, Fu Z, Yang J. Fast and lightweight network for single frame structured illumination microscopy super-resolution. IEEE Trans Instrum Meas. 2022;71:1–11.
[50]	Ganaie MA, Hu M, Malik AK, Tanveer M, Suganthan PN. Ensemble deep learning: A review. Eng Appl Artif Intell. 2022;115:105151.
[51]	Rahman MSS, Li J, Mengu D, Rivenson Y, Ozcan A. Ensemble learning of diffractive optical networks. Light Sci Appl. 2021;10(1):14.
[52]	Feng S, Xiao Y, Yin W, Hu Y, Li Y, Zuo C, et al. Fringe-pattern analysis with ensemble deep learning. Adv Photon Nexus. 2023;2(3):036010.
[53]	Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. Switzerland: Springer; 2015. pp. 234–41.
[54]	He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: Institute of Electrical and Electronics Engineers (IEEE); 2016. pp. 770–8.
[55]	Chen J, Lu Y, Yu Q, Luo X, Adeli E, Wang Y, et al. Transunet: Transformers make strong encoders for medical image segmentation. 2021. arXiv preprint arXiv:2102.04306. https://arxiv.org/abs/2102.04306.
[56]	Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O. 3D U-Net: learning dense volumetric segmentation from sparse annotation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19. Switzerland: Springer; 2016. pp. 424–32.
[57]	Li Z, Sun J, Fan Y, Jin Y, Shen Q, Trusiak M, et al. Deep learning assisted variational Hilbert quantitative phase imaging. Opto-Electron Sci. 2023;2:220023.
[58]	Oktay O, Schlemper J, Folgoc LL, Lee M, Heinrich M, Misawa K, et al. Attention u-net: Learning where to look for the pancreas. 2018. arXiv preprint arXiv:1804.03999. https://arxiv.org/abs/1804.03999.
[59]	Wang Z, Zhu L, Zhang H, Li G, Yi C, Li Y, et al. Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning. Nat Methods. 2021;18(5):551–6.
[60]	Li X, Hu X, Chen X, Fan J, Zhao Z, Wu J, et al. Spatial redundancy transformer for self-supervised fluorescence image denoising. Nat Comput Sci. 2023;3(12):1067–80.
[61]	Chen R, Tang X, Zhao Y, Shen Z, Zhang M, Shen Y, et al. Single-frame deep-learning super-resolution microscopy for intracellular dynamics imaging. Nat Commun. 2023;14(1):2854.
[62]	Tai Y, Yang J, Liu X. Image super-resolution via deep recursive residual network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. pp. 3147–55.
[63]	Fan J, Tang X, Bi X, Miao Y, Li W, Xiao B, et al. High-Fidelity Reconstruction of Structured Illumination Microscopy by an Amplitude-Phase Channel Attention Network With Multitemporal Information. IEEE Trans Instrum Meas. 2023;72:1–13.
[64]	Choi H, Lee J, Yang J. N-gram in swin transformers for efficient lightweight image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: Institute of Electrical and Electronics Engineers (IEEE); 2023. pp. 2071–81.
[65]	Chen J, Kao Sh, He H, Zhuo W, Wen S, Lee CH, et al. Run, don’t walk: chasing higher FLOPS for faster neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: Institute of Electrical and Electronics Engineers (IEEE); 2023. pp. 12021–31.
[66]	Cao H, Wang Y, Chen J, Jiang D, Zhang X, Tian Q, et al. Swin-unet: Unet-like pure transformer for medical image segmentation. In: European Conference on Computer Vision. Switzerland: Springer; 2022. pp. 205–18.
[67]	Shi W, Caballero J, Huszár F, Totz J, Aitken AP, Bishop R, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. Piscataway: Institute of Electrical and Electronics Engineers (IEEE); 2016. pp. 1874–83.
[68]	Zuo C, Qian J, Feng S, Yin W, Li Y, Fan P, et al. Deep learning in optical metrology: a review. Light Sci Appl. 2022;11(1):1–54.
[69]	Li YJ, Cao YL, Feng JX, Qi Y, Meng S, Yang JF, et al. Structural insights of human mitofusin-2 into mitochondrial fusion and CMT2A onset. Nat Commun. 2019;10(1):4914.
[70]	LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521(7553):436–44.
[71]	Weiss K, Khoshgoftaar TM, Wang D. A survey of transfer learning. J Big Data. 2016;3:1–40.
[72]	Shorten C, Khoshgoftaar TM. A survey on image data augmentation for deep learning. J Big Data. 2019;6(1):1–48.
[73]	Sun Y, Zhu H, Yin L, Wu H, Cai M, Sun W, et al. Fluorescence interference structured illumination microscopy for 3D morphology imaging with high axial resolution. Adv Photon. 2023;5(5):056007.
[74]	Hendrycks D, Gimpel K. Gaussian error linear units (gelus). 2016. arXiv preprint arXiv:1606.08415. https://arxiv.org/abs/1606.08415.
[75]	Zhang R, Isola P, Efros AA, Shechtman E, Wang O. The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: Institute of Electrical and Electronics Engineers (IEEE); 2018. pp. 586–95.
[76]	Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. 2014. arXiv preprint arXiv:1409.1556. https://arxiv.org/abs/1409.1556.
[77]	Kingma DP, Ba J. Adam: A method for stochastic optimization. 2014. arXiv preprint arXiv:1412.6980. https://arxiv.org/abs/1412.6980.

施引文献

资源附件(0)

点击查看大图

计量

文章访问数: 31
HTML全文浏览量: 0
PDF下载量: 0
被引次数: 0

姓名
邮箱
手机号码
标题
留言内容
验证码

留言板

doi: 10.1186/s43074-025-00171-w

计量

Ensemble deep learning-enabled single-shot composite structured illumination microscopy (eDL-cSIM)

计量

目录

PhotoniX

联系我们

友情链接

留言板

doi: 10.1186/s43074-025-00171-w

计量

出版历程

Ensemble deep learning-enabled single-shot composite structured illumination microscopy (eDL-cSIM)

计量

出版历程

目录

PhotoniX

联系我们

友情链接