Skip to main content
SpringerLink
Log in

An Ensemble Deep Learning Approach Combining Phenotypic Data and fMRI for ADHD Diagnosis

  • Published:
Journal of Signal Processing Systems Aims and scope Submit manuscript

Abstract

As a common neurological disorder in early childhood and adolescence, an efficient and accurate diagnosis of Attention-Defect/Hyperactivity Disorder (ADHD) has always been one of the important goals in the field of psychiatry. However, most of the current diagnostic methods are based on a single table or fMRI images, which may result in the loss of much complementary information useful for diagnosis. Therefore, this paper presents a strategy for multimodal data fusion and an ensemble learning model (Trans3D-ensemble) to classify the ADHDs and typicals. The base classifiers of Trans3D-ensemble include two parts, Trans3D (Transformer for 3D images) to extract spatio-temporal features from fMRI images and random forest to extract clinical features from phenotypic data. Specifically, Trans3D utilizes 3D-CNN to capture volumetric spatial information and convert the 3D images into patch embeddings. Meanwhile, temporal pooling operation fuses the images tokens output by Transformer encoders across time and obtains the representative features of fMRI samples. Existing methods basically only build the above independent models, and few consider the effective combination of the those for diagnosis and treatment. Looking into this issue, we use stacking, one of ensemble learning methods, on multimodal data by merging the outputs from the two base classifiers. The proposed method reaches excellent results compared to most methods based on single modality and gets the accuracy of 74.5% on ADHD-200 data set, demonstrating the potential of the combination of phenotypic data and fMRI for ADHD diagnosis. Our strategy of multimodal data fusion provides a novel comprehensive diagnosis mode and the above results show that the proposed Trans3D-ensemble can effectively improve the auxiliary diagnosis of ADHD.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5

Similar content being viewed by others

Data Availability

The authors are very grateful to the ADHD-200 competition party for collecting and disclosing such valuable data without any financial return. Meanwhile, thanks are also given to [20, 21] for publishing the ADHD data preprocessing and providing NIAK pipeline, which greatly reduces the researchers’ work on fMRI data preprocessing.

Notes

  1. https://www.nitrc.org/frs/?group_id=383

  2. https://www.nitrc.org/plugins/mwiki/index.php/nitrc:ADHD-200_Dataset

References

  1. Bruer, J. T. (1998). Brain science, brain fiction. Educational Leadership, 56, 14–18.

    Google Scholar 

  2. DeYoe, E.A., Bandettini, P., Neitz, J., Miller, D., & Winans, P. (1994). Functional magnetic resonance imaging (fmri) of the human brain. Journal of Neuroscience Methods, 54, 171–187. Imaging Techniques in Neurobiology. https://doi.org/10.1016/0165-0270(94)90191-0

  3. Qiu, M., Jia, Z., Xue, C., Shao, Z., & Sha, E. (2007). Voltage assignment with guaranteed probability satisfying timing constraint for real-time multiproceesor DSP. The Journal of VLSI Signal Processing Systems for Signal, Image, and Video Technology.

  4. Qiu, M., Khisamutdinov, E., et al. (2013). RNA nanotechnology for computer design and in vivo computation. Philosophical Transactions of the Royal Society A.

  5. Qiu, M., Yang, L., Shao, Z., & Sha, E. (2009). Dynamic and leakage energy minimization with soft real-time loop scheduling and voltage assignment. IEEE TVLSI, 18, 501–504.

    Google Scholar 

  6. Gao, J., Li, P., Chen, Z., & Zhang, J. (2020). A survey on deep learning for multimodal data fusion. Neural Computation, 32, 829–864.

    Article  MathSciNet  MATH  Google Scholar 

  7. Gai, K., Qiu, M., & Elnagdy, S. (2016). A novel secure big data cyber incident analytics framework for cloud-based cybersecurity insurance. In IEEE BigDataSecurity Conference.

  8. Qiu, H., Qiu, M., Liu, M., & Memmi, G. (2020b). Secure health data sharing for medical cyber-physical systems for the healthcare 4.0. IEEE Journal of Biomedical and Health Informatics, 24, 2499–2505.

  9. Qiu, M., Gai, K., & Xiong, Z. (2018). Privacy-preserving wireless communications using bipartite matching in social big data. FGCS, 87, 772–781.

    Article  Google Scholar 

  10. Lahat, D., Adali, T., & Jutten, C. (2015). Multimodal data fusion: an overview of methods, challenges, and prospects. Proceedings of the IEEE, 103, 1449–1477. https://doi.org/10.1109/JPROC.2015.2460697

    Article  Google Scholar 

  11. Noroozi, O., Pijeira-Díaz, H. J., Sobocinski, M., Dindar, M., Järvelä, S., & Kirschner, P. A. (2020). Multimodal data indicators for capturing cognitive, motivational, and emotional learning processes: a systematic literature review. Education and Information Technologies, 25, 5499–5547.

    Article  Google Scholar 

  12. Shankar, S. K., Prieto, L. P., Rodríguez-Triana, M. J., & Ruiz-Calleja, A. (2018). A review of multimodal learning analytics architectures. In 2018 IEEE 18th International Conference on Advanced Learning Technologies (ICALT), pp. 212–214. https://doi.org/10.1109/ICALT.2018.00057

  13. Hu, F., Lakdawala, S., et al. (2009). Low-power, intelligent sensor hardware interface for medical data preprocessing. IEEE Transactions on Information Technology in Biomedicine, 13, 656–663.

  14. Li, Y., Gai, K., et al. (2016). Intercrossed access controls for secure financial services on multimedia big data in cloud systems. ACM Transactions on Multimedia Computing, Communications, and Applications.

  15. Qiu, M., Chen, Z., Ming, Z., Qin, X., & Niu, J. (2014). Energy-aware data allocation with hybrid memory for mobile cloud systems. IEEE Systems Journal, 11, 813–822.

    Article  Google Scholar 

  16. Cai, Q., Wang, H., Li, Z., & Liu, X. (2019). A survey on multimodal data-driven smart healthcare systems: Approaches and applications. IEEE Access, 7, 133583–133599. https://doi.org/10.1109/ACCESS.2019.2941419

    Article  Google Scholar 

  17. Barkley, R. A., & Murphy, K. R. (2006). Attention-deficit hyperactivity disorder: a clinical workbook. Guilford Press.

  18. Polanczyk, G., De Lima, M. S., Horta, B. L., Biederman, J., & Rohde, L. A. (2007). The worldwide prevalence of ADHD: a systematic review and metaregression analysis. American Journal of Psychiatry, 164, 942–948.

    Article  Google Scholar 

  19. Milham, M., Fair, D., Mennes, M., & Mostofsky, S. (2012). The ADHD-200 consortium: a model to advance the translational potential of neuroimaging in clinical neuroscience. Frontiers in Systems Neuroscience, 6. https://doi.org/10.3389/fnsys.2012.00062

  20. Bellec, P., Chu, C., Chouinard-Decorte, F., Benhajali, Y., Margulies, D. S., & Craddock, R. C. (2017). The neuro bureau ADHD-200 preprocessed repository. NeuroImage, 144, 275–286. Data Sharing Part II. https://doi.org/10.1016/j.neuroimage.2016.06.034

  21. Lavoie-Courchesne, S., Rioux, P., Chouinard-Decorte, F., Sherif, T., Rousseau, M. -E., Das, S., Adalat, R., Doyon, J., Craddock, C., Margulies, D., et al. (2012). Integration of a neuroimaging processing pipeline into a pan-canadian computing grid. In Journal of Physics: Conference Series (vol. 341). IOP Publishing, p. 012032.

  22. Chang, C. -W., Ho, C. -C., & Chen, J. -H. (2012). ADHD classification by a texture analysis of anatomical brain MRI data. Frontiers in Systems Neuroscience, 6. https://doi.org/10.3389/fnsys.2012.00066

  23. Brown, M., Sidhu, G., Greiner, R., Asgarian, N., Bastani, M., Silverstone, P., Greenshaw, A., & Dursun, S. (2012). ADHD-200 global competition: Diagnosing ADHD using personal characteristic data can outperform resting state fmri measurements. Frontiers in Systems Neuroscience, 6. https://doi.org/10.3389/fnsys.2012.00069

  24. Casado-Aranda, L. A., Martinez-Fiestas, M., & Sanchez-Fernandez, J. (2018). Neural effects of environmental advertising: an fmri analysis of voice age and temporal framing. Journal of Environmental Management, 206, 664–675.

    Article  Google Scholar 

  25. Czerniak, S. M. (2014). ADHD-200 patient characterization and classification using resting state networks: a dissertation.

  26. Kuang, D., Guo, X., An, X., Zhao, Y., & He, L. (2014). Discrimination of ADHD based on fmri data with deep belief network. In International Conference on Intelligent Computing. Springer, pp. 225–232.

  27. Guo, X., An, X., Kuang, D., Zhao, Y., & He, L. (2014). ADHD-200 classification based on social network method. In International Conference on Intelligent Computing. Springer, pp. 233–240.

  28. Mao, Z., Su, Y., Xu, G., Wang, X., Huang, Y., Yue, W., Sun, L., & Xiong, N. (2019). Spatio-temporal deep learning method for ADHD fmri classification. Information Sciences, 499, 1–11. https://doi.org/10.1016/j.ins.2019.05.043

    Article  Google Scholar 

  29. Dou, C., Zhang, S., Wang, H., Sun, L., Huang, Y., & Yue, W. (2020). ADHD fmri short-time analysis method for edge computing based on multi-instance learning. Journal of Systems Architecture, 111, 101834. https://doi.org/10.1016/j.sysarc.2020.101834

    Article  Google Scholar 

  30. Itani, S., Lecron, F., & Fortemps, P. (2018). A multi-level classification framework for multi-site medical data: Application to the ADHD-200 collection. Expert Systems with Applications, 91, 36–45. https://doi.org/10.1016/j.eswa.2017.08.044

    Article  Google Scholar 

  31. Gai, K., Du, Z., et al. (2015). Efficiency-aware workload optimizations of heterogeneous cloud computing for capacity planning in financial industry. In IEEE 2nd CSCloud.

  32. Li, J., Ming, Z., et al. (2011). Resource allocation robustness in multi-core embedded systems with inaccurate information. Journal of Systems Architecture, 57, 840–849.

    Article  Google Scholar 

  33. Li, J., Qiu, M., et al. (2013). Thermal-aware task scheduling in 3D chip multiprocessor with real-time constrained workloads. ACM Transactions on Embedded Computing Systems (TECS), 12, 1–22.

    Article  Google Scholar 

  34. Qiu, H., Dong, T., et al. (2020). Adversarial attacks against network intrusion detection in IoT systems. IEEE Internet of Things Journal, 8, 10327–10335.

    Article  Google Scholar 

  35. Qiu, H., Zheng, Q., et al. (2020). Deep residual learning-based enhanced JPEG compression in the internet of things. IEEE Transactions on Industrial Informatics, 17, 2124–2133.

    Google Scholar 

  36. Qiu, H., Zheng, Q., et al. (2020d). Topological graph convolutional network-based urban traffic flow and density prediction. IEEE Transactions on ITS.

  37. Chen, L., Wang, S., Fan, W., Sun, J., & Naoi, S. (2015). Beyond human recognition: a CNN-based framework for handwritten character recognition. In 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), pp. 695–699. https://doi.org/10.1109/ACPR.2015.7486592

  38. Khan, S., Rahmani, H., Shah, S. A. A., & Bennamoun, M. (2018). A guide to convolutional neural networks for computer vision. Synthesis Lectures on Computer Vision, 8, 1–207.

    Article  Google Scholar 

  39. Kwon, M.-C., Park, G., & Choi, S. (2018). Smartwatch user interface implementation using cnn-based gesture pattern recognition. Sensors, 18(9). https://doi.org/10.3390/s18092997

  40. Sharif Razavian, A., Azizpour, H., Sullivan, J., & Carlsson, S. (2014). Cnn features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.

  41. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

  42. Jung, H., Choi, M. -K., Jung, J., Lee, J. -H., Kwon, S., & Young Jung, W. (2017). Resnet-based vehicle classification and localization in traffic surveillance systems. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.

  43. Lin, H., & Jegelka, S. (2018). Resnet with one-neuron hidden layers is a universal approximator. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, & R. Garnett (Eds.), Advances in Neural Information Processing Systems (vol. 31). Curran Associates, Inc.

  44. Zhao, J., Lei, W., Li, Z., Zhao, D., Han, M., & Hou, X. (2022). Detection of crowdedness in bus compartments based on resnet algorithm and video images. Multimedia Tools and Applications, 81, 4753–4780.

    Article  Google Scholar 

  45. Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao, A., Xu, C., Xu, Y., Yang, Z., Zhang, Y., & Tao, D. (2022). A survey on vision transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–1. https://doi.org/10.1109/TPAMI.2022.3152247

  46. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. In arXiv.

  47. Kim, T. H., Sajjadi, M. S. M., Hirsch, M., & Scholkopf, B. (2018). Spatio-temporal transformer network for video restoration. In Proceedings of the European Conference on Computer Vision (ECCV).

  48. Cao, C., Zhang, Y., Wu, Y., Lu, H., & Cheng, J. (2017). Egocentric gesture recognition using recurrent 3D convolutional neural networks with spatiotemporal transformer modules. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).

  49. Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., & Schmid, C. (2021). Vivit: a video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6836–6846.

  50. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021). Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10012–10022.

  51. Xu, Y., Zhang, Q., Zhang, J., & Tao, D. (2021). Vitae: Vision transformer advanced by exploring intrinsic inductive bias. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, & J. W. Vaughan (Eds.), Advances in Neural Information Processing Systems (vol. 34, pp. 28522–28535). Curran Associates, Inc.

  52. Kim, W., Son, B., & Kim, I. (2021). Vilt: Vision-and-language transformer without convolution or region supervision. In M. Meila, & T. Zhang (Eds.), Proceedings of the 38th International Conference on Machine Learning (pp. 5583–5594). PMLR volume 139 of Proceedings of Machine Learning Research.

  53. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

  54. Fu, X., & Zhang, L. (2021). Spatio-temporal feature fusion for real-time prediction of tbm operating parameters: a deep learning approach. Automation in Construction, 132, 103937. https://doi.org/10.1016/j.autcon.2021.103937

    Article  Google Scholar 

  55. Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., & Paluri, M. (2017). A closer look at spatiotemporal convolutions for action recognition. CoRR. arXiv:1711.11248

  56. Korkmaz, D. (2021). Solarnet: a hybrid reliable model based on convolutional neural network and variational mode decomposition for hourly photovoltaic power forecasting. Applied Energy, 300.

  57. Lin, M., Chen, Q., & Yan, S. (2013). Network in network. Computer Science.

  58. Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). Smote: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16, 321–357.

    Article  MATH  Google Scholar 

  59. Japkowicz, N., & Stephen, S. (2002). The class imbalance problem: a systematic study. Intelligent Data Analysis, 6, 429–449.

    Article  MATH  Google Scholar 

  60. Rangarajan, B., Subramaian, K., & Suresh, S. (2015). Importance of phenotypic information in ADHD diagnosis. In 2015 International Conference on Cognitive Computing and Information Processing (CCIP).

  61. Yin, C. H., He, B. L., & Hao, A. J. (2016). Discrimination of ADHD children based on deep bayesian network. In IET International Conference on Biomedical Image & Signal Processing.

  62. Zou, L., Zheng, J., Miao, C., Mckeown, M. J., & Wang, Z. J. (2017). 3d cnn based automatic diagnosis of attention deficit hyperactivity disorder using functional and structural mri (pp. 1–1). PP: IEEE Access.

    Google Scholar 

  63. Zhang, J., Zhou, L., & Wang, L.  (2017). Subject-adaptive integration of multiple sice brain networks with different sparsity. Pattern Recognition: The Journal of the Pattern Recognition Society, 63, 642–652.

  64. Wang, X., Yao, L., Rekik, I., & Zhang, Y. (2022). Contrastive graph learning for population-based fmri classification.

  65. Riaz, A., Alonso, E., & Slabaugh, G. (2016). Phenotypic integrated framework for classification of ADHD using fmri. In A. Campilho & F. Karray (Eds.), Image Analysis and Recognition (pp. 217–225). Cham: Springer International Publishing.

    Chapter  Google Scholar 

Download references

Funding

This study was funded by the Scientific and technological project in Sichuan Province (No.2020YFSY0042), National Key RD Program of China (No. 2021YFF1201100, 2020AAA0109400), Project 2020BD025 supported by PKU-Baidu Fund.

Author information

Authors and Affiliations

  1. National Engineering Research Center for Software Engineering, Peking University, Beijing, 100871, China

    Yuanze Qin, Yiwei Lou & Yu Huang

  2. School of Software & Microelectronics, Peking University, Beijing, 100871, China

    Yuanze Qin

  3. Key Laboratory of High Confidence Software Technologies, School of Computer Science, Peking University, Beijing, 100871, China

    Yiwei Lou

  4. Department of Spine Surgery, Hospital of Chengdu University of Traditional Chinese Medicine, Chengdu, 610041, China

    Rigao Chen

  5. Institute of Mental Health, the Sixth Hospital, Peking University, Beijing, 100191, China

    Weihua Yue

  6. Key Laboratory of Mental Health, Ministry of Health & National Clinical Research Center for Mental Disorders, Peking University, Beijing, 100191, China

    Weihua Yue

Authors
  1. Yuanze Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yiwei Lou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yu Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rigao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Weihua Yue
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yu Huang.

Ethics declarations

Conflict of Interest

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the Topical Collection on Big Data Security Track

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qin, Y., Lou, Y., Huang, Y. et al. An Ensemble Deep Learning Approach Combining Phenotypic Data and fMRI for ADHD Diagnosis. J Sign Process Syst 94, 1269–1281 (2022). https://doi.org/10.1007/s11265-022-01812-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11265-022-01812-0

Keywords

Search

Navigation