Comparative Analysis of Optimization Techniques for Enhancing Machine Learning Model Performance
DOI:
https://doi.org/10.64229/x3xjnj96Keywords:
Optimization Technique, Hyperparameter, Feature Selection, Dimensionality Reduction, Gridsearchcv, Ensemble Methods, Regularization, Sustainable Development Goals (SDGs)Abstract
Optimization methods are crucial in machine learning, and significantly improve predictive model performance. In the age of highly complex machine learning models, scalable optimization methods that can enhance optimal accuracy, computational speed and improve overall model efficiency, have been in great demand. In this paper, comparative study is based on the selection of the optimization techniques, which are vital in multiple phases of the machine learning workflow. The paper is mainly concerned with testing preprocessing methods, feature selection, dimension reduction, regularization and ensemble learning for supervised learning. Data Rescaling methods are compared for the potential optimization of model convergence and performance. In addition, feature selection methods such as SelectKBest and RFE are investigated for the integrity of the formant space with reduced dimensions. This paper also investigates the application of PCA as dimensionality reduction technique and presents its capability of preserving the important data variance. This library also teaches us about regularization techniques like L1 (Lasso) and hyperparameter tuning using GridSearchCV, and how they help in preventing the model from being overfitted and models performance optimisation. Last but not least, ensemble learning with approaches such as Voting Classifiers is explored to show how multiple models can be combined to obtain better predictive power and stability. Thus, the contrasting study here aims to offer useful guidance for academicians and practitioners to choose the best optimization methods for the targeted ML applications.
References
[1]G. Chugh, S. Kumar, and N. Singh, “Survey on machine learning and deep learning applications in breast cancer diagnosis,” Cognitive Computation, vol. 13, no. 6, pp. 1451–1470, 2021.
[2]Kushwaha, S. (2023). A futuristic perspective on artificial intelligence. In Proceedings of the IEEE OPJU International Technology Conference on Emerging Technologies For Sustainable Development (pp. 1-6). O.P. Jindal University, Raigarh, Chhattisgarh, India.
[3]Ali, L., Rahman, A., Khan, A., Zhou, M., Javeed, A., & Khan, J. A. (2019). An automated diagnostic system for heart disease prediction based on χ² statistical model and optimally configured deep neural network. IEEE Access, 7, 34938–34945.
[4]Gkontzis, A. F., Kotsiantis, S., Panagiotakopoulos, C. T., & Verykios, V. S. (2022). A predictive analytics framework as a countermeasure for attrition of students. Interactive Learning Environments, 30(6), 1028–1043.
[5]Duncker, D.(2021). Smart wearables for cardiac monitoring—real-world use beyond atrial fibrillation. Sensors, 21(7), 2539.
[6]Isakadze, N., & Martin, S. S. (2020). How useful is the smartwatch ECG? Trends in Cardiovascular Medicine, 30(7), 442–448.
[7]Kushwaha, S. (2023). An effective adaptive fuzzy filter for speckle noise reduction. Multimedia Tools and Applications, 2023, 1-16. Springer.
[8]Martins, M. V., Tolledo, D., Machado, J., Baptista, L. M., & Realinho, V. (2021). Early prediction of student’s performance in higher education: A case study. In A. Rocha, H. Adeli, G. Dzemyda, F. Moreira, & A. M. R. Correia (Eds.), Trends and applications in information systems and technologies (Vol. 9, pp. 166–175).
[9]Bijari, S., Sayfollahi, S., Mardokh-Rouhani, S., Bijari, S., Moradian, S., Zahiri, Z. (2024). Radiomics and deep features: Robust classification of brain hemorrhages and reproducibility analysis using a 3D autoencoder neural network. Bioengineering, 11, 643.
[10]Thammasiri, D., Delen, D., Meesad, P., & Kasap, N. (2014). A critical assessment of imbalanced class distribution problem: The case of predicting freshmen student attrition. Expert Systems with Applications, 41(2), 321–330.
[11]Tembhurne, J. V., Hebbar, N., Patil, H. Y., & Diwan, T. (2023). Skin cancer detection using ensemble of machine learning and deep learning techniques. Multimedia Tools and Applications, 16, 1–24.
[12]Wang, C., Zhao, H., & Zhang, H. (2020). Chinese college students have higher anxiety in the new semester of online learning during COVID-19: A machine learning approach. Frontiers in Psychology, 11, 587413.
[13]Li, Y., Li, C., Li, X., Wang, K., Rahaman, M. M., Sun, C., Chen, H., Wu, X., Zhang, H., & Wang, Q. (2022). A comprehensive review of Markov random field and conditional random field approaches in pathology image analysis. Archives of Computational Methods in Engineering, 29(1), 609–639.
[14]Islam, A., Belhaouari, S. B., Rehman, A. U., & Bensmail, H. (2022). KNNOR: An oversampling technique for imbalanced datasets. Applied Soft Computing, 115, 108288.
[15]Bulbul, H. I., Usta, N., & Yildiz, M. (2017). Classification of ECG arrhythmia with machine learning techniques. In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA) (pp. 546–549). IEEE.
[16]Hamim, T., Benabbou, F., & Sael, N. (2022). Student profile modeling using boosting algorithms. International Journal of Web-Based Learning and Teaching Technologies, 17(5), 1–13.
[17]Sharma, P., & Gupta, D. V. (2018). Disease classification from ECG signal using R-peak analysis with artificial intelligence. International Journal of Signal Processing, Image Processing and Pattern Recognition, 11(3), 29–40.
[18]Ullah, A., Anwar, S. M., Bilal, M., & Mehmood, R. M. (2020). Classification of arrhythmia by using deep learning with 2-D ECG spectral image representation. Remote Sensing, 12(10), 1685.
[19]enpipat, W., & Akkarajitsakul, K. (2020). Student dropout prediction: A KMUTT case study. In 2020 1st International Conference on Big Data Analytics and Practices (IBDAP) (pp. 1–5). IEEE.
[20]Khan, M. U., Aziz, S., Naqvi, S. Z. H., & Rehman, A. (2020). Classification of coronary artery diseases using electrocardiogram signals. In 2020 International Conference on Emerging Trends in Smart Technologies (ICETST). IEEE.
[21]Hew, K. F., Hu, X., Qiao, C., & Tang, Y. (2020). What predicts student satisfaction with MOOCs: A gradient boosting trees supervised machine learning and sentiment analysis approach. Computers & Education, 145, 103724.
[22]Abidalkareem, A., Ibrahim, A. K., Abd, M., Rehman, O., & Zhuang, H. (2024). Identification of gene expression in different stages of breast cancer with machine learning. Cancers (Basel), 16, 1864.
[23]Alshawwa, I. A., El-Mashharawi, H. Q., Salman, F. M., Al-Qumboz, M. N. A., Abunasser, B. S., & Abu-Naser, S. S. (2024). Advancements in early detection of breast cancer: Innovations and future directions. (Publisher and journal info missing – complete if available.)
[24]Bostanci, E., Kocak, E., Unal, M., Guzel, M. S., Acici, K., & Asuroglu, T. (2023). Machine learning analysis of RNA-seq data for diagnostic and prognostic prediction of colon cancer. Sensors, 23, 3080.
[25]Kushwaha, S. (2023). Review on artificial intelligence and human computer interaction. In Proceedings of the IEEE OPJU International Technology Conference on Emerging Technologies For Sustainable Development (pp. 1-6). O.P. Jindal University, Raigarh, Chhattisgarh, India.
[26]Chugh, G., Kumar, S., & Singh, N. (2021). Survey on machine learning and deep learning applications in breast cancer diagnosis. Cognitive Computation, 13, 1451–1470.
[27]Mahboubisarighieh, A., Shahverdi, H., Jafarpoor Nesheli, S., Alipoor Kermani, M., Niknam, M., Torkashvand, M. (2024). Assessing the efficacy of 3D Dual-CycleGAN model for multi-contrast MRI synthesis. Egyptian Journal of Radiology and Nuclear Medicine, 55, 1–12.
[28]Rezaeijo, S. M., Rezaei, M., Poursheikhani, A., Mohammadkhani, S., Goharifar, N., Shayankia, G. (2023). Integrative bioinformatics analysis of miRNA and mRNA expression profiles identified some potential biomarkers for breast cancer. Egyptian Journal of Medical Human Genetics, 24, 62.
[29]Sahu, A., Qazi, S., Raza, K., Singh, A., & Verma, S. (2022). Machine learning-based approach for early diagnosis of breast cancer using biomarkers and gene expression profiles. In Computational Intelligence in Oncology Applications: Diagnosis, Prognosis, and Therapy of Cancers (pp. 285–306). Springer.
[30]Asadi, O., Yekkalam, A., & Manthouri, M. (2023). MDDC: Melanoma detection using discrete wavelet transform and convolutional neural network. Journal of Ambient Intelligence and Humanized Computing, 14(9), 12959–12966.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Sumit Kushwaha, Anish Kumar, Akarsh Kumar Mishra, Ansh Thakur (Author)
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
