Beyond Performance Metrics: The Critical Role of Resource-Based Evaluation in Assessing IoT Attack Detectors
DOI:
https://doi.org/10.47839/ijc.23.3.3659Keywords:
IoT, IDS, Tiny ML, Attacks, CybersecurityAbstract
The proliferation of threats within the Internet of Things (IoT) environment is intensifying, largely due to the inherent limitations of this technology. The panoply of anti-threats based on artificial intelligence suffer from the complete embedment of models in limited resources. Tiny Machine Learning (TinyML) is presented as an opportunity in optimizing and selecting machine learning algorithms specifically tailored for intrusion detection systems (IDS) on limited-resource devices. This article addresses the challenges that must be overcome to enable the deployment of machine learning models on devices with constrained resources. In particular, it introduces additional indicators that could influence the algorithmic design of IoT models. Utilizing the PyCaret tool on the TON_IoT dataset, which encompasses nine distinct attacks, we developed and evaluated our approach for selecting the optimal algorithm from fourteen supervised learning models. The proposed tool, beyond the traditional six performance metrics, emphasizes resource consumption metrics, including memory, processor usage, battery life, and execution time – key considerations for TinyML in model refinement and selection. This study has identified less resource-intensive models suitable for developers in the design of IDS for IoT systems. We believe this research offers a foundational framework for the development of lightweight and efficient IoT vulnerability detection solutions.
References
B. Ankur and P. R. Udai, “Context-aware computing for IoT: History, applications and research challenges,” Proceedings of the Second International Conference on Smart Energy and Communication, January 2021, pp. 719-726. https://doi.org/10.1007/978-981-15-6707-0_70.
L. Ravie, “New IoT RapperBot malware targeting Linux servers via SSH brute-forcing attack,” The Hacker News, 07 August 2022. [Online]. available at: https://thehackernews.com/2022/08/new-iot-rapperbot-malware-targeting.html.
N. Paul, “Five most famous DDoS attacks and then some,” A10 Blog, Network Security, 4 May 2022. [Online]. Available at: https://www.a10networks.com/blog/5-most-famous-ddos-attacks/
O. Amroussi, “Revisited: OWASP Top 10 Vulnerabilities 2022,” Vulnerability Management, 2024. https://vulcan.io/blog/owasp-top-10-vulnerabilities-2022-what-we-learned/.
D. A. S. Lachit, “TinyML meets IoT: A comprehensive survey,” Internet of things, vol. 16, article 100461, 2021. https://doi.org/10.1016/j.iot.2021.100461.
M. Nour, “A new distributed architecture for evaluating AI-based security systems at the edge: Network TONIoT datasets,” Sustainable Cities and Society, vol. 72, article 102994, 2021, https://doi.org/10.1016/j.scs.2021.102994.
P. Marc-Oliver, A. François-Xavier, “All eyes on you: Distributed multi-dimensional iot microservice anomaly detection,” Proceedings of the 2018 14th International Conference on Network and Service Management (CNSM), December 2018, pp. 72-80.
M. Nour, “A systemic IoT-Fog-Cloud architecture for big-data analytics and cyber security systems: A review of fog computing,” arXiv preprint arXiv:1906.01055, 4 May 2019.
K. V. Karthik, A. A. R. K. Nilofar, “Investigation on intrusion detection systems (IDSs) in IoT,” International Journal of Emerging Trends in Engineering Research, vol. 10, pp. 2347-3983, 2022. https://doi.org/10.30534/ijeter/2022/041032022.
N. Schizas, A. Karras, C. Karras, “TinyML for ultra-low power AI and large scale IoT deployments: A systematic review,” Network Cost Reduction in Cloud and Fog Computing Environments, vol. 14, issue 12, article 363, 2022. https://doi.org/10.3390/fi14120363.
M. Inês, S. João, R. Patrícia, S. Simão et al, “Host-based IDS: A review and open issues of an anomaly detection system in IoT,” Future Generation Computer Systems, vol. 133, pp. 95–113, 2022. https://doi.org/10.1016/j.future.2022.03.001.
A. Abdullah, M. Nour, T. Zahir, M. Abdun, A. Adnan, “TON_IoT telemetry dataset: A new generation dataset of IoT and IIoT for data-driven intrusion detection systems,” IEEE Access, pp. 2169-3536, 2020.
M. Nour, “The TONIoT Datasets,” UNSW Canberra, 2021.
N. Chaabouni, M. Mosbah, A. Zemmari, C. Sauvignac and P. Faruki, “Network intrusion detection for IoT security based on learning techniques,” IEEE Communications Surveys & Tutorials, vol. 21, no. 3, pp. 2671-2701, 2019. https://doi.org/10.1109/COMST.2019.2896380.
Y. L., L. D. Xu, “Internet of Things (IoT) cybersecurity research: A review of current research topics,” IEEE Internet of Things Journal, vol. 6, no. 12, pp. 2103-2115, 2018. https://doi.org/10.1109/JIOT.2018.2869847.
C. Shanzhi, X. Hui, L. Dake, H. Bo, W. Hucheng, “A vision of IoT: Applications, challenges, and opportunities with China perspective,” IEEE Internet of Things Journal, vol. 1, no. 4, pp. 349-359, 2014. https://doi.org/10.1109/JIOT.2014.2337336.
J. Ashraf, M. Keshk, N. Moustafa, M. Abdel-Basset, H. Khurshid, A. D. Bakhshi, R. R. Mostafa, “IoTBoT-IDS: A novel statistical learning-enabled botnet detection framework for protecting networks of smart cities,” Sustainable Cities and Society, vol. 72, article 103041, 2021, https://doi.org/10.1016/j.scs.2021.103041https://doi.org/10.1016/j.scs.2021.103041.
T. M. Booij, I. Chiscop, E. Meeuwissen, N. Moustafa, “ToN_IoT: The role of heterogeneity and the need for standardization of features and attack types in IoT network intrusion data sets,” IEEE Internet of Things Journal, vol. 9, pp. 485-496, 31 May 2021. https://doi.org/10.1109/JIOT.2021.3085194.
U. T. Maurras, C. Yousra, B. Aliou et C. Raja, “Etude comparative des méthodes de détection d’anomalies,” Revue des Nouvelles Technologies de l’Information, pp.1-13, 2020.
B. Govindraj, “Guide to OWASP IoT top 10 for proactive security,” 11 May 2021. [Online]: available at: https://www.appsealing.com/owasp-iot-top-10/
I. Mukherjee, N. K. Sahu, S. K. Sahana, “Simulation and modeling for anomaly detection in IoT network using machine learning,” International Journal of Wireless Information Networks, vol. 30, pp. 173–189, 2022. https://doi.org/10.1007/s10776-021-00542-7.
NVD, “National vulnerability database,” Understanding Vulnerability Detail Pages, 20 September 2022, [Online]. available at: https://nvd.nist.gov/vuln
S. Raza, L. Wallgren, T. Voigt, “SVELTE: Real-time intrusion detection in the Internet of Things,” Ad Hoc Networks, vol. 11, issue 8, pp. 2661-2674, 2013. https://doi.org/10.1016/j.adhoc.2013.04.014.
B. Blinowski, P. Piotrowski, “CVE based classification of vulnerable IoT systems,” Theory and Applications of Dependable Computer Systems,vol. 1173, pp. 82–93, 2020. https://doi.org/10.1007/978-3-030-48256-5_9.
A. Sivanathan, H. H. Gharakheili, F. Loi, A. Radford, C. Wijenayake, A. Vishwanath, V. Sivaraman, “Classifying IoT devices in smart environments using network traffic characteristics,” IEEE Transactions on Mobile Computing, vol. 18, pp. 1745-1759, 2019. https://doi.org/10.1109/TMC.2018.2866249.
N. Koroniotis, N. Moustafa, E. Sitnikova, B. P. Turnbull, “Towards the development of realistic botnet dataset in the Internet of Things for network forensic analytics: Bot-IoT dataset,” Future Generation Computer Systems, vol. 100, pp. 779-796, 2019. https://doi.org/10.1016/j.future.2019.05.041.
A. Hamza, H. H. Gharakheili, T. A. Benson, V. Sivaraman, “Detecting volumetric attacks on loT devices via SDN-based monitoring of MUD activity,” Proceedings of the 2019 ACM Symposium on SDN Research SOSR'19, April 2019, pp. 36–48. https://doi.org/10.1145/3314148.3314352.
S. Suthaharan, M. Alzahrani, S. Rajasegarar, C. Leckie and M. Palaniswami, “Labelled data collection for anomaly detection in wireless sensor networks,” Proceedings of the 2010 Sixth International Conference on Intelligent Sensors, Sensor Networks and Information Processing, February 2011, pp. 269-274. https://doi.org/10.1109/ISSNIP.2010.5706782,
O. Brun, Y. Yin, E. Gelenbe, “Deep learning with dense random neural network for detecting attacks against IoT-connected home environments,” Procedia Computer Science, vol. 134, pp. 458-463, 2018. https://doi.org/10.1016/j.procs.2018.07.183.
Ö. A. Aslan, R. Samet, “A comprehensive review on malware detection approaches,” IEEE Access, vol. 8, pp. 6249-6271, 2020. https://doi.org/10.1109/ACCESS.2019.2963724.
Amit, A. Dhingra, V. Sindhu, A. Sangwan, “A comprehensive review of DDoS attack, types and mitigation techniques in the Internet of Things network,” International Journal for Modern Trends in Science and Technology, vol. 8, pp. 72-79, 2022.
C. Kolias, G. Kambourakis, A. Stavrou, S. Gritzalis, “Intrusion detection in 802.11 networks: Empirical evaluation of threats and a public dataset,” IEEE Communications Surveys & Tutorials, vol. 18, no. 11, pp. 184-208, 2015. https://doi.org/10.1109/COMST.2015.2402161.
Q.-D. Ngo, H.-T. Nguyen, V.-H. Le, D.-H. Nguyen, “A survey of IoT malware and detection methods based on static features,” ICT Express, vol. 6, n. 4, pp. 280-286, 2020. https://doi.org/10.1016/j.icte.2020.04.005.
A. Cano, D. T. Nguyen, S. Ventura, K. Cios, “ur-CAIM: improved CAIM discretization for unbalanced and balanced data,” Soft Computing, vol. 6, p. 173–188, 2014. https://doi.org/10.1007/s00500-014-1488-1.
C. Jun et C. Chi, “Design of complex event-processing IDS in Internet of Things,” Proceedings of the 2014 Sixth International Conference on Measuring Technology and Mechatronics Automation, April 2014, pp. 226-229, https://doi.org/10.1109/ICMTMA.2014.57.
Downloads
Published
How to Cite
Issue
Section
License
International Journal of Computing is an open access journal. Authors who publish with this journal agree to the following terms:• Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
• Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
• Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
