📣 Share your perspective

Help shape this research 🚀

Your experience with AI/ML-enabled software development can directly strengthen the next iteration of this work. If you have a few minutes, please answer the survey and contribute evidence that matters.

Respond survey now!

Structuring a Development Process for AI/ML Projects: a look into industry driven issues

Authors: Felipe Sonntag Manzoni, Camilla R. Gomes, Rayssa C. dos Reis, Ana Oran, Leonardo Marques

Venue: ICEIS’26 (Research Track), Benidorm, Spain

Companion page Full paper Last updated:

TL;DR (1-minute)

🎧 2-Minute Audio Summary

Prefer reading? The TL;DR bellow is designed to be read in about 2~3 minute.

Abstract

AI and ML projects differ in several aspects from traditional software development, so the traditional and agile development process cannot overcome the number of difficulties and aspects of these types of software development. In many cases, the AI and ML development cycles, even though representing just a fraction of the system, are made using non-standardized and inadequate development processes that cannot guarantee the developed product’s quality and effectiveness. This paper intends to elucidate and shed light on this issue, showcasing a novel development lifecycle process for AI/ML contexts that accounts for the quality activities and results to improve the project results. This research is developed in collaboration with the industry and is the first contribution on the developed process grounded on empirical information and validated on an empirical quality process. We present the first version of the development lifecycle process for AI/ML enabled systems and also present a initial evaluation of the developed artifact from the view of industry specialists, presenting the elicitation of further research and development for the next iteration.

2-Minute paper resume

Context: The growing adoption of Artificial Intelligence and Machine Learning inside modern software systems has introduced new development challenges that traditional software engineering processes were not designed to handle. AI/ML systems depend heavily on data availability, dataset quality, and iterative experimentation. In practice, many organizations still rely on traditional or agile software processes that treat model development as an isolated activity, often postponing evaluation and quality assurance until the end of the project lifecycle.

Problem: This separation between model experimentation and software engineering practices frequently leads to critical issues such as poor dataset governance, unclear relationships between requirements and data, weak model evaluation strategies, and late discovery of quality problems. These challenges increase the risk of rework, reduce system reliability, and make it harder for teams to deliver AI-enabled systems that meet production standards.

Approach: This work proposes a structured development lifecycle tailored for AI/ML-enabled systems that explicitly incorporates quality assurance activities throughout the development pipeline. The process was designed using empirical insights gathered from industry projects and qualitative interviews with experienced AI/ML specialists. The collected knowledge was analyzed through a structured qualitative analysis process, allowing the identification of recurring issues and development bottlenecks observed in real projects.

Key Contributions:

Definition of a structured development lifecycle that integrates AI/ML activities with software engineering practices.
Explicit inclusion of QA roles and checkpoints during dataset preparation, model training, evaluation, and selection.
Introduction of dataset governance and independent evaluation stages to reduce bias and improve model reliability.
Guidance on appropriate evaluation metrics such as Accuracy, Precision, Recall, F1-Score, AUC-ROC, MAE, MSE, and RMSE depending on the problem domain.

Initial Evaluation: The proposed lifecycle was reviewed by industry specialists who originally contributed to the interview study. Their feedback confirmed the relevance of early QA involvement, dataset quality validation, and independent model evaluation. Specialists highlighted that these steps could significantly reduce common issues observed in AI/ML projects, such as incorrect dataset partitioning, weak metric selection, and misalignment between requirements and available data.

Impact and Future Work: The results indicate that integrating structured quality checkpoints into the AI/ML lifecycle can help teams detect issues earlier, improve collaboration between development and QA teams, and increase the reliability of AI-enabled software systems. Future work will involve applying the proposed process to real-world projects across multiple organizations to further validate and refine the framework.

Resources

📄 Paper (PDF) 🖼️ Poster (PDF) 📎 Appendix

Full References

Cruzes, D. S. and Dybå, T. (2017). A content analysis process for qualitative software engineering research. Innovations in Systems and Software Engineering, 13(2–3):129–141.
Gezici, B. and Tarhan, A. K. (2022). Systematic literature review on software quality for ai-based software. Empirical Software Engineering, 27(3):66.
Giray, G. (2021). A software engineering perspective on engineering machine learning systems: State of the art and challenges. Journal of Systems and Software, 180:111031.
Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press, United Kingdom.
Lorenzoni, G., Alencar, P., Nascimento, N., and Cowan, D. (2021). Machine learning model development from a software engineering perspective: A systematic literature review. arXiv preprint.
Lwakatare, L. E., Raj, A., Crnkovic, I., Bosch, J., and Olsson, H. H. (2020). Large-scale machine learning systems in real-world industrial settings: A review of challenges and solutions. Information and Software Technology, 127:106368–106385.
Martínez-Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A. M., and Wagner, S. (2022). Software engineering for ai-based systems: A survey. ACM TOSEM, 31(2).
Nascimento, E., Nguyen-Duc, A., Sundbø, I., and Conte, T. (2020). Software engineering for artificial intelligence and machine learning software: A systematic literature review. arXiv preprint.
Ozkaya, I. (2020). What is really different in engineering ai-enabled systems? IEEE Software, 37(4):3–6.
Polkowski, Z., Vora, J., Tanwar, S., Tyagi, S., Singh, P. K., and Singh, Y. (2019). Machine learning-based software effort estimation: An analysis. In 2019 11th International Conference on Electronics, Computers and Artificial Intelligence (ECAI), pages 1–6. IEEE.
Serban, A., Blom, K. V. D., Hoos, H., and Visser, J. (2020). Adoption and effects of software engineering best practices in machine learning. ESEM, Article 3, 1–12. IEEE Computer Society.
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., and Wesslén, A. (2024). Experimentation in Software Engineering. Springer Berlin Heidelberg, Germany.

Cite this work (BibTeX)

@inproceedings{manzoni2026_industry_ai_ml_process,
  title     = {Structuring a Development Process for AI/ML Projects: a look into industry driven issues},
  author    = {Felipe Sonntag Manzoni and others},
  booktitle = {Proceedings of ICEIS 2026},
  year      = {2026},
  note      = {To appear / venue details to be confirmed}
}

How about a espresso patronum?