Workshop on GenAI Based Software Engineering

Objective of Workshop

Software engineering in the large is an effort-intensive and time-consuming activity whereas IT systems today need to make changes in the shortest possible duration. Most complex, large-scale software systems of today derive their requirements from existing (legacy) software and partial (incomplete) descriptions. Software development is thus a complex combination of transformation, reverse and forward engineering, involving code, data, and specifications, wherein data is both structured and unstructured. Expertise from subject matter experts (SMEs) is essential at each phase, which brings in the important component of domain knowledge. While Model-Driven Engineering (MDE), Knowledge Engineering (KE), and Reverse Engineering (RE) have mitigated some of the challenges, the emergence of Generative AI techniques holds the potential for a substantial breakthrough, though is yet to bring a consistent and substantial jump in productivity. There are challenges in understanding and resolving issues reported in GitHub. The absence of high-quality datasets that encompass a wide range of programming tasks, styles, and languages add to the challenges. With dynamically changing businesses of enterprises, and evolution of fast-changing and new technologies, the LLMs need to keep pace with the evolving code knowledge.

Use of GenAI for software development has seen increasing maturity in the past 1+ year. Different tools like GitHub Copilot, Amazon Q developer agent (Code Whisperer), AutoCodeRover + GPT 4o, Assistant GRU, SWE agent + Claude 3.5 Sonnet have enhanced and are enhancing code development, code completion, test case generation, debugging and issue fixing tasks. Some methods have exploited the instruction tuning and reinforcement learning with feedback. Several more small and large models and tools are exploiting RACG techniques and claim to considerably enhance the software development tasks.

The new paradigm of LLM Based AI agents, have demonstrated effectiveness in variety of Software Engineering (SE) tasks, such as program generation, software testing and debugging and program improvement as well as end-to-end software engineering. These agents can extend the capabilities of the backbone LLMs by utilizing external resources and tools and soliciting human interactions. From SE perspective, there is a need to analyze how LLM-based agents can tackle different software development and maintenance tasks. Whereas, from the agent perspective, there is a need to throw some light on the basic agentic design components, including planning, memory, perception, action and their roles and collaboration mechanisms, in multi-agent settings.

The proposed workshop aims to provide a collaborative platform for researchers and practitioners to delve into the convergence of traditional MDE, KE, and RE methodologies together with Generative AI technologies. By synergizing the strengths of Gen AI, LLM Agentic Frameworks, modeling, and knowledge representation for SDLC, our goal is to define a trajectory toward enhanced software engineering practices. We seek discussion on the following pivotal questions:

1. Architecting, designing, developing and maintaining industry-strength software is a multi-skill, long-drawn activity that cannot be effectively addressed by LLMs alone. What kinds of augmentation make it effective?
2. LLM is a vast storehouse of general information, but the typical need during SDLC is rather sharply focused. How best to bear local knowledge at scale to get the required focus?
3. How can Generative AI be effectively utilized for mining and constructing purpose-driven knowledge from software artifacts? Can Generative AI catalyze existing legacy modernization techniques to reduce cost, time and improve correctness?
4. How can Generative AI enhance the synthesis of tests and test data during SDLC?
5. Can bug fixes and change requests be analyzed and implemented expeditiously using Generative AI?
6. What are the best practices for maintaining the accuracy, relevance and reliability of Generative AI generated software artifacts?
7. What are the state-of-the-art techniques, experimental models, methodologies, benchmark datasets and evaluation metrics employed for the usage of LLMs and LLM-based agents in SE applications? What are the key differences in task performance between LLMs and LLM-based agents?

The workshop aims to foster interactive discussions, enabling participants to collectively shape the future of advanced software engineering. The inaugural edition will feature talks by invited speakers who are exploring one or more of the above questions, interspersed with short experiences of researchers who are exploring specific challenges of software engineering using GenAI techniques. Time permitting, we will have short lightening talks by researchers to share interesting observations and anecdotes.