Speeding Up the Engineering of Interactive Systems with Generative AI

Processing, Arduino, and the growth of the associated communities of practice, also called maker communities, has motivated a broader participation of non-technical users in the engineering of interactive systems. Besides online sharing, maker communities meet regularly and share knowledge for various purposes (e.g., creative hacking, social networking, lifelong learning). In the context of maker communities, the understanding of engineering interactive systems (e.g., motivations, objectives, collaboration, process, reports) and the design of the respective tools (e.g., end-user programming for artists, or children) are not well documented. As a remedy, we present a coherent overview of related work, as well as our own experiences in the organization and running of maker workshops. The tutorial format (lecture and hands-on workshop) benefits both practitioners and researchers with an understanding of creative software tools and practices. Moreover, participants become familiar with the organization of maker workshops as 1) a research method for understanding users, 2) an engineering process for interactive computer systems, and 3) a practice for teaching and learning.

Development of interactive systems and their user interfaces combines engineering and design, formal and informal development methods from different domains. Diverse models and tools can be used to support the developers’ work. In model-driven development, software systems are specified systematically using dedicated models for different aspects of the system. Yet, appropriate design of user interfaces is as important as functional correctness. This workshop provides a forum of multi-disciplinary discussion on how to integrate model-driven development with the often more informal methodologies used in user-centered design and engineering. Starting point of the discussion are the tools, models, methods and experiences of the workshop participants.KeywordsUser-interface engineeringinteraction designmodel-driven developmentuser-centered designmodels

As generative AI becomes embedded in education, research has largely emphasised outputs, ethics, and performance, with less attention to how these systems shape learners’ conduct and dispositions through interaction. This conceptual paper addresses that gap by treating large language models (LLMs) not as neutral tools but as meaning-mediating infrastructures: interactive systems that organise what kinds of learning moves become easy, sensible, and repeatable, and how learners come to interpret effort, struggle, and success. Extending socio-cognitive accounts of feedback and norm perception through Luhmann’s systems theory, we introduce programmatic behaviour to describe how repeated engagement with structured environments – including AI systems – stabilises habitual orientations toward help-seeking, completion, and contestation. We also develop meaning mediation to explain how tone, pacing, dialogue structure, and closure shape the lived significance of error, progress, and competence in practice. On this basis, we outline three heuristic trajectories that may emerge through sustained LLM use – supportive, corrosive, and manipulative patterns – showing how behavioural drift is co-produced by interaction design and institutional framing. The paper argues that evaluating generative AI in education therefore requires attention to interactional defaults and their normative effects, and it proposes an AI literacy agenda centred on noticing and steering participation within AI-mediated infrastructures.

Exploring nursing students' acceptance of RAG-enhanced GenAI through the AIDUA model: A qualitative study.

This study investigates the integration of generative artificial intelligence into multimodal learning frameworks, emphasizing the fusion of vision, text, and audio to revolutionize human-computer interaction. Leveraging advanced deep learning architectures, our research explores how generative models can simultaneously process and synthesize diverse data streams, thereby enabling more natural and intuitive communication between users and machines. The proposed framework employs convolutional neural networks for detailed image analysis, state-of-the-art natural language processing algorithms for text understanding, and recurrent neural networks for comprehensive audio interpretation. By bridging these modalities, the system identifies contextual relationships and produces coherent, context-aware responses that closely mimic human reasoning. Experimental evaluations indicate that this integrative approach significantly enhances interaction accuracy, system responsiveness, and overall user engagement compared to traditional unimodal systems. The study also addresses critical challenges, including the complexities of cross-modal data alignment, increased computational demands, and issues related to data heterogeneity. We propose innovative solutions such as adaptive weighting strategies and modular architectures to mitigate these challenges. The findings demonstrate that generative AI not only improves the efficiency of human-computer interactions but also paves the way for the development of more adaptive, intelligent, and user-centric systems. This work lays a robust foundation for future research aimed at refining multimodal learning systems and advancing the capabilities of interactive AI technologies. In summary, the integration of generative AI into multimodal learning represents a promising frontier that blends multiple sensory inputs into a unified analytical framework. Our research highlights the potential to significantly transform digital interaction landscapes globally.

In this instructional design case, I described a final project implemented in a Software Systems Engineering course and explored the integration of generative AI tools in undergraduate engineering education. I, as the course instructor, asked students to design and develop text-based educational games using C programming, incorporating Microsoft Copilot as an assistive technology. I generated the project description and rubric with a generative AI tool, focusing on ethical, creative, and effective AI use. Students reflected on their design process, learning outcomes, and challenges. Highlights of this design include a practical approach to preparing students to use generative AI tools responsibly and productively in Software System Engineering, while identifying areas for improving guidance on their implementation.

Generative AI (GenAI) has significantly advanced the ease and flexibility of image creation. However, it remains a challenge to precisely control spatial compositions, including object arrangement and scene conditions. To bridge this gap, we propose Canvas3D, an interactive system leveraging a 3D engine to enable precise spatial manipulation for image generation. Upon user prompt, Canvas3D automatically converts textual descriptions into interactive objects within a 3D engine-driven virtual canvas, empowering direct and precise spatial configuration. These user-defined arrangements generate explicit spatial constraints that guide generative models in accurately reflecting user intentions in the resulting images. We conducted a closed-ended comparative study between Canvas3D and a baseline system, and an open-ended, free-form study to assess overall system usability. The results indicate that Canvas3D outperforms the baseline on spatial control, interactivity, and overall user experience.

It is our great pleasure to welcome you to the 4th ACM SIGCHI Symposium on Engineering Interactive Computing Systems -- EICS'12 held in Copenhagen (25--28 June 2012). EICS is an international conference devoted to all aspects of engineering usable and effective interactive computing systems. Topics of interest include multi-device interactive systems, new and emerging modalities (e.g., gesture), entertaining applications (e.g., mobile and ubiquitous games), safety critical systems (e.g. medical devices), and design and development methods (e.g., extreme programming). EICS focuses on tools, techniques and methods for designing and developing interactive systems. The conference brings together people who study or practice the engineering of interactive systems, drawing from Human-Computer Interaction (HCI), Software Engineering, Requirements Engineering, Computer-Supported Cooperative Work (CSCW), Ubiquitous & Pervasive Systems, Game Development, and Cognitive Engineering communities. The conference is the legatee of a number of conferences and workshops series: EHCI (Engineering Human Computer Interaction), DSV-IS (International Workshop on the Design, Specification and Verification of Interactive Systems), CADUI (International Conference on Computer-Aided Design of User Interfaces) and TAMODIA (International Workshop on Task Models and Diagrams). Since its beginning EICS has witnessed a growing number of submissions. This year the program contains 21 full papers carefully chosen from a total of 95 submissions (22% acceptance rate). There are also 12 late breaking papers (five of which presented as posters) as well as a number of doctoral reports, workshop reports, tutorial abstracts and demonstration descriptions. The published material originates from 15 countries, including New Zealand, North and South America, and Europe. In addition, keynote addresses will be offered by Robert Jacob (Tufts University, USA) and Jakob Bardram (IT University of Copenhagen, Denmark). We believe that for this fourth EICS edition we obtained an exciting and interactive program, which stimulates fruitful discussion in the relevant research fields. Topics range from model-based approaches to the design, analysis and generation of user interfaces, to toolkits supporting their development. A diversity of interaction styles and application areas is covered including multimodal, multidevice or multi-touch interfaces, ubiquitous computing, and health. We hope that you will find this year program interesting and thought provoking. The symposium aims to provide you with a valuable opportunity to share ideas with other researchers and practitioners from institutions around the world. We also wish the best to the next edition, EICS 2013 to be held in London, UK, in June 2013.

Welcome to the historic, vibrant and beautiful city of Edinburgh. Scotland's capital is imbued with a rich tradition of creativity, design, innovation and research, making it a most fitting home for the 12th ACM SIGCHI Designing Interactive Systems (DIS'17) conference. Running biennially since 1995, the Designing Interactive Systems (DIS) Conference is the premier, international arena where designers, artists, psychologists, user experience researchers, systems engineers, and many more, come together to debate and shape the future of interactive systems design and practice. DIS is owned by the ACM Special Interest Group on Computer- Human Interaction (SIGCHI). In the past, DIS has been held in Ann Arbor (1995), Amsterdam (1997), New York (2000), London (2002), Boston (2004), State College (2006), Cape Town (2008), Aarhus (2010), Newcastle (2012), Vancouver (2014), Brisbane (2016) and now, in 2017, moves to a new era of annual conferences. The DIS'17 conference schedule starts with two days of pre-conference workshops and a doctoral consortium and continues with three main conference days into which we have packed 130 peer reviewed paper and pictorial presentations into 30 sessions across three parallel tracks, alongside which we have demos, provocations and works in progress, a lunchtime symposium held by Facebook and two invited keynote speakers. The social program includes a welcome reception on Sunday evening, a Demos & Posters Industry Reception on Monday night, the DIS conference dinner on Tuesday night, and a post-conference closing drinks party on Wednesday night. The theme of DIS 2017 is bridging and connecting -- across disciplines, practices, places and understandings. The most interesting things happen at edges and boundaries, and so the aim of the 2017 conference is to examine different approaches to framing knowledge about the design of interactive systems. As advancements in interactive technology continue to blur the demarcations between people and data, and between things and software, interaction designers and researchers are finding new ways to explore this evolving, interdisciplinary landscape. At DIS 2017 we shall consider the contrasts and commonalities that are central in shaping the landscape of emerging interaction paradigms. I am very proud that we have been able to curate and assemble a stellar program, including two invited keynote presentations by Salvatore Iaconesi (ISIA Design Florence) and Di Mainstone (QMU), I am sure they and the program will encourage much thought, discussion and future endeavours.

Under the new financial market environment along with increasing individualization demand of customer value, innovative processes for the financial engineering products had been still limited in style of inner experts in financial enterprises guiding products design, which would lead to many financial products not be accepted by clients. According to the puzzle of disjoint between product design and market lied in financial enterprises, based on the analysis of design characteristic of the financial engineering product, interactive design information platform for financial engineering product was primarily conceived and constructed in virtue of computer, network engineering and system engineering, etc.. Designing financial products on the platform, Financial enterprises can try their best to meet the client's requirement.

The Bell System Telephone Office Planning and Engineering System (TOPES) is a time-shared interactive graphics system for planning building space. TOPES Programmed Instructions is a function-oriented book of special design. Step-by-step instructions are accompanied by exact reproductions of the graphic display on the screen. Each page contains a few terse statements that direct action at the keyboard, cursor, and display screen, and a short description of how TOPES responds. On the lower half of each page is a photographic reproduction of the graphics that should appear on the screen. By following the instructions the user becomes familiar with most TOPES conventions and commands, and creates charts, drawings, and plans that can serve as examples for future work. Although it is typical to provide personalized instruction for operators of complex computer graphics systems, this has been found unnecessary for TOPES.

The traditional simulation teaching system is limited by the system support software, which cannot realize good interactive teaching. To this end, an interactive simulation teaching system based on virtual reality for electrical engineering is designed. Based on the hardware consisting of ATmega328 chip as the core, sensors and surrounding circuits, the development tool OpenGL is used to draw 3D virtual teaching scenes. The system design is completed by using a double-layer hidden Markov chain to realize simultaneous interactive teaching of multiple paths. The comparison experiment shows that the teaching system has better interactivity, can effectively enhance students' learning interest, and has good user feedback.

Building resilient infrastructures is cross-disciplinary topic which is important in systems engineering, network security, transportation systems, etc. This paper describes dynamic quantitative model for estimating infrastructure resilience and it potential costs. Currently there are no models that could describe dynamic effects of threat impact on infrastructure and estimate costs of different prevention strategies. This work defines multilayered graph model that describes threat impact through system interdependencies at every moment of time and chooses optimal response strategy in the terms of system performance and resilience strategy costs.

Specifying and implementing flexible human-computer dialogs, such as those used in kiosks, is complex because of the numerous and varied directions in which each user might steer a dialog. The objective of this research is to improve dialog specification and implementation. To do so we developed a model for specifying and staging mixed-initiative dialogs. The model involves a dialog authoring notation, based on concepts from programming languages, for specifying a variety of unsolicited reporting, mixed-initiative dialogs in a concise representation that serves as a design for dialog implementation. Guided by this foundation, we built a dialog staging engine which operationalizes dialogs specified in this notation. The model, notation, and engine help automate the engineering of mixed-initiative dialog systems. These results also provide a proof-of-concept for dialog specification and implementation from the perspective of theoretical programming languages. The ubiquity of dialogs in domains such as travel, education, and health care with the increased use of interactive voice-response systems and virtual environments provide a fertile landscape for further investigation of these results.

Three dimensional modeling and simulation software are becoming more widespread in many fields with the emphasis on engineering, military and medical applications. The main important features of these applications are fast prototyping, simulation of scientific and engineering problems under different conditions and interactive user interface. Therefore, an opportunity for a fast visual education course and understanding the physical and mathematical reasons lying under the problems is presented to engineers, military and scientific researches in the respective fields. The main requirements of these applications are high performance computer systems and user interfaces that feature an easier interaction. The fast developing computer architecture maintains the necessary architecture and enables the applications to improve. Considering above and the expectations of Dokuz Eylul University Faculty of Medicine, the development of a virtual environment software that will model the predefined medical processes with user interaction has been aimed. In this paper, the 1. section is reserved for basic concepts and field applications. The hardware and software infrastructure and algorithm development process we used are presented in the 2. section. The 3. section presents the completed stages and first results of our application. The final section which is the 4. section is reserved for the conclusion and future work.