Planning for Non-player Characters Using HTN and Visual Perception

On the way of the creation of virtual autonomous characters that are lifelike, intelligent and convey empathy, a strategic planning architecture is presented in the domain of serious games, where games are designed and built for an educational primary purpose other than pure entertainment. Our serious game is in the domain of emergency management; firefighters and first responders in accidents situations. Two contributions are presented in this paper; firstly, a visual perception system for non-player characters (NPCs) along with a short-term memory (working memory) are implemented so that NPCs will have access to only limited information and have to build their plans and make their decisions based on what they can perceive from their surrounding environment. Secondly, an HTN planning approach is implemented based on SHOP for the domain of our serious game. Five modules were developed in this part; Controller, World Model, Domain, Interface and HTN Planner. The plan is generated online based on the information gathered from the visual perception system and stored in the working memory. Our experiment shows the effectiveness of the proposed approach to create believable NPC behavior.

Robustness of visual perception system in progressive challenging weather scenarios

The visual perception system is the most important system for human learning since it receives over 80% of the learning information from the outside world. With the exponential growth of artificial intelligence technology, there is a pressing need for high-energy and area-efficiency visual perception systems capable of processing efficiently the received natural information. Currently, memristors with their elaborate dynamics, excellent scalability, and information (e.g., visual, pressure, sound, etc.) perception ability exhibit tremendous potential for the application of visual perception. Here, we propose a fully memristor-based artificial visual perception nervous system (AVPNS) which consists of a quantum-dot-based photoelectric memristor and a nanosheet-based threshold-switching (TS) memristor. We use a photoelectric and a TS memristor to implement the synapse and leaky integrate-and-fire (LIF) neuron functions, respectively. With the proposed AVPNS we successfully demonstrate the biological image perception, integration and fire, as well as the biosensitization process. Furthermore, the self-regulation process of a speed meeting control system in driverless automobiles can be accurately and conceptually emulated by this system. Our work shows that the functions of the biological visual nervous system may be systematically emulated by a memristor-based hardware system, thus expanding the spectrum of memristor applications in artificial intelligence.

The application and design of neural computation in visual perception

The increasing trend of autonomous driving vehicles in smart cities emphasizes the need for safe travel. However, the presence of obstacles, potholes, and complex road environments, such as poor illumination and occlusion, can cause blurred road images that may impact the accuracy of maneuver prediction in visual perception systems. To address these challenges, a novel ensemble model named ABHO-based deep CNN-BiLSTM has been proposed for traffic sign detection. This model combines a hybrid convolutional neural network (CNN) and bidirectional long short-term memory (BiLSTM) with the alarming-based hunting optimization (ABHO) algorithm to improve maneuver prediction accuracy. Additionally, a modified hough-enabled lane generative adversarial network (ABHO based HoughGAN) has been proposed, which is designed to be robust to blurred images. The ABHO algorithm, inspired by the defending and social characteristics of starling birds and Canis kojot, allows the model to efficiently search for the optimal solution from the available solutions in the search space. The proposed ensemble model has shown significantly improved accuracy, sensitivity, and specificity in maneuver prediction compared to previously utilized methods, with minimal error during lane detection. Overall, the proposed ensemble model addresses the challenges faced by autonomous driving vehicles in complex and obstructed road environments, offering a promising solution for enhancing safety and reliability in smart cities.

Visual perception, as a core component of Intelligent Transportation Systems (ITS), plays a key role in enhancing safety and efficiency in urban mobility. While single-task visual perception methods have applications in areas like pedestrian detection and traffic sign recognition, the complexity of real-world scenarios necessitates a shift toward multi-task approaches. This paper introduces the Edge-assisted Multi-task Visual Perception (EMVP) system, which is specifically designed to address the computational intensity and dynamic concurrency challenges inherent to multi-task processing in edge environments. EMVP adopts a collaborative architecture that strategically partitions computational tasks between vehicles and Road-Side Units (RSUs). By integrating Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), the system achieves a lightweight yet efficient multi-task model for resource-constrained environments. To adapt to the dynamic and concurrent nature of multi-vehicle scenarios, EMVP incorporates a content-aware adaptive inference mechanism based on reinforcement learning, enabling dynamic task scheduling to improve Quality of Service (QoS). Experimental results demonstrate that, compared to the single-task baseline model, the multi-task model of EMVP reduces the computational cost by 86.59% on average while achieving a 4.32% improvement in accuracy. Additionally, in dynamic multi-access environments, EMVP’s adaptive scheduling mechanism, which leverages spatiotemporal content awareness, achieves an average QoS improvement of 7.36% over the sub-optimal method.

This paper focuses on the application of visual perception in autonomous driving. The types of cameras in visual perception systems are introduced, including charge-coupled device (CCD), and Complementary metal oxide semiconductor (CMOS) image sensors, and the LiDAR system with vision sensors. The applications of visual perception in autonomous driving are discussed, such as environmental perception (including road recognition, lane line detection, and traffic sign and signal light recognition) and vehicle positioning. However, visual perception technology faces challenges, such as the effects of light changes and interference from harsh weather on visual sensors. To address these challenges, solutions include the fusion of vision and other sensors, as well as progress in hardware and software technology. By optimizing and improving visual sensors in terms of precision, cost, power consumption, stability, and data perception and processing ability, the performance of autonomous driving systems can be enhanced. The multi-sensor fusion method utilizes the advantages of different sensors to obtain more comprehensive environmental information and improve the reliability and adaptability of the vehicle system. This paper can help autonomous driving researchers understand the application of visual sensors in autonomous driving more quickly.

Color-based indexing is an important tool in image database retrieval. Compared with other features of the image, color features are less sensitive to noise and background complication. Based on the human visual system's perception of color information, this paper presents a dependent scalar quantization approach to extract the characteristic colors of an image as color features. The characteristic colors are suitably arranged in order to obtain a sequence of feature vectors. Using this sequence of feature vectors, a dynamic matching method is then employed to match the query image with database images for a nonstationary identification environment. The empirical results show that the characteristic colors are reliable color features for image database retrieval. In addition, the proposed matching method has acceptable accuracy of image retrieval compared with existing methods.

Crushing of blasted ore is an essential phase in extraction of valuable minerals in mining industry. It is typically performed in multiple stages with each stage producing finer fragmentation. Performance and throughput of the first stage of crushing is highly dependent on the size distribution of the blasted ore. In the crushing plant, a metal grate prevents oversized boulders from getting into the crusher jaws, and a human‐controlled hydraulic manipulator equipped with a rock hammer is required to break oversized boulders and ensure continuous material flow. This secondary breaking task is event‐based in the sense that ore trucks deliver boulders at irregular intervals, thus requiring constant human supervision to ensure continuous material flow and prevent blockages. To automatize such breaking tasks, an intelligent robotic control system along with a visual perception system (VPS) is essential. In this manuscript, we propose an autonomous breaker system that includes a VPS capable of detecting multiple irregularly shaped rocks, a robotic control system featuring a decision‐making mechanism for determining the breaking order when dealing with multiple rocks, and a comprehensive manipulator control system. We present a proof of concept for an autonomous robotic boulder breaking system, which consists of a stereo‐camera‐based VPS and an industrial rock‐breaking manipulator robotized with our retrofitted system design. The experiments in this study were conducted in a real‐world setup, and the results were evaluated based on the success rates of breaking. The experiments yielded an average success rate of 34% and a break pace of 3.3 attempts per minute.

Vision-based perception systems are widely used in unmanned platforms and have an important role in tasks such as object classification, object detection, and instance segmentation. The function and performance of visual perception system is not only related to the sensing hardware and recognition algorithm, but also influenced by the external environment. In this paper, the detection performance of the artificial intelligence model is used as a metric to study the impact of vibration-blurred images on the environmental adaptability of visual perception system. The structure and function of AI-baesd visual perception system are analyzed, and two vibration-blurred image generation methods, vibration test and vibration simulation, are introduced. The experiment quantitatively verified the effect of mechanical vibration stress on the performance of AI model during the operation of visual perception system.

Este trabalho apresenta um panorama de algumas pesquisas a respeito de como a cultura está relacionada de forma causal com o entendimento da cognição humana. Particularmente, avalio estudos a respeito da influência da cultura na memória de curto prazo, na percepção visual, na gramática, na cognição numérica, e na evolução linguística. Além disso, apresento um desiderato de metodologias sobre as conexões entre cultura e cognição, e um direcionamento para pesquisas futuras.

Common research in the field of Virtual Reality (VR) considers acoustic stimulation as a highly important necessity for enhanced immersion into virtual scenes. However, most common VR toolkits do only marginally support the integration of sound for the application programmer. Furthermore, the quality of stimulation that is provided usually ranges from system sounds (e.g. beeps while selecting a menu) to simple 3D panning. In the latter case, these approaches do only allow the user to correctly detect sounds that are at quite a distance from his current position. Binaural synthesis is an interesting way to allow the spatial auditory representation by using few loudspeakers or headphones. This paper describes a system that combines the efforts of creating a binaural representation for the listener who is interacting in a common visual VR application in real-time, thus allowing the research on interaction between visual and auditory human perception systems. It will describe the theoretical background to establishing a binaural representation of a sound and the necessary hardware set-up for this. Afterwards, the infrastructure and software interface which will allow the connection of the audio renderer to a visual VR toolkit is discusse

The result of a cognitive process, a symbol is a type of channel for visual communication. For complicated thinking behaviors, designers concretize and visualize abstract concepts. Visual design is one of the major elements in the formation and construction of meaning. Cognition is the key to constructing the communication of a message (symbolic design). For visual designers, communication and the visual cognition thereof are the ultimate goals. Designers need not only seek unique expression of visual beauty but also achieve an organic goal in visual communication, wherein, as a result, symbolic communication and cognition are required to reduce the cognitive noise in the process. The Swiss linguist Saussure has stated that ”symbols are combined with the signifier and the signified” to explain that symbols consist of the combination of formation and concept. He also discussed an arbitrary concept of symbols to illustrate the interaction between meanings and formation. By using concrete surface features transferred to abstract meaning to design cognitive procedures for symbolic design, an operational model of symbolic and cognitive design is explored. The purpose of this study was to understand the cognition and transference utilized by designers in the process of symbolic formation of meanings. On the basis of semiotics and cognitive psychology, a set of systemized-design operational methods was arranged and a study of designers' ”meaning” and ”formation” in symbolic design applied to the basic structure of visual cognition. The basic structure of both visual cognition and the creation of visual design procedures was used to explore the structural relationships among visual symbol formation, meanings and cognition.

Research on environmental adaptability of AI-based visual perception system under the perspective of vibration

This paper introduces a visual perception system of UAV which solves the perception problem of UAV entering the house through the window during street warfare reconnaissance, that is how to identify and locate the window accurately in real-time and give the maximum safe crossing radius and plane normal vector. In order to solve the above problems, we propose an airborne visual perception system composed of window detection and localization. In the detection part, YOLOv3 algorithm is used to identify the window, and Harris is used to detect the corners of the window edge. In the localization part, the depth camera and visual odometer are combined to locate the window, then the maximum safe crossing radius and the plane normal vector of the window is calculated. In order to verify the performance of the system, we set up a tent for the actual flight experiment.