A Novel Flexible Endomicroscopic Scanning Instrument with Visual Servoing Control

This paper aims to develop a visual servo control of a robotic manipulator for cherry tomato harvesting. In the robotic manipulator, an RGB-depth camera was mounted to the end effector to acquire the poses of the target cherry tomatoes in space. The eye-in-hand-based visual servo controller guides the end effector to implement eye–hand coordination to harvest the target cherry tomatoes, in which a hybrid visual servo control method (HVSC) with the fuzzy dynamic control parameters was proposed by combining position-based visual servo (PBVS) control and image-based visual servo (IBVS) control for the tradeoff of both performances. In addition, a novel cutting and clipping integrated mechanism was designed to pick the target cherry tomatoes. The proposed tomato-harvesting robotic manipulator with HVSC was validated and evaluated in a laboratory testbed based on harvesting implementation. The results show that the developed robotic manipulator using HVSC has an average harvesting time of 9.40 s/per and an average harvesting success rate of 96.25% in picking cherry tomatoes.

For over 100 years white-light cystoscopy has remained the gold-standard technique for the detection of bladder cancer (BCa). Some limitations in the detection of flat lesions (CIS), the differentiation between inflammation and malignancy, the inaccurate determination of the tumor margin status as well as the tumor depth, have led to a variety of technological improvements. The aim of this review is to evaluate the impact of these improvements in the diagnosis of BCa and their effectiveness in clinical practice. A systematic literature search was conducted according to the PRISMA statement to identify studies reporting on imaging modalities in the diagnosis of NMIBC between 2000 and 2017. A two-stage selection process was utilized to determine eligible studies. A total of 74 studies were considered for final analysis. Optical imaging technologies have emerged as an adjunct to white-light cystoscopy and can be classified according to their scope as macroscopic, microscopic and molecular. Macroscopic techniques including photodynamic diagnosis (PDD), narrow-band imaging (NBI) and the Storz Professional Image Enhancement System (IMAGE1 S, formerly known as SPIES) are similar to white-light cystoscopy, but are superior in the detection of bladder tumors by means of contrast enhancement. Especially the detection rate of very mute lesions in the bladder mucosa (CIS) could be significantly increased by the use of these methods. Microscopic imaging techniques like confocal laser endomicroscopy and optical coherence tomography permit a real-time high-resolution assessment of the bladder mucosa at a cellular and sub-cellular level with spatial resolutions similar to histology, enabling the surgeon to perform an 'optical biopsy'. Molecular techniques are based on the combination of optical imaging technologies with fluorescence labeling of cancer-specific molecular agents like antibodies. This labeling is intended to favor an optical distinction between benign and malignant tissue. Optical improvements of the standard white-light cystoscopy have proven their benefit in the detection of BCa and have found their way into clinical practice. Especially the combination of macroscopic and microscopic techniques may improve diagnostic accuracy. Nevertheless, HAL-PDD guided cystoscopy is the only approach approved for routine use in the diagnosis of BCa by most urological associations in the EU and USA to date.

At the beginning of its application, arm-robot manipulator worked blindly in completing the assigned task, and along with the improvement of camera technology as it is getting cheaper and smaller, the application of visual sensor lead to visual servoing control which utilized target detection as the input to control/move the end-effector of an arm-robot. The application of camera in arm-robot manipulator is divided into eye-in-hand where a camera is mounted to the robot, and eye-to-hand where the camera is fixed elsewhere. Visual control involves kinematics modeling and target position in image plane provided by image processing to ensure the motion accuracy in grasping and harvesting the fruit. The end-effector movement for harvesting relies heavily on the accuracy of target position detection that can be obtained by image processing method. This paper is a review study that discusses the design of visual servoing control of an arm-robot manipulator applied in agriculture starting from kinematics and inverse kinematics for controlling the motion and trajectory of the robot, and image processing for target detecting. An example of an image processing method is given by using Edge Detection method for detecting a dragon fruit simulated in SCILAB.

Image enhancement techniques and optical diagnostic tools have emerged in the quest to improve diagnostics and treatment in patients with upper urinary tract carcinoma (UTUC). The aim of the present report is to describe their mechanisms of action and to assess the current evidence on applicability and utility in UTUC. Image enhancement techniques including narrow band imaging, Image1 S, and photodynamic diagnosis aim at better visualization of malignant urothelium and especially flat tumours or carcinoma in situ which are often difficult to see with white light endoscopy. Conducted studies in the upper tract are scarce but first results show that tumour detection seems to be better for narrow band imaging and photodynamic diagnosis-guided inspection.Optical diagnostic tools as confocal laser endomicroscopy and optical coherence tomography aim at providing real time optical biopsies giving pathologic information. A pilot study of optical coherence tomography proved its feasibility to visualize, grade and stage low and high-grade UTUC. Better visualization of suspect lesions and optical diagnostic techniques providing real time optical biopsies aim to facilitate in higher diagnostic precision and optimal individualized treatment of our patients with UTUC. As the disease is rare, studies are scarce but indispensable for future implication of these techniques.

Comparative Study of Conventional and Ultrahigh-resolution Optical Coherence Tomography Imaging in Esophageal Neoplasia

The use of unmanned aerial vehicles (UAVs) for military, scientific, and civilian sectors are increasing drastically in recent years. This study presents algorithms for the visual-servo control of an UAV, in which a quadrotor helicopter has been stabilized with visual information through the control loop. Unlike previous study that use pose estimation approach which is time consuming and subject to various errors, the visual-servo control is more reliable and fast. The method requires a camera on-board the vehicle, which is already available on various UAV systems. The UAV with a camera behaves like an eye-in-hand visual servoing system. In this study the controller was designed by using two different approaches; image based visual servo control method and hybrid visual servo control method. Various simulations are developed on Matlab, in which the quadrotor aerial vehicle has been visual-servo controlled. In order to show the effectiveness of the algorithms, experiments were performed on a model quadrotor UAV, which suggest successful performance.

Optical coherence tomography (OCT) technology is analogous to ultrasound imaging, except that OCT employs light instead of sound. The non-invasive imaging method works by projecting light on test target and detecting the backscattering from the underlying layers. As the OCT technology is based on optical interference, the internal structural features and inhomogeneities induced by different refractive index contrast could be detected and displayed in the form of a gray scale or false color image. In this paper, a typical microfluidic device was produced and measured by a spectral domain OCT instrument. The internal dimensions of the lab-on-chip device were determined using the OCT imaging technology and were in agreement with results obtained with conventional confocal microscope. In order to study the effect of different refractive index contrast on OCT imaging, fluid with various refractive indexes was injected into the microfluidic channel respectively, and the acquired OCT images of the internal microfluidic channel were compared. The results demonstrate that optical coherence tomography could be used as a new metrology tool to determine the internal channel dimensions of lab-on-chip devices. Furthermore, the experiment results reveal the relations between the refractive index contrast and OCT image quality.

With the growing interest of integrating robotics into everyday life and industry, the requirements towards the quality and quantity of applications grows equally hard. This trend is profoundly recognized in applications involving visual perception. Whereas visual sensing in home environments tend to be mainly used for recognition and localization, safety becomes the driving factor for developing intelligent visual control algorithms. More specifically, a robot operating in a human environment should not collide with obstacles and executed motion should be as smooth as possible. Furthermore, as the environment is not known on beforehand, a high demand on the robustness of visual processing is a necessity. On the other hand, in an industrial setting, the environment is known on beforehand and safety is mainly guaranteed by excluding a human operator. Despite these reasons, and the fact that visual servoing has gained much attention from industry to become a standard solution for robotic automation tasks, applications are highly simplified. For example, methods such as visual fault detection are already a mature technique in industrial manufacturing, where a fixed camera observes a product (e.g., on a conveyor belt) and checks whether it meets certain requirements. These operations can be executed at a fairly high rate due to the simplicity of the system (e.g., static camera) and the simplification of the processing task (e.g., binary images). For both areas the identified difficulties are similar. Foremost, this is the slow nature of (robust) visual processing, in respect to the ever growing demand of increasing speed and reducing delay. These two application areas with analogous limitations motivate the design of more direct approaches of vision in visual control systems. Therefore, in order to meet the requirements for next generation visual control systems, this thesis presents approaches which employ visual measurements as a direct feedback to design constrained motion. First, for industrial robotics, in order to obtain the required positioning accuracy, the measurement and fixation system have to be highly rigid and welldesigned, implying high cost and long design time. By measuring the position of objects directly with a camera, instead of indirectly by motor encoders, the requirements of the measurement and fixation system are less demanding. Moreover, this motivates the miniaturization of the complete control system. The approach is validated in experiments on a simplified 2D planar stage (i.e., considerable friction, poor fixation), which attains similar performance compared to encoder-based positioning systems. Secondly, in a human-centred environment, this direct sensing can improve traditional visual control systems, when subject to certain disturbances. More specifically, a method is proposed that uses an image-based feedforward controller on top of traditional position-based visual servo control to overcome disturbances such as friction or poorly designed local motor controllers. This visual feedforward control action is only active when an image-based error is present and vanishes when that error goes to zero. The method is validated on an anthropomorphic robotic manipulator with 7 degrees of freedom, intended for operation in the human care environment. Third, sensing the product directly gives rise to designing motion directly. Whereas in traditional approaches the motion trajectory is designed offline and can not be changed at runtime, direct trajectory generation computes the motion of the next step based on current state and events. This means that at any instance in time, the trajectory of a motion system can be altered with respect to certain desired kinematic or dynamic constraints. For industrial applications this makes manufacturing on near-repetitive or non-rigid structures (e.g. flexible displays) possible. When applied to a robotic manipulator, this enables obstacle avoidance to no longer be on path planning level, but on trajectory planning level, where kinematic or dynamic constraints can be taken into account. This results in a motion that is smoother than when obstacle avoidance with path planning is employed. For both application areas this direct trajectory generation method is implemented and shows high flexibility in constrained motion trajectory design.

Epithelial cancers account for substantial mortality and are an important public health concern. With the need for earlier detection and treatment of these malignancies, the ability to accurately detect precancerous lesions has an increasingly important role in controlling cancer incidence and mortality. New optical technologies are capable of identifying early pathology in tissues or organs in which cancer is known to develop through stages of dysplasia, including the esophagus, colon, pancreas, liver, bladder, and cervix. These diagnostic imaging advances, together as a field known as optical endomicroscopy, are based on confocal microscopy, spectroscopy-based imaging, and optical coherence tomography (OCT), and function as “optical biopsies,” enabling tissue pathology to be imaged in situ and in real time without the need to excise and process specimens as in conventional biopsy and histopathology. Optical biopsy techniques can acquire high-resolution, cross-sectional images of tissue structure on the micron scale through the use of endoscopes, catheters, laparoscopes, and needles. Since the inception of these technologies, dramatic technological advances in accuracy, speed, and functionality have been realized. The current paradigm of optical biopsy, or single-area, point-based images, is slowly shifting to more comprehensive microscopy of larger tracts of mucosa. With the development of Fourier-domain OCT, also known as optical frequency domain imaging or, more recently, volumetric laser endomicroscopy, comprehensive surveillance of the entire distal esophagus is now achievable at speeds that were not possible with conventional OCT technologies. Optical diagnostic technologies are emerging as clinically useful tools with the potential to set a new standard for real-time diagnosis. New imaging techniques enable visualization of high-resolution, cross-sectional images and offer the opportunity to guide biopsy, allowing maximal diagnostic yields and appropriate staging without the limitations and risks inherent with current random biopsy protocols. However, the ability of these techniques to achieve widespread adoption in clinical practice depends on future research designed to improve accuracy and allow real-time data transmission and storage, thereby linking pathology to the treating physician. These imaging advances are expected to eventually offer a see-and-treat paradigm, leading to improved patient care and potential cost reduction.Virtual SlidesThe virtual slide(s) for this article can be found here:http://www.diagnosticpathology.diagnomx.eu/vs/5372548637202968

This paper focuses on the real-time target tracking for a Cartesian robot in 2 dimensions by using visual servo control. It is the process of controlling a system in servoloop by using visual image data as a referenced command to control the motion of a robot. The visual servo control can be divided into 2 loops. The first loop is image processing loop for tracking a target in video frame and the second loop is servo loop for feedback control of a Cartesian robot. A target tracking is based on color probability image using a Continuously Adaptive Mean Shift (CAMShift) algorithm that finds the most probable target position from color histogram in the current frame with a single stationary camera. The experimental results illustrate that a Cartesian robot is automatically follow a moving target and record it. The visual servo control can be adapted to more complex control applications.

In this paper we study image based visual servo (IBVS) control of a four rotor vertical take-off and landing (VTOL) craft known as an X4-flyer. The approach taken is tailored to planar visual targets and is based on a reformulation and extension of 2 1/2 D image-based visual servo design to the control of dynamic under-actuated systems. Exponential stability of the closed loop system is proved and simulation results are presented.

Oral cancer is among the most common malignancies worldwide, therefore early detection and treatment is imperative. The 5-year survival rate has remained at a dismal 50% for the past several decades. The main reason for the poor survival rate is the fact that most of the oral cancers, despite the general accessibility of the oral cavity, are not diagnosed until the advanced stage. Early detection of the oral tumors and its precursor lesions may be the most effective means to improve clinical outcome and cure most patients. One of the emerging technologies is the use of non-invasive in vivo tissue imaging to capture the molecular changes at high-resolution to improve the detection capability of early stage disease. This review will discuss the use of optical probes and highlight the role of optical imaging such as autofluorescence, fluorescence diagnosis (FD), laser confocal endomicroscopy (LCE), surface enhanced Raman spectroscopy (SERS), optical coherence tomography (OCT) and confocal reflectance microscopy (CRM) in early oral cancer detection. FD is a promising method to differentiate cancerous lesions from benign, thus helping in the determination of adequate resolution of surgical resection margin. LCE offers in vivo cellular imaging of tissue structures from surface to subsurface layers and has demonstrated the potential to be used as a minimally invasive optical biopsy technique for early diagnosis of oral cancer lesions. SERS was able to differentiate between normal and oral cancer patients based on the spectra acquired from saliva of patients. OCT has been used to visualize the detailed histological features of the oral lesions with an imaging depth down to 2–3 mm. CRM is an optical tool to noninvasively image tissue with near histological resolution. These comprehensive diagnostic modalities can also be used to define surgical margin and to provide a direct assessment of the therapeutic effectiveness.

This letter deals with the development of a vision based control law to achieve high-accuracy automatic six degrees of freedom (DoF) positioning tasks. The objective of this work is to be able to replace a biological sample under an optical device for a non-invasive depth examination at any given time (i.e., performing repetitive and accurate optical characterizations of the sample). The optical examination, also called optical biopsy, is performed thanks to an optical coherence tomography (OCT) system. The OCT device is used to perform a 3-dimensional optical biopsy, and as a sensor to control the robot motion during the repositioning process. The proposed visual servoing controller uses the 3D pose of the studied biological sample estimated directly from the C-scan OCT images using a Principal Component Analysis (PCA) framework. The proposed materials and methods were experimentally validated using a spectral-domain OCT and a 6-DoF robotic platform. The obtained results have demonstrated the pertinence of such methods which offer a positioning accuracy around 0.052 ± 0.03 mm (mean error ± standard deviation) for linear errors and 0.41 ± 0.16° for angular ones over a 8 × 9 × 3.5 mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> workspace.

Simple SummaryEarly diagnosis of digestive tract neoplasms is beneficial for maximising the possibility of cure. The diagnostic efficacy of conventional endoscopy is currently limited. Confocal laser endomicroscopy (CLE) can magnify tissue to the cellular level, which is beneficial for endoscopic biopsy. Currently, its clinical application has attracted wide interest among endoscopists. Many studies have explored its application potential in the early diagnosis of digestive tract tumours. This paper reviewed a large number of clinical studies on CLE in early esophageal and gastric cancers, providing more evidence-based support for endoscopists in making clinical decisions.Esophageal and gastric cancers are common diseases with high morbidity and mortality; thus, early detection and treatment are beneficial to improve prognosis. Confocal laser endomicroscopy (CLE) is a novel imaging technique that permits the histological analysis of tissues during endoscopy. CLE has been shown to uniquely affect the diagnosis of early upper gastrointestinal cancers. Relevant literature was searched using PubMed and Google Scholar databases. Despite inherent flaws, CLE can reduce tissue damage and improve diagnostic accuracy to a certain extent. CLE in combination with other imaging methods can help enhance the detection rate and avoid unnecessary biopsies in the management of esophageal or gastric cancer and precancerous lesions. CLE is of great significance in the diagnosis and surveillance of early cancers of the upper gastrointestinal tract. Further technical innovations and the standardisation of CLE will make it more responsive to the needs of routine clinical applications.

This paper is oriented to the transmission line inspection environment, aiming to solve the problem of how to control the robotic arm to quickly and accurately grasp the target object, and work on the planning of the motion path and the position-based visual servo control method. For the spatial trajectory planning of the robotic arm, linear trajectory planning and circular arc trajectory planning are carried out respectively. Between the reaction speed problem of the traditional RRT motion algorithm in collision-free motion planning, a two-way RRT-Connect path planning method is proposed. Then the visual servo control algorithm of the UAV robotic arm is proposed. The position-based visual servo control method is adopted, while the EK-SVSF position estimation algorithm and the OWA-based stepwise data fusion algorithm are combined for optimal position estimation of the target during transmission line inspection. It is verified in simulation experiments that the improved RRT algorithm can obtain the shortest path planning in a shorter time with better performance. Finally, the grasping experiment is carried out in a real robotic arm transmission line inspection scene. The robotic arm path planning algorithm and visual servo control proposed in this paper are combined to carry out grasping experiments in a static cluttered environment. The experimental results show that the algorithm proposed in this paper has the best grasping success rate, servo success rate, and grasping similarity. In addition, the average errors in x-axis, y-axis and z-axis of the proposed method are 0.19cm, 1.11cm and 0.18cm, respectively, which show a good grasping accuracy, indicating that the control method proposed in this paper can be applied to the real transmission line inspection work scenarios.