Performance Evaluation of Object Detection Models in Maritime Applications

Decision analytic models play an increasingly important role in the economic evaluation of health technologies. Given uncertainties around the assumptions used to develop such models, several guidelines have been published to identify and assess 'best practice' in the model development process, including general modelling approach (e.g., time horizon), model structure, input data and model performance evaluation. This paper focuses on model performance evaluation. In the absence of a sufficient level of detail around model performance evaluation, concerns regarding the accuracy of model outputs, and hence the credibility of such models, are frequently raised. Following presentation of its components, a review of the application and reporting of model performance evaluation is presented. Taking cardiovascular disease as an illustrative example, the review investigates the use of face validity, internal validity, external validity, and cross model validity. As a part of the performance evaluation process, model calibration is also discussed and its use in applied studies investigated. The review found that the application and reporting of model performance evaluation across 81 studies of treatment for cardiovascular disease was variable. Cross-model validation was reported in 55% of the reviewed studies, though the level of detail provided varied considerably. We found that very few studies documented other types of validity, and only 6% of the reviewed articles reported a calibration process. Considering the above findings, we propose a comprehensive model performance evaluation framework (checklist), informed by a review of best-practice guidelines. This framework provides a basis for more accurate and consistent documentation of model performance evaluation. This will improve the peer review process and the comparability of modelling studies. Recognising the fundamental role of decision analytic models in informing public funding decisions, the proposed framework should usefully inform guidelines for preparing submissions to reimbursement bodies.

The Predictive Analytics Toolkit (PAT) was developed to facilitate use of new approach methodologies (NAMs) to predict health hazards and risks from chemicals. PAT is a user-friendly web application that integrates many R packages to enable development and testing of prediction models without any programming. We drew from the work of Ring et al. 2021 (https://doi.org/10.1016/j.comtox.2021.100166), who used random forest models to predict in vivo transcriptomic responses in rat liver from in vitro Tox21 AC50 values for a set of 221 chemicals. Gene ontologies helped identify 735 biological pathways based on differential in vivo expression of specific gene sets. Ring et al. used 12 models that varied in use of toxicokinetics to predict in vivo activity using 5000 random forest iterations for each chemical/pathway combination (the area under the receiver-operator characteristic curve (AUC-ROC) was the measure of model performance). The highest-ranking model (Model 10) used Tox21 AC50 nominal concentrations converted to media concentrations and in vivo doses converted to circulating plasma concentrations; the lowest ranking model (Model 2) used nominal in vitro concentrations and administered in vivo dose levels. Using a subset of 10 pathways from the Ring et al. data, we used PAT to predict the AUC-ROC and to compare the best (Model 10) and worst (Model 2) performing models with only 100 random forest iterations. Using the results from PAT, Model 10 “won” in 60% of the comparisons, a value similar to that calculated for the identical set of comparisons using the supplemental data from Ring et al. (52.2%). Hence, PAT can provide a useful alternative to programming in R for prediction modeling and model performance evaluation, even for extensive genomic data sets.

Evaluating the spatial and temporal model performance of distributed hydrological models is necessary to ensure that the simulated spatial and temporal patterns are meaningful. In recent years, spatial and temporal remote sensing data have been increasingly used for model performance evaluation. Previous studies, however, have focused on either the temporal or spatial model performance evaluation. In addition, temporal (or spatial) model performance evaluation is often conducted in a spatially (or temporally) lumped approach. Here, we evaluated (1) the temporal model performance evaluation in a spatially distributed approach (spatiotemporal) and (2) the spatial model performance in a temporally distributed approach (temporospatial). We further demonstrated that both spatiotemporal and temporospatial model performance evaluations are necessary since they provide different aspects of the model performance. For this, a case study was developed using the Soil and Water Assessment Tool (SWAT) for the Upper Baitarani catchment in India, and the spatiotemporal and temporospatial model performance was evaluated against three different remotely based actual evapotranspiration (ETa) products (MOD16 A2, SSEBop, and TerraClimate). The results showed that an increase in the spatiotemporal model performance would not necessarily lead to an increase in the temporospatial model performance and vice versa, depending on the evaluation statistics. Overall, this study has highlighted the necessity of a joint spatiotemporal and temporospatial model performance evaluation to understand/improve spatial and temporal model behavior/performance.

This study proposes a novel pipeline that starts with NIR video data to advance the field of near-infrared (NIR) object detection. NIR video sequences must be split into individual frames to analyze the environment accurately. These frames are then produced and structured for the cutting-edge object identification model YOLOv8. The formatted NIR frames are then used to train the YOLOv8 model, enabling reliable and effective object recognition in complicated maritime settings. Unmanned surface vehicles (USVs) and autonomous maritime systems have a wide range of potential applications, and the integration of NIR video data and YOLOv8 promises improved object detection capabilities, particularly in low-light and challenging settings. Combining NIR imagery, frame conversion, and YOLOv8 training can dramatically enhance USVs' navigational capabilities and make them more effective in maritime conditions. This study aims to increase autonomy and safety in maritime applications, from surveillance and environmental monitoring to logistics and search and rescue operations.

This research develops a performance evaluation model of humans in avoiding ship collision situations compared to the AI ship collision avoidance system (CAS). A human, the remote operator (RO) of Maritime Autonomous Surface Ships (MASS), ought to make compact collision avoidance (CA) decisions to secure safety and efficiency during remote operations. Hence, evaluating RO’s CA performance is important; however, research on developing evaluation methods concentrates merely on evaluating trainees based on instructor’s guidelines, while artificial intelligence (AI) makes CA decisions through parameter-based calculation. Therefore, this research proposes a CA performance evaluation model for RO of MASS based on CAS intervening of RO trainee’s simulation training in ship collision avoidance. In each time interval, the RO’s decisions showed divergent behaviors under given situational conditions compared to the CAS’s behaviors in equivalent situations. Findings denote the benefits of CAS intervention methods and RO performance evaluation model based on the proposed performance features.

A balanced scorecard approach to establish a performance evaluation and relationship model for hot spring hotels based on a hybrid MCDM model combining DEMATEL and ANP

The Human Toxicity Potential (HTP) is a quantita tive toxic equivalency potential (TEP) that has been introduced previously to express the potential harm of a unit of chemical released into the environment. HTP includes both inherent toxicity and generic source-to-dose relationships for pollutant emissions. Three issues associated with the use of HTP in Life Cycle Impact Assessment (LCIA) are evaluated here. First is the use of regional multimedia models to define source-to-dose relationships for the HTP. Second is uncertainty and variability in sourceto-dose calculations. And third is model performance evaluation for TEP models. Using the HTP as a case study, we consider important sources of uncertainty/variability in the development of source-to-dose models — including parameter variability/uncertainty, model uncertainty, and decision rule uncertainty. Once sources of uncertainty are made explicit, a model performance evaluation is appropriate and useful and thus introduced. Model performance evaluation can illustrate the relative value of increasing model complexity, assembling more data, and/or providing a more explicit representation of uncertainty. This work reveals that an understanding of the uncertainty in TEPs as well as a model performance evaluation are needed to a) refine and target the assessment process and b) improve decision making.

Performance appraisal of manager is important in human resources management, which is still uncertainty and fuzziness. This paper in the construction of enterprise middle-level managers' performance appraisal evaluation index, proposes an enterprise middle-level managers' performance evaluation model under uncertain information, and examples to verify the feasibility and rationality of the method.

Solid oxide fuel cells for shipping: A machine learning model for early detection of hazardous system deviations

The theory of evidence is widely used to model uncertainty in many applications. Representation of relationship between evidence and hypothesis space are not obvious and is to be explored in many practical cases. Evidential mappings as relationship models are sort of belief assignments. The assignments engage binary and fuzzy sets with crisp, interval and fuzzy valued masses. In all cases reasoning mechanism available in Dempster-Shafer theory is used to depict relation between the mentioned spaces. In the paper interval valued masses with binary and fuzzy evidence representations are presented. The concept is exploited in order to solve floating objects detection.

PurposeThis study aimed at developing and testing a model to evaluate employee performance in Isfahan municipality.Design/methodology/approachA mixed-method design is applied in this study. To extract the model, a semi-structured interview based on the thematic analysis approach was employed. The qualitative data were obtained using a researcher-made questionnaire from a sample of 12 municipal experts selected based on purposive sampling. In the quantitative phase, the sample consisted of 76 managers and interim managers. The validity of the questionnaire was determined by the content validity index, while the structural validity was tested based on structural equation modeling using SmartPLS software. The reliability of the questionnaire was confirmed using Cronbach's alpha and composite reliability indices.FindingsThe factors obtained in the qualitative model included performance evaluation criteria, the desired time interval for performance evaluation, results announcement, performance evaluation approach, performance evaluation method and evaluator-related variables. There should have been an agreement between evaluators and those who were evaluated in all components of the model. In the quantitative section, performance evaluation criteria, evaluators, the evaluation method and time interval were confirmed with coefficients of 0.871, 0.815, 0.646 and 0.615, respectively.Practical implicationsThe novelty of this study is that it uses a mixed-method research approach to extract a performance evaluation model that is specific to the Isfahan municipality.Originality/valueThe novelty of this study is that it uses a mixed-method research approach to extract a performance evaluation model that is specific to the Isfahan municipality.

With the development of northbound transport network management interface technology, the performance of interface is more and more concerned. The standard of evaluating the performance consist of many factors like function completeness, structure complexity, syntax accuracy, reliability and so on. However, how to manage and evaluate certain network interface models is always a perplexing problem. At present, most evaluation methods of network interface performance still rely on expert experience after deployment and use, resulting in a lot of waste of time and cost. Since the generation of YANG model provides a new idea for operating unified model of different network architectures and standardization of platform interfaces, it is necessary to establish an interface model performance analysis and evaluation tool to measure the performance of YANG models of transport network management interface. This report raises a transport network northbound management interface evaluation system. The system supports users evaluating certain network interface immediately. This evaluation system reduces reliance on expert experience for interface performance evaluation and the cost of time compared with traditional methods.KeywordsTransport network management interfaceYANGEvaluationSpringBoot frameworkPyang

Asset management performance evaluation is an important part of enterprise asset management. In order to improve asset management performance evaluation system of distribution network (DN) in China, a new performance evaluation model of asset life cycle management(LCM) is proposed. Firstly, through an in-depth analysis of development characteristics of DN and the various demands of users in China, based on interface between users and DN, a comprehensive performance evaluation system is constructed for asset LCM covering economy, service efficiency and social benefits and other aspects. According to the purpose of enterprise and the characteristics of each index, an optimal combination weighting method, which takes both subjective and objective factors into consideration, is used to weight the service efficiency and social benefit indexes. Based on this, the performance evaluation model and calculation method of DN asset management are established. Finally, the evaluation method is applied to the performance evaluation of annual assets management of the DN in 14 counties of a municipal power company.

Sea turtles exhibit high migratory rates and occupy a broad range of habitats, which in turn makes monitoring these taxa challenging. Applying deep learning (DL) models to vast image datasets collected from citizen science programs can offer promising solutions to overcome the challenge of monitoring the wide habitats of wildlife, particularly sea turtles. Among DL models, object detection models, such as the You Only Look Once (YOLO) series, have been extensively employed for wildlife classification. Despite their successful application in this domain, detecting objects in images with complex backgrounds, including underwater environments, remains a significant challenge. Recently, instance segmentation models have been developed to address this issue by providing more accurate classification of complex images compared to traditional object detection models. This study compared the performance of two state-of-the-art DL methods namely; the object detection model (YOLOv5) and instance segmentation model (YOLOv5-seg), to detect and classify sea turtles. The images were collected from iNaturalist and Google and then divided into 64% for training, 16% for validation, and 20% for test sets. Model performance during and after finishing training was evaluated by loss functions and various indexes, respectively. Based on loss functions, YOLOv5-seg demonstrated a lower error rate in detecting rather than classifying sea turtles than the YOLOv5. According to mean Average Precision (mAP) values, which reflect precision and recall, the YOLOv5-seg model showed superior performance than YOLOv5. The mAP0.5 and mAP0.5:0.95 for the YOLOv5 model were 0.885 and 0.795, respectively, whereas for the YOLOv5-seg, these values were 0.918 and 0.831, respectively. In particular, based on the loss functions and classification results, the YOLOv5-seg showed improved performance for detecting rather than classifying sea turtles compared to the YOLOv5. The results of this study may help improve sea turtle monitoring in the future.

Mutual Fund Families and Performance Evaluation