OTDR data augmentation and event detection using GAN under imbalanced samples

This work is concerned with research of Haar wavelet threshold sensitivity for detection of non-reflective events corresponding to optical fiber fusion splices on fragments of fiber optic link traces measured by optical time domain reflectometer (OTDR). We performed test series contained 40 step-by-step repeatable splicing of the same couple of standard singlemode fibers of ITU-T Rec. G.652 spools with length 2.2 km and 4.2 km and outer termination by conventional singlemode FC/UPC pigtails for OTDR connection and following bidirectional trace measurement. Results of insertion loss estimation and accomplished by OTDR software automated detection of non-reflective event corresponding to fusion splice of described optical fibers as well as ability of its visual manual localization on trace were recorded. The first 30 splices were realized by fusion splicer conventional program for jointing standard singlemode optical fibers of ITU-T Rec. ITU-T G.652, while the next 10 splices were done under “attenuator” mode with preset increased insertion loss. Described link traces were measured by OTDR operating in a standard mode with high spatial resolution under pulse width 20 ns and appropriate dynamic range with far-end OSNR 6.5 dB at wavelength 1550 nm. As a result 80 traces were analyzed in both automated and manual modes with fusion splice insertion loss varied from 0.002 dB to 0.554 dB. At the next stage we applied continues Haar wavelet transform for detection non-reflective event on measured traces of lab fiber optic link containing described one fusion splice with total length about 6.4 km. According to results, wavelet transform of described link trace provides detection of extremely low loss non-reflective events with OTDR reading down to own error corresponding to 0.020…0.030 dB range under improved scaling over normalized spatial diagram of wavelet coefficients.

Two methods employed during fusion splicing as well as two methods for calculating the average splice loss per fiber-span are presented. The two splicing methods described are the telemetry and optical time domain ref lectometer (OTDR) methods. The cut-back and OTDR measuring methods for calculating the average splice loss per fiber-span are also described. A statistical model relating the various measuring uncertainties with the sample size uncertainties is used to predict the population parameters from the sample statistics. Laboratory results are also included to analyze the important parameters needed for splice optimization. The accuracy of splice loss measurements obtained using OTDRs is studied in the laboratory and is used to analyze field data. Our results show that there is no significant difference in the quality of the telemetry splicing method as compared to the OTDR splicing method. In addition, the average splice loss per fiber-span calculated from cut-back measurements was found to yield a distribution that is only 80% as accurate as the one derived from the OTDR measurements.

Chapter 15 - Considerations when selecting an OTDR

An effective measurement method for investigating the transmission characteristics of an optical fiber segment using automated backscattering techniques is described in this paper. Three different Optical Time Domain Reflectometers (OTDRs) are employed in the manual and automated modes of operation to characterize several single mode fibers. Length dependent optical loss measurements are made on several precision-wound fiber optic bobbins (containing many layers) by setting the distance markers at the start and end locations which bound the desired fiber segment. Discrepancies are observed in the data acquired using the manual vs the automated procedure. Apparent round-off errors and inaccurate measurements are observed when operating the OTDR in the automated mode due to the elimination of the operator's observations and adjustments. Algorithms are written to automatically frame the fiber segment of interest by monitoring the actual OTDR marker settings and adjusting the marker setting request(s) when the fiber segment boundaries are violated by the OTDR. This novel framing technique ensures consistent and precise measurements. The technique can provide assistance in using OTDRs in the automated mode of operation.

Optical Time-Domain Reflectometer (OTDR) is the most common way for quality evaluation and fault location of optical fibers. Aiming at the problem of low detection rate caused by low sensitivity in traditional OTDR event detection methods, a novel detection method based on machine learning is proposed. Firstly, the OTDR signal is processed with n-order difference, then the differential signal is denoised. In order to reduce the computation complexity, the peaks of the differential signal are found and the features of the peaks are extracted. Finally, the features are labeled and sent to a machine learning based classifier for offline training. The trained model is used for online prediction to output the detected events. The algorithm is verified with 500 OTDR trace, the results show that the detection rate of connection splice events is up to 95% after 200 iterations. The sensitivity is much higher than that of the world-famous OTDR instrument.

The integration of DC networks including DC microgrids (DCMGs) into power systems is rapidly increasing. This is notably attributed to the distinctive fault current characteristics arising from inverter-based distributed generation resources in DCMGs, which differentiate them from conventional networks. As a result, the protection of DCMGs presents considerable challenges. Leveraging the recent strides in artificial intelligence, this paper introduces a novel multi-agent-based protection scheme for DC microgrids. Subsequently, three fault classification approaches are proposed in an intelligent protection scheme platform, employing diverse machine learning-based methods as a backup protection for fault detection and the main protection for fault location. The proposed protection scheme uses three main protection layers—namely, equipment, substation, and system—each endowed with specialized agents. In this way, the first and second fault classification approaches employ classifiers based on machine learning algorithms, such as Support Vector Machine (SVM) and Decision Tree (DT), to ascertain the microgrid status. Subsequent fault location is accomplished through various neural networks dedicated to the fault location. In the third approach, three Deep Neural Networks (DNNs) are proposed for fault classification, prompting the exclusion of classifiers from the substation layer due to the heightened training proficiency of DNNs. Intelligent Electronic Devices (IEDs) are placed at the beginning of the lines and send voltage and current information to the substation layer. Communication among agents and layers is performed by the IEC-61850 protocol. Comprehensive simulations and analyses are conducted using DIgSILENT, MATLAB, and Python (TensorFlow platform and Keras library) software. The findings underscore the efficacy of the proposed scheme and Fault Detection and Location (FDL) approaches, affirming their capability for precise fault classification and location determination.

Predictive models using microbiome data often suffer from covariate imbalance and class imbalance, biasing results. Propensity Score Matching (PSM) balances covariates but reduces sample size, while borderline synthetic minority oversampling technique (borderline-SMOTE) oversamples minority classes but can generate uninformative examples. To develop and evaluate PSM-SMOTE, a novel hybrid sampling method that integrates PSM and borderline-SMOTE to handle both covariate and class imbalance in microbiome data. We developed PSM-SMOTE, a three-step hybrid sampling algorithm for microbiome data: (1) PSM at four caliper levels to balance covariates, (2) selection of at least ten robust differential markers via seven statistical tests with false discovery rate correction, and (3) application of borderline-SMOTE on the marker-based distance matrix to oversample minority classes. We evaluated PSM-SMOTE on three publicly available microbiome case-control datasets: pancreatic ductal adenocarcinoma (PDAC), colorectal cancer (CRC), and obesity, using logistic regression (LR), random forest (RF), and support vector machine (SVM) classifiers. Performance was assessed via area under the ROC curve (AUC). PSM-SMOTE improved test AUCs in multiple model-dataset combinations compared with using PSM alone. Notably, for the RF model, PSM-SMOTE consistently enhanced AUC across nearly all oversampling settings in the PDAC and obesity cohorts. For the SVM model, PSM-SMOTE also achieved a significant AUC increase in the CRC cohort. For the LR model, PSM-SMOTE showed modest improvement under strict matching. PSM-SMOTE effectively addresses dual imbalance in microbiome data and consistently enhances performance, providing a practical solution for imbalanced data analyses.

A new optical time domain reflectometer is proposed that uses a semiconductor laser diode (LD) as a pulse emitter and a photodetector. By employing LDs with center wavelengths of 1.31 and 1.56 µm, fault location lengths longer than 10 km were attained in both wavelengths for a graded-index multimode fiber.

In recent years, a significant amount of research has focused on analyzing the effectiveness of machine learning (ML) models for malware detection. These approaches have ranged from methods such as decision trees and clustering to more complex approaches like support vector machine (SVM) and deep neural networks. In particular, neural networks have proven to be very effective in detecting complex and advanced malware. This, however, comes with a caveat. Neural networks are notoriously complex. Therefore, the decisions that they make are often just accepted without questioning why the model made that specific decision. The black box characteristic of neural networks has challenged researchers to explore methods to explain black-box models such as SVM and neural networks and their decision-making process. Transparency and explainability give the experts and malware analysts assurance and trustworthiness about the ML models’ decisions. In addition, it helps in generating comprehensive reports that can be used to enhance cyber threat intelligence sharing. As such, this much needed analysis drives our work in this paper to explore the explainability and interpretability of ML models in the field of online malware detection. In this paper, we used the Shapley Additive exPlanations (SHAP) explainability technique to achieve efficient performance in interpreting the outcome of different ML models such as SVM Linear, SVM-RBF (Radial Basis Function), Random Forest (RF), Feed-Forward Neural Net (FFNN), and Convolutional Neural Network (CNN) models trained on an online malware dataset. To explain the output of these models, explainability techniques such as KernalSHAP, TreeSHAP, and DeepSHAP are applied to the obtained results.

Traditional Image Processing Techniques (IPT), used for automating the detection and classification of weld defects from radiography images, have their own limitations, which can be overcome by Deep Neural Networks (DNN). DNN produces considerably good results in fields which offer big dataset for it to train. DNN trained with small datasets by conventional methods produces less accurate results. This limits the use of DNN in many fields. This study focuses to overcome this limitation, by adopting transfer learning using Pre-trained deep convolutional neural networks. By this method, a weld defect radiographic image classifier, which can classify 14 types of weld defects, was constructed. 940 Image patches of weld defects were manually collected and labelled from GDXray database. Subsequently the features of this weld defect dataset were extracted using VGG16 and ResNet50 CNNs, both pre-trained on the ImageNet database. Then machine learning models such as Logistic Regression, Support Vector Machine (SVM) and Random Forest were trained on these extracted features. The Classifier based on SVM trained on features extracted by ResNet50 outperforms the other counter parts with an accuracy of 98%. In all these cases, transfer learning improves performance and reduces the training time and computational system requirements.

This paper describes a coherent optical time domain reflectometer (OTDR) with a high dynamic range for locating faults in optical cable systems incorporating repeaters based on erbium-doped fiber amplifiers (EDFA). We also present experimental results achieved with transmission lines containing several fiber spans and EDFAs. >

Biomedical data are widely accepted in developing prediction models for identifying a specific tumour, drug discovery and human cancers detection. However, previous studies usually focused on different classifiers, and overlook the class imbalance problem in real-world biomedical datasets. This paper mainly focuses on reviewing and evaluating some popular and recently developed resampling and feature selection (FS) methods for class imbalance learning with data distribution being considered. Experimental results show that: 1) resampling and FS techniques exhibit better performance using support vector machine (SVM) classifier; 2) techniques such as random undersampling and FS perform better than other data pre-processing methods with T location-scale distribution when using SVM and K-nearest neighbours (KNN) classifiers. Random oversampling outperforms other methods on negative binomial distribution using Random Forest with lower level of imbalance ratio; 3) FS outperforms other data pre-processing methods in most cases, thus, FS with SVM classifier is the best choice for imbalanced biomedical data learning.

Optical fiber links are customarily monitored by Optical Time Domain Reflectometer (OTDR), an optoelectronic instrument that measures the scattered or reflected light along the fiber and returns a signal, namely the OTDR trace. OTDR traces are typically analyzed by experts in laboratories or by hand-crafted algorithms running in embedded systems to localize critical events occurring along the fiber. In this work, we address the problem of automatically detecting optical events in OTDR traces through a deep learning model that can be deployed in embedded systems. In particular, we take inspiration from Faster R-CNN and present the first 1D object-detection neural network for OTDR traces. Thanks to an ad-hoc preprocessing pipeline for OTDR traces, we can also identify unknown events, namely events that are not represented in training data but that might indicate rare and unforeseen situations that need to be reported. The resulting network brings several advantages with respect to existing solutions, as these typically classify fixed-size windows of OTDR traces, thus are less accurate in the localization. Moreover, existing solutions do not report events that cannot be safely associated to any label in the training set. Our experiments, performed on real OTDR traces, show very promising performance, and can be directly executed on embedded OTDR devices.

Classification of protein sequences is one big task in bioinformatics and has many applications. Different machine learning methods exist and are applied on these problems, such as support vector machines (SVM), random forests (RF) and neural networks (NN). All of these methods have in common that protein sequences have to be made machine-readable and comparable in the first step, for which different encodings exist. These encodings are typically based on physical or chemical properties of the sequence. However, due to the outstanding performance of deep neural networks (DNN) on image recognition, we used frequency matrix chaos game representation (FCGR) for encoding of protein sequences into images. In this study, we compare the performance of SVMs, RFs and DNNs, trained on FCGR encoded protein sequences. While the original chaos game representation (CGR) has been used mainly for genome sequence encoding and classification, we modified it to work also for protein sequences, resulting in n-flakes representation, an image with several icosagons. We could show that all applied machine learning techniques (RF, SVM and DNN) show promising results compared to the state-of-the-art methods on our benchmark datasets, with DNNs outperforming the other methods and that FCGR is a promising new encoding method for protein sequences. https://cran.r-project.org/. Supplementary data are available at Bioinformatics online.

<abstract><p>In the realm of machine learning, where data-driven insights guide decision-making, addressing the challenges posed by class imbalance in datasets has emerged as a crucial concern. The effectiveness of classification algorithms hinges not only on their intrinsic capabilities but also on their adaptability to uneven class distributions, a common issue encountered across diverse domains. This study delves into the intricate interplay between varying class imbalance levels and the performance of ten distinct classification models, unravelling the critical impact of this imbalance on the landscape of predictive analytics. Results showed that random forest (RF) and decision tree (DT) models outperformed others, exhibiting robustness to class imbalance. Logistic regression (LR), stochastic gradient descent classifier (SGDC) and naïve Bayes (NB) models struggled with imbalanced datasets. Adaptive boosting (ADA), gradient boosting (GB), extreme gradient boosting (XGB), light gradient boosting machine (LGBM), and k-nearest neighbour (kNN) models improved with balanced data. Adaptive synthetic sampling (ADASYN) yielded more reliable predictions than the under-sampling (UNDER) technique. This study provides insights for practitioners and researchers dealing with imbalanced datasets, guiding model selection and data balancing techniques. RF and DT models demonstrate superior performance, while LR, SGDC and NB models have limitations. By leveraging the strengths of RF and DT models and addressing class imbalance, classification performance in imbalanced datasets can be enhanced. This study enriches credit risk modelling literature by revealing how class imbalance impacts default probability estimation. The research deepens our understanding of class imbalance's critical role in predictive analytics. Serving as a roadmap for practitioners and researchers dealing with imbalanced data, the findings guide model selection and data balancing strategies, enhancing classification performance despite class imbalance.</p></abstract>