Crop mapping from image time series: Deep learning with multi-scale label hierarchies

Crop type identification and mapping have been a classic issue essential for agriculture characterisation and food security as well as crop health monitoring. Remote sensing was proven to be an efficient tool to define and study agriculture. The complementarity between optical and radar data can hugely improve crop type identification and mapping. This document presents an overview of more than 40 recent works between 2013 and 2019, related to crop type mapping using satellite imagery, and points the general techniques used. Different classification algorithms was studied and compared, as well as the comparison of the inputs, optimal bands and polarization for radar data, and process giving the best results. It is often concluded that fusion improves results when compared with single data using. Nevertheless, it had been challenging to combine datasets from sensors that had different physical concepts, and progress in this domain require the development of systematic procedures.

Business process modelling is an important part of system design. When designing or redesigning a business process, stakeholders specify, negotiate, and agree on business requirements to be satisfied, including non-functional requirements that concern the quality of the business process. This thesis addresses the question of how to specify and compute the quality of a business process, given the model that stakeholders use. The motivation for this thesis is the increasing importance of the quality of business processes. Knowing the quality of specific business processes enables stakeholders to judge if these processes need improvement. Knowing the quality of the constructs of those processes (viz., events, inputs, activities, and outputs) and the way they are structured enables a more detailed analysis of their shortcomings and provides a basis for the design of improvements. The research challenge of this thesis is grounded in the assumption that: “Organisations need an appropriate means to effectively compute achievement of their goals and objectives by their business processes.” Given this challenge, the main research question on which this thesis focuses is: “Can the quality of a business process be computed quantitatively at different levels of granularity?” The research objective is: “To develop frameworks, factors, and metrics for computing non-functional requirements (quality) of business processes quantitatively at different levels of granularity.” The outcomes of this thesis are: 1) BPIMM, a language-independent business process integrating meta-model, based on the concepts of seven mainstream business process modelling languages: BPMN, EPC, RAD, UML AD, SADT, IDEF0, and IDEF3. 2) BPC-QC (Business Process Concept - Quality Computation), an approach to quality computation at the lowest level of granularity of a business process. The approach consists of: i. BPC-QEF (Business Process Concept - Quality Evaluation Framework), a language-independent generic framework and algorithm to compute the quality of the constructs of a business process: event, input, activity, and output. ii. A set of business process quality dimensions and factors. The following quality dimensions are distinguished: performance, efficiency, reliability, recoverability, permissibility, and availability. Each dimension categorises different quality aspects in terms of factors. A non-exhaustive set of sixteen quantitative factors is provided. iii. Quality metrics for each of the quality factors, to facilitate a quantitative computation of the quality of a specific construct of a business process. 3) BP-QC (Business Process - Quality Computation), an approach to compute the quality at the highest level of granularity of a business process. The approach consists of: i. BP-CQCF (Business Process - Compositional Quality Computation Framework), a language-independent generic framework and algorithm to compute the quality of a business process as a whole, given the quality of its constructs. ii. A set of generic business process modelling patterns to decompose a business process into more succinct parts, namely: sequential, parallel with synchronisation, exclusive, inclusive, simple loop, and complex loop. iii. A set of over one hundred computational formulae. For each combination of modelling pattern and a quality factor, there is a formula to compute the quality. 4) AAV (Approach to Application and Validation), an evaluation plan to evaluate BPIMM, BPC-QC and BP-QC in practice, together with expert stakeholders. The plan consists of the units of measure, a measurement model, and a case study procedure. To evaluate the applicability of the contributions of this thesis to real world business needs, four case studies have been conducted in different environments: a Dutch educational institution, a global financial institution, an international financial service provider, and a Dutch research project on crisis management. Each of these case studies concerns a different, single business process. This thesis shows that: 1) A quality computation approach can be adopted independent of a business process modelling language. 2) Quantitative quality factors can be introduced specifically for the constructs of a business process. 3) Quantitative metrics and computational formulae can be developed for specific quality factors, allowing the computation of different aspects of the quality of a business process quantitatively at different levels of granularity. 4) An evaluation plan can be developed to evaluate the applicability of the contributions of this thesis (viz., BPIMM, BPC-QC, and BP-QC). The contributions of this thesis are designed to be beneficial to the areas of business and management, requirements engineering, software engineering, and business process modelling. In the areas of requirements engineering and software engineering, these contributions are intended to help practitioners to consider non-functional requirements at the earliest stage. In the area of business process modelling, information systems, service computing, and cloud computing, the contributions can be used for quality-driven modelling, design, and redesign. To conclude, knowing the quality value of a business process at different levels of granularity provides a basis for its improvement.

Crop-type mapping using time-series remote sensing data is crucial for a wide range of agricultural applications. Crop mapping during the growing season is particularly critical in timely monitoring of the agricultural system. Most existing studies focusing on within-season crop mapping leverage historical remote sensing and crop type reference data for model building, due to the difficulty in obtaining timely crop type samples for the current growing season. Yet the crop type samples from previous years may not be used directly considering the diverse patterns of crop phenology across years and locations, which hampers the scalability and transferability of the model to the current season for timely crop mapping. This article proposes an innovative within-season emergence (WISE) phenology normalized deep learning model towards scalable within-season crop mapping. The crop time-series remote sensing data are first normalized by the WISE crop emergence dates before being fed into an attention-based one-dimensional convolutional neural network classifier. Compared to conventional calendar-based approaches, the WISE-phenology normalization approach substantially helps the deep learning crop mapping model accommodate the spatiotemporal variations in crop phenological dynamics. Results in Illinois from 2017 to 2020 indicate that the proposed model outperforms calendar-based approaches and yields over 90% overall accuracy for classifying corn and soybeans at the end of season. During the growing season, the proposed model can give satisfactory performance (85% overall accuracy) one to four weeks earlier than calendar-based approaches. With WISE-phenology normalization, the proposed model exhibits more stable performance across Illinois and can be transferred to different years with enhanced scalability and robustness.

LiDAR data are becoming increasingly available, which has opened up many new applications. One such application is crop type mapping. Accurate crop type maps are critical for monitoring water use, estimating harvests and in precision agriculture. The traditional approach to obtaining maps of cultivated fields is by manually digitizing the fields from satellite or aerial imagery and then assigning crop type labels to each field – often informed by data collected during ground and aerial surveys. However, manual digitizing and labeling is time-consuming, expensive and subject to human error. Automated remote sensing methods is a cost-effective alternative, with machine learning gaining popularity for classifying crop types. This study evaluated the use of LiDAR data, Sentinel-2 imagery, aerial imagery and machine learning for differentiating five crop types in an intensively cultivated area. Different combinations of the three datasets were evaluated along with ten machine learning. The classification results were interpreted by comparing overall accuracies, kappa, standard deviation and f-score. It was found that LiDAR data successfully differentiated between different crop types, with XGBoost providing the highest overall accuracy of 87.8%. Furthermore, the crop type maps produced using the LiDAR data were in general agreement with those obtained by using Sentinel-2 data, with LiDAR obtaining a mean overall accuracy of 84.3% and Sentinel-2 a mean overall accuracy of 83.6%. However, the combination of all three datasets proved to be the most effective at differentiating between the crop types, with RF providing the highest overall accuracy of 94.4%. These findings provide a foundation for selecting the appropriate combination of remotely sensed data sources and machine learning algorithms for operational crop type mapping.

In traditional formal concept analysis, the attributes in the formal context are considered fixed. However, in the real world data set, attributes always have different levels of granularity, correspondingly, the derived concept lattice may reveal different information and patterns. Therefore, the capability to change the level of granularity of an attribute in formal concept analysis to capture relevant patterns in data is a natural requirement. In this paper, a theoretical study has been undertaken in multi-scale formal contexts, where attributes with different levels of granularity possess different attribute values. Two types of formal concepts, i.e., object-oriented and property-oriented multi-scale concepts, are introduced and studied in detail. The collection of object-oriented concept lattices and property-oriented concept lattices can be obtained at different granularity levels of attributes. It has been shown that the set of extents in the derived concept lattices increases when we choose to use a finer level of granularity. Moreover, a corresponding bidirectional approach to concept construction(i.e., from coarser to finer and from finer to coarser, respectively) is exhibited, and some characterization theorems have been obtained.

Conceptual knowledge is acquired through recurrent experiences, by extracting statistical regularities at different levels of granularity. At a fine level, patterns of feature co-occurrence are categorized into objects. At a coarser level, patterns of concept co-occurrence are categorized into contexts. We present and test CONCAT, a connectionist model that simultaneously learns to categorize objects and contexts. The model contains two hierarchically organized CALM modules (Murre, Phaf, & Wolters, 1992). The first module, the Object Module, forms object representations based on co-occurrences between features. These representations are used as input for the second module, the Context Module, which categorizes contexts based on object co-occurrences. Feedback connections from the Context Module to the Object Module send activation from the active context to those objects that frequently occur within this context. We demonstrate that context feedback contributes to the successful categorization of objects, especially when bottom-up feature information is degraded or ambiguous.

In this paper we present a system for automatic Arabic text diacritization using three levels of analysis granularity in a layered back off manner. We build and exploit diacritized language models (LM) for each of three different levels of granularity: surface form, morphologically segmented into prefix/stem/suffix, and character level. For each of the passes, we use Viterbi search to pick the most probable diacritization per word in the input. We start with the surface form LM, followed by the morphological level, then finally we leverage the character level LM. Our system outperforms all of the published systems evaluated against the same training and test data. It achieves a 10.87% WER for complete full diacritization including lexical and syntactic diacritization, and 3.0% WER for lexical diacritization, ignoring syntactic diacritization.

Data warehousing is widely used in industry for reporting and analysis of huge volumes of data at different levels of detail. In general, data warehouses use standard dimensional schema designs to organize their data. However, current data warehousing schema designs fall short in their ability to model the multi-granular data found in various real-world application domains. For example, modern farm equipment in a field produces massive amounts of data at different levels of granularity that has to be stored and queried. A study of the commonly used data warehousing schemas exposes the limitation that the schema designs are intended to simply store data at the same single level of granularity. This paper on the other hand, presents several extended dimensional data warehousing schema design alternatives to store both detail and aggregated data at different levels of granularity. The paper presents three solutions to design the time dimension tables and four solutions to design the fact tables. Moreover, each of these solutions is evaluated in different combinations of the time dimension and the fact tables based on a real world farming case study.KeywordsMulti-granular datadata warehousingdata warehousing schema design

Geo‐SOMis a useful geovisualization technique for revealing patterns in spatial data, but is ineffective in supporting interactive exploration of patterns hidden in differentGeo‐SOMsizes. Based on the divide and group principle in geovisualization, the article proposes a new methodology that combinesGeo‐SOMand hierarchical clustering to tackle this problem.Geo‐SOMwas used to “divide” the dataset into several homogeneous subsets; hierarchical clustering was then used to “group” neighboring homogeneous subsets for pattern exploration in different levels of granularity, thus permitting exploration of patterns at multiple scales. An artificial dataset was used for validating the method's effectiveness. As a case study, the rush hour motorcycle flow data inTaipeiCity,Taiwan were analyzed. Compared with the best result generated solely byGeo‐SOM, the proposed method performed better in capturing the homogeneous zones in the artificial dataset. For the case study, the proposed method discovered six clusters with unique data and spatial patterns at different levels of granularity, while the originalGeo‐SOMonly identified two. Among the four hierarchical clustering methods,Ward's clustering performed the best in pattern discovery. The results demonstrated the effectiveness of the approach in visually and interactively exploring data and spatial patterns in geospatial data.

Due to the rapid development of social media networks, as well as the recent advances on smartphone technologies, users stopped being only consumers and became data producers. It enabled a new form of sensing, engaging a variety of studies using Location-Based Social Networks (LBSN) data. For instance, in device-to-device(D2D) communication systems, the user location is required to make mobile data offloading possible. However, getting the right location is a challenging task, and the analyses of LBSN datasets provide large-scale studies at a low-cost. On the other hand, depending the the level of granularity used on the analyses, it may result in the success or failure of the prediction. Therefore, in this paper, we show the importance of combining the different levels of granularities for studying user behavior in order to obtain a robuster analysis.

MODIS phenology-derived, multi-year distribution of conterminous U.S. crop types

This paper introduces the concept of granularity into routine dynamics studies. To establish levels of granularity we develop a phenomenological perspective on action, the central category of routine dynamics studies. By conducting an extensive literature review we show that routine dynamics scholars have at least unconsciously applied diverging levels of granularity in their empirical studies. We argue that side-stepping questions of explicitly addressing granularity significantly impedes the comparability of studies since different levels of granularity bring different aspects of performing and patterning processes into focus. Following this, we develop a methodological framework that allows scholars to establish and report on the level of granularity in their study. Finally, we sketch out some implications of taking granularity seriously for writing up research on routine dynamics.

Conformance checking provides a set of techniques for comparing observed behavior (event logs) with process models. Health care providers apply it for verifying compliance between activities realized by the patients and clinical protocols. However, health care processes are characterized by high complexity and variability, and clinical protocols can have a level of granularity different from the one in the recorded events. We proposed a new method for configuring the cost function for the alignment-based conformance checking, handling the different levels of granularity to address this problem. The costs are configured based on grouping activities and the specification of undesirable events. We applied this method using the ProM Framework in a real-world dataset containing events from patients who realized a bariatric surgery in 2016. The technique can also be applied in other conformance checking scenarios, where the granularity of the events differs from the process model. The case study result allowed us to automatically identify critical violations of the clinical protocol by analyzing the fitness metric. The proposed method could help auditors to identify crucial cases.

Timely and accurate crop mapping is crucial for providing essential data support for agricultural production management. Reliable ground truth samples form the foundation for crop mapping using remote sensing imagery, a task that presents significant challenges in regions with limited sample availability. To address this issue, this study evaluates instance-based transfer learning methods, using the Hexi Corridor as a case study to explore crop mapping strategies in areas with scarce samples. High-confidence pixels from the United States Cropland Data Layer (CDL), along with high-density time series data derived from Sentinel-1, Sentinel-2, and Landsat-8 satellite imagery, as well as key vegetation indices, were selected as training samples for the source domain. Various algorithms, including Random Forest (RF), Extreme Gradient Boosting (XGBoost), and TrAdaBoost, were employed to transfer knowledge from the source domain to the target domain for crop type mapping. The results demonstrated that during the transfer learning process using only source domain data—without utilizing any target domain data—the overall classification accuracy reached 73.88%, with optimal accuracies for maize and alfalfa at 88.97% and 85.23%, respectively. As target domain data were gradually incorporated, the total accuracy for all models ranged from 0.77 to 0.92, with F1-scores ranging from 0.76 to 0.92, showing a consistent improvement in model performance. This study highlights the feasibility of employing transfer learning for crop mapping in the Hexi Corridor, demonstrating its potential to reduce labeling costs for target domain samples and providing a valuable reference for crop mapping in regions with limited sample availability.

Crop type information is essential for many practical applications, yet its mapping is often constrained by inherent characteristics of most farming areas, such as fragmentation and small farm plots, changes in crop morphology across the season, and cloud cover. This study investigated whether these limitations could be overcome by using time-series of Synthetic Aperture Radar (SAR) and crop phenological information combined with different Dynamic Time Warping implementation strategies. Focusing on a fragmented landscape with small farm plots in the Netherlands, we used Sentinel-1 dual polarimetry (VV + VH) and TerraSAR-X single polarimetry (HH) images and the Dutch Basic Registration of Crop Plots (BRP) dataset for training and validation. Image pre-processing was followed by the generation of radar vegetation indices and polarimetric decomposition. Crop-specific responses to incident radar signal were analyzed, as well as the accuracy of the crop classification by Time-Weighted Dynamic Time Warping (twDTW) using either backscatter bands only or backscatter bands in combination with derived indices and polarimetric decomposition features. In addition, two further modified Dynamic Time Warping strategies, namely Variable Time Weight Dynamic Time Warping (vtwDTW) and Angular Metric for Shape Similarity (AMSS), were tested for their performance. Furthermore, we investigated the accuracy performance of a decision level fusion of TerraSAR-X and Sentinel-1 classification outputs. Results show that even in a fragmented landscape with relatively small plots (around 0.08 ha), crop types can be successfully mapped by using decision level fusion of the twDTW results of both sensors, reaching an accuracy of 77.1%. When using twDTW on Sentinel-1 (VV + VH) only, including Ratio, MRVI, and DPSVI indices, overall accuracy reached 69.5%; without those indices, accuracy was slightly lower (67.5%). Merging six different grain crops with similar leaf geometry into winter grains and summer grains improved classification accuracy to 80.6%. Our findings demonstrate that twDTW on SAR imagery allows to map crop types in fragmented landscapes with relatively small farm plots, offering potential for crop type mapping in areas with smallholder farming.