OBH-RSI: Object-Based Hierarchical Classification Using Remote Sensing Indices for Coastal Wetland

Zhaoyang Lin, Jianbu Wang, Wei Li, Xiangyang Jiang,Wenbo Zhu, Yuanqing Ma, Andong Wang

Abstract: With the deterioration of the environment, it is imperative to protect coastal wetlands.Using multi-source remote sensing data and object-based hierarchical classification to classify coastal wetlands is an effective method. The object-based hierarchical classification using remote sensing indices (OBH-RSI) for coastal wetland is proposed to achieve fine classification of coastal wetland. First, the original categories are divided into four groups according to the category characteristics. Second, the training and test maps of each group are extracted according to the remote sensing indices. Third, four groups are passed through the classifier in order. Finally, the results of the four groups are combined to get the final classification result map. The experimental results demonstrate that the overall accuracy, average accuracy and kappa coefficient of the proposed strategy are over 94% using the Yellow River Delta dataset.

Keywords: Yellow River Delta; vegetation index; object-based; hierarchical classification; wetland;multi-source remote sensing

1 Introduction

Wetland is called “the Kidney of the Earth”. Its role is to protect biodiversity, regulate the atmosphere, purify water quality, and provide food and industrial raw materials. China’s coastal wetland area is 5 795 900 hectares, accounting for 10.85% of the country’s total wetland area [1].Since the 1960s, the area of China’s coastal wetlands has decreased by about 50% [1]. Although the causes of wetland degradation include climate change, sea level rise, biological invasion,marine disasters and hurricanes and other natural factors, the most important factor causing degradation is human activities [1]. In order to grasp the status of wetland in real-time, it is very necessary to carry out real-time monitoring and classification of wetland species.

Multispectral images are the most used data source for wetland remote sensing classification [2].Before the rise of hyperspectral remote sensing,multispectral sensor was the data source most used by researchers to study wetlands. Hyperspectral remote sensing imagery is one of the emerging and most prominent achievements in the field of remote sensing in the past 30 years.In previous studies, Sentinel-2, Landsat MSS,Landsat TM, Landsat ETM+ and SPOT data have been widely used in the mapping and classification tasks of coastal wetlands [3−5]. The classification of hyperspectral images has subsequently become the focus of research by researchers. Due to the revolutionary development of hyperspectral images in the spectral dimension, researchers first only used the spectral dimension for classification. The classic maximum likelihood classification method was first applied to the classification of hyperspectral and multispectral images [6]. Support vector machines(SVM) are also quickly used by researchers in classification tasks due to their fast and robust characteristics [7]. Decision tree classification algorithms, especially the classification and regression tree (CART), quickly demonstrated its effectiveness and robustness in hyperspectral and multispectral image classification [8]. The development of ensemble learning and the advantages of decision trees have led to the invention of the random forest (RF), and great progress has been made in the classification of hyperspectral and multispectral images [9]. With the success of deep learning in computer vision and image classification, researchers have introduced classic models such as multilayer perceptrons (MLP) and convolutional neural networks (CNN) into the field of hyperspectral and multispectral image classification. Especially CNN has made great achievements in the field of image processing, and it is also used the most in the classification of hyperspectral and multispectral images. In the field of coastal wetlands, CNN has also been applied to wetlands and achieved good results [10].

But a single multi-spectral or single-spectrum image is not perfect. For example, the contradiction between the spatial resolution and the spectral resolution of hyperspectral technology cannot be achieved at the same time, the problem of insufficient spectral resolution of the multi-spectral technology, and the problems of the same spectrum and the same spectrum of foreign matter.

In order to solve these problems, researchers use a method of fusing multi-source remote sensing data. By using remote sensing images with different spatial and spectral resolutions, researchers can combine the advantages of different data. Xu et al. combined multispectral laser radar (MS-LIDAR) data, hyperspectral data and very high resolution (VHR) data and used fully convolutional networks to win the first place in the 2018 Data Fusion Contest [11]. Based on one enhanced thematic mapper (ETM) data in 2001,one China-Brazil earth resources satellite(CBERS) data in 2004 and one advanced visible and near infrared radiometer (AVNIR) data in 2007 of the study area, Zhang et al. used the decision tree classification method to classify the ground features of the Anguinao Lake wetland [12].Based on LANDSAT TM/ETM+ imagery combined with DEM data, Ji et al. used normalized vegetation index and SVM to map wetlands [13].

Although multi-source data have effectively alleviated the problems mentioned above, there are still some problems that plague researchers.At present, most classifiers can only classify according to the characteristics of the samples themselves, and cannot consider the relationship between the samples and expert knowledge.Moreover, if only hyperspectral and multispectral data are used, the noise in the data affects the accuracy of classification. In order to solve this problem, researchers have proposed an object-based method. The biggest difficulty of object-based methods is the accuracy of segmentation [14]. Mohammadimanesh et al. used a new hierarchical object-based classification algorithm to map the wetlands, and the classifier was RF [15].Huang and Jia integrated region-based segmentation and cluster-based segmentation to obtain hybrid image segmentation [16].

However, the general object-based method of classifier is relatively backward, and it rarely combines vegetation index and expert knowledge.Many researchers use vegetation indices to monitor the types and distribution of plants on the ground，but they rarely combine vegetation index with expert knowledge. So, this paper combines vegetation index with expert knowledge,and use deep learning methods to improve traditional classifiers.

In this paper, an object-based hierarchical classification using remote sensing indices for coastal wetland (OBH-RSI) is proposed. The main work is as follows:

1) The remote sensing indices are designed to get segmentation and the entire training set and testing map are divided into several groups.The last group is the entire training set and test set. The purpose of dividing the training set and the test set into four groups with the remote sensing index is to correct the obvious errors of the expert experience due to noise on the basis of the classifiers trained on all the training sets,that is the last group.

2) The hierarchical classification is developed to classify each group separately in order. The hierarchical classification system is mainly composed of four proposed modules,which are threshold and morphology module, extraction and elimination module, two-branch inception net and integration module. The performance of the proposed segmentation method and the classifier are excellent, and both have improved after comparing with the classic method.

2 Study Area and Datasets Description

The Yellow River Delta Nature Reserve(36°55'–38°16'N, 117°31'–119°18'E) is located in Dongying City, Shandong Province. The original data and selected labeled samples are shown in Fig.1. The chosen experimental data are sentinel 2A and GF-5 data. Sentinel 2A has 13 bands and GF-5 has 330 bands. Since GF-5 has some bad bands, we only kept 285 of them. Sentinel 2A has a spatial resolution of 10 m and GF-5 image has a spatial resolution of 30 m. The land cover in the study area is greatly affected by salinization,and the vegetation is mainly a whole plant community dominated by grass and shrubs [17, 18].The data collection time is November 2018.

In this research area, we selected 20 categories for experiments. As shown in Tab. 1, there are not only plants unique to wetlands, but also different types of land and water. Mix area 1 is composed oftamarixandsuaedasalsa. Mix area 2 is composed of phragmitescommunisand tamarisk Mix area 3 is composed oftamarix, phragmitescommunisandsuaedasalsa. According to field investigation, this kind of mixing phenomenon does exist and has classification value.

Tab. 1 Classes and quantity of training set and test set

3 Proposed OBH-RSI Framework

As shown in Fig.2, the proposed method has four main modules. They are threshold and morphology module, extraction and elimination module,two-branch inception net and integration module.A vegetation index (VI) is radiation value that reflects the relative abundance and activity of green vegetation [19]. The vegetation index is a simple, effective and empirical measure of the vegetation status on the ground. At present, more than 40 vegetation indexes have been defined.Three remote sensing indices such as the normalized difference vegetation index (NDVI) [19], differential vegetation index (DVI) [20] and normalized difference water index (NDWI) [21] are mainly used.

Fig.2 The overall flow of the proposed OBH-RSI framework

3.1 Threshold and Morphology Module

Our goal is to divide the original 20 categories into several groups. Based on expert knowledge and field experience, we use remote sensing indices as the basis for segmentation. First of all,because the Yellow River Delta is a coastal wetland, the main feature types can be roughly divided into ground and water. Since NDVI and DVI are greatly affected by soil, the ground is divided into ground with low vegetation coverage and ground with high vegetation coverage.Moreover, in the various distributions of field observation and remote sensing index, Spartina alterniflora has a large coverage and excludes the growth of other plants. And Spartina alterniflora is easy to identify in NDWI.

So, the entire 20 categories are divided into four groups. As shown in Tab. 2, they are Spartina alterniflora group, high-coverage vegetation group, low-coverage vegetation group and water group. We name these groups group 1 to group 4 in order. They are extracted from NDWI, DVI,NDVI and NDVI respectively. The following is how we get the masks of all groups. Finally, in order to prevent the previous four groups from missing the training set or test set and improve accuracy, the entire training set and test set are called group 5. Fig.3 is a graphical display of the threshold selection in Tab. 2, where the lines represent the probability distribution of the combination of groups. The following is the detailed process of this module.

Tab. 2 Name, content and extraction index of groups

1) GF-5 data are employed to calculate three remote sensing indexes. According to normal distribution, the distribution of each category in each index is obtained.

2) The probability distribution of each class is combined and found to be very similar to the normal distribution. So, the normal distribution fitting function is used to approximate each class.For each index, according to the grouping in Tab. 2, each group is added up. In this way, the distribution of each group is obtained. As shown in the Fig.3, the threshold is selected based on the intersection of the two sets of probability densities.

Fig.3 Probability distribution and threshold selection for each group: (a) extract the NDWI of group 1; (b) extract the DVI of group 2;(c) extract the NDVI of group 3-4. The abscissa x represents the value of the vegetation index and the ordinate f(x) represents the sum of the probability density of x for each line. X is the thresholds we choose in Tab. 2

3) The thresholds between adjacent groups are found. The remote sensing index maps extracting each group are binarized according to the thresholds and morphology and connected domain methods are used to get the masks of each group. The mathematical morphology and connected domain operations in the threshold and morphology module are shown in the Tab. 3.Among them, the “Bwareaopen” in the second column refers to the connected component filtering operation, and the “Dilate” refers to the morphological dilation operation. Connected domain filtering refers to the operation of removing small pixel blocks whose selected value is less than the threshold value in a binary image. “Dilate” is the operation of finding the local maximum.

Tab. 3 Morphology and connected domain operations are in the threshold and morphology module

3.2 Extraction and Elimination Module

The main function of this module is to extract the training set and test set of group 1–4 respectively. First, group1 is extracted from the training set and the test set. Then group 2–4 are extracted from the test set and test set. When generating each grouped training set and test set,the training set and test set of group 1 need to be excluded.

This module multiplies the training and test map with masks according to the order of the group 1–4. Each group is extracted and deleted from the training and test map. After this module, four pairs of training maps and test maps are obtained. Next, these four pairs are employed to train four two-branch inception nets separately.Of course, the group 5 will also be trained and tested.

3.3 Two-Branch Inception Net

In 2014, the Google team proposed the inception block to overcome the shortcomings of traditional CNN’s performance saturation and high computational overhead [22]. In order to fully utilize the spectral dimension features of the hyperspectral image GF-5 data, a one-dimensional convolution layer (Conv1D) is used to extract the spectral dimension features of each sample. At the same time, in order to use the spatial dimension information of the hyperspectral image, a two-dimensional convolutional layer (Conv2D) is used to extract the spatial characteristics of each intercepted pixel block.

For Sentinel 2 data, its main advantage is its high spatial resolution. And inspired by the inception module, the parallel convolutional layer is used to extract the spatial information of the Sentinel 2A data. According to the results of the inception module, multiple small convolution kernels can well solve the risk of large computational complexity and overfitting of a large convolution kernel. And multiple network branches extract the spatial texture information of sentinel 2 data at the same time, which can improve the performance of spatial texture features. Finally,the features extracted from the two data are combined through several fully connected layers(Dense) to obtain the proposed two-branch inception net. The overall structure of the twobranch inception net is shown in Fig.4.

Fig.4 The structure of two-branch inception net

In this module, five groups of training sets are used to train five classifiers, and these five classifiers are used to test the corresponding test sets, and five sets of results are obtained.

3.4 Integration Module

The main function of this module is to combine the five sets of results produced by the previous two-branch inception net to get the final classification map. The first four groups have a high priority, and the fifth group has a low priority. If the first four groups have obtained a majority results for a sample of the test set, then the results of the first four groups are used. If a sample point is not included in the classification results of the first four groups, then the results of the fifth group are used. If the first four groups have two or more different results for the same sample, then the result of the group 5 is chosen.The process is shown by Algorithm 1.

Algorithm 1 Integration module.Input: The results of group 1–5; The test samples;Output: The labels of all test samples;1: Check whether this test sample has a result in group 1–4. If yes, turn to 2, if not, turn to 3;2: Check whether the sample has a majority result in group 1–4. Yes, turn to 4, not turn to 5;3: The output is determined by the result of group 5.Turn to 6;4: The result of this sample is determined by this majority result. Turn to 6;5: The label result is determined by group 5. Turn to 6;6: Repeat 1–5 until all test samples have been executed;7: Return the results of all test sample.;

4 Experimental Results and Analysis

4.1 Parameter Tuning

This part is to introduce the problem of parameter optimization. First of all, there are 4 parameters for the threshold and morphology module,which are the number of pixels for data morphological dilation and the minimum number of pixels for connected domain filtering. After longterm experiments, the optimal value is obtained.According to the order of Tab. 3, they are 500, 4,1 000, and 50 respectively. The learning rate is 0.001, the optimization method is Adam, the training epochs is 30, and the training strategy that saves the smallest verification loss is used.

The extraction and elimination module and the integration module have no parameters that need to be optimized, so they are not discussed.Next, the parameter selection of two-branch inception net will be highlighted.

This is a deep learning network, so the optimization of the number of network layers and network structure is very complicated. After long-term experiments and testing, the structure of the two-branch inception net and the parameters of each layer are shown in Tab.4. Next, the choice of patch size will be discussed emphatically.

Tab. 4 Detailed structure and parameter selection of twobranch inception net

It can be seen from Fig.5 that in the group 3 of experiments in the Yellow River Delta, it is not that the higher the patch size, the better.When the batch size is equal to 9×9, the overall accuracy rate reaches a maximum. This is because although the larger the batch size, the more spatial information it contains, the information redundancy and noise also increase.

Fig.5 The influence of the input patch size

4.2 Performance of the Proposed OBH-RSI Framework

In this part, two sets of experiments would be conducted. The first set of experiments is to test inception net. Several other comparison methods will be compared with inception net to prove the effectiveness of inception net. The second set is to test the threshold and morphology module at the front and the whole OBH-RSI. The experiment will be carried out with and without the threshold and morphology module. The classic object-based method will also be used as a comparative experiment to compare with OBH-RSI.

4.2.1 Performance of Inception Net

In order to get the best comparison result, we evaluate the performance of inception net from all aspects. In the field of remote sensing, extreme learning machine (ELM) [23], SVM and RF [24] have been the most widely used. So, we first choose these two as the algorithm for comparison. Due to their lack of spatial texture information, we choose the classic dual-tunnel CNN [25] and the very practical cascade CNN [26]as the comparison algorithm. We compare the above five algorithms with our inception net, and get Tab. 5.

Tab. 5 The specific-class accuracy for different methods using Yellow River Delta dataset %

Three experiments are performed for each method, namely GF-5, Sen2 and All. For the RF,SVM and ELM, we use the spectral dimension of each test or training data as the data input. Because the original inputs of other methods except dual-tunnel CNN just accept only one data.GF-5 and Sentinel 2A data are combined into one cube to obtain better classification performance, which we name ALL. For every experiment, the accuracy of each category, average accuracy (AA), overall accuracy (OA) and kappa coefficient are calculated in order to make a full comparison at multiple scales.

As shown in Tab.5, the results of SVM and RF are very poor due to the huge data dimension. ELM’s OA, AA and kappa are maintained above 80%, which is a relatively stable method.For the three deep learning methods, their OA reached around 90% and above. Compared with the previous three traditional methods, there is a significant improvement. But the proposed method named two-branch inception net achieved the best in all three indicators, leading the second place by more than 1 percentage point.

It can be seen from the experiment that,generally speaking, combining the two sets of data is better than using one data alone. This is because although GF-5 has poor spatial resolution, it has excellent spectral resolution. The spatial resolution of Sentinel 2A is much better than that of GF-5 and Sentinel 2A just makes up for the shortcomings of GF-5 data. Therefore, the accuracy of the deep learning methods has improved. SVM and RF are two exceptions. This is because these two methods do not use spatial information and are not effective for high-dimensional data.

4.2.2 Performance of Segmentation Method

In order to compare the performance of the algorithm that makes the segmentation, several classic segmentation algorithms are compared to verify their effectiveness. The simple linear iterative clustering (SLIC) [27], entropy rate superpixel segmentation (ERSS) [28] and watershed segmentation algorithm are validated as comparison algorithms. For this set of experiments, the classifier is dual-tunnel CNN and the selected data is GF-5 data.

The idea is consistent with the proposed method, which is to divide the entire training set and test set into several groups, train the classifiers separately, and finally integrate the results to obtain the final evaluation parameters. Both SLIC and ERSS divide the image into four blocks, and the watershed algorithm also automatically divides the image into blocks. The experimental results are shown in Tab. 6. The first four columns in Tab.6 use dual-tunnel CNN and GF-5, the fifth column uses SVM and GF5+Sentinel 2A, and the last column uses the proposed method and GF5+Sentinel 2A. The proposed method has obvious advantages in three parameters.

Compared with the results of SLIC, ERSS and watershed algorithm, the proposed method has made improvements in three aspects which are AA, OA and kappe. This is mainly due to the reasonable selection of the threshold. For remote sensing image classification tasks, because our training set and test set are basically in the field of probability distribution, it is more reasonable to use the threshold selected by the normal distribution.

Next is the ablation analysis. We compare the results of removing and adding the previous mask method. Whether it is AA, OA or kappa,the results have improved. This proves the effectiveness of the proposed module. This is because this module can effectively divide the training set and test set into four basically non-overlapping groups according to the threshold of the normal distribution, which isolates the error between the groups.

In the last pair of experiments, only one comparison method is selected. This set of experiments is called ERSS+SVM and OBH-RSI in the last two columns of Tab. 6. This is because enough object-based methods have been compared in the second set of experiments. We choose the most classic SVM classifier to compare with OBH-RSI. As a result, there is no suspense after the previous experiments, and the advantage of OBH-RSI is great. After these two sets of experiments, the effectiveness of OBH-RSI has been fully proven.

Tab. 6 The specific-class accuracy for Different segmentation algorithms using GF-5 %

Fig.6 is a diagram of the full classification results of the three methods. Among them, Fig.6(a) is the result image of dual-tunnel CNN, Fig.6(b) is the result image of cascade net and Fig.6(c) is the result image of the proposed OBHRSI. It can be seen from Fig.6 that OBH-RSI ismuch more effective than the other two methods for classifying both sides of the Yellow River. According to field experience, there are not very wide reed belts on the side of the river. In Fig.6(c), it can be seen that there is a small long reed belt on the side of the river, but there are no or very few in Fig.6(a) and Fig.6(b). The classification of the Yellow River Fig.6(c) is the most accurate at the location of the Yellow River’s estuary.

Fig.6 Full classification diagram of the three methods: (a) the full classification map of dual-tunnel CNN; (b) the full classification map of cascade net; (c) the full classification map of OBH-RSI

5 Conclusion

In summary, the experiment verified the effectiveness and feasibility of OBH-RSI. This is mainly due to the initial division into four groups to reduce possible errors between each group.And the results of group 1–4 are used to correct the result of the group 5. It definitely optimizes the results of the original classifier. Of course, the contribution of the two-branch inception net is also excellent.

Our main contribution is to find a reasonable group strategy. In the experiment, the probability distribution of the vegetation index of each sample conforms to the normal distribution,which is the basis for using the threshold and morphology module. The remote sensing index and expert knowledge are combined to carry out scientific and effective groups. In the construct of the classifier, we refer to the inception module and the idea of sub-group classification to design an effective classifier. Both the ablation analysis experiment and the lateral comparison experiment carried out are sufficient. Not only traditional segmentation and machine learning algorithms are used, but CNN networks with good performance are also used. This is enough to prove the validity of the experiment. Finally, on the Yellow River Delta wetland dataset, our proposed method has achieved excellent results.

Since only wetlands can meet the requirements of the proposed OBH-RSI, object-based methods that integrate expert knowledge and field experience in other scenarios need to continue to be explored. The next work is mainly to continue to use object-based and remote sensing images to complete the fine classification of other scenes.