An Improved Image Fusion Method Based on a Wavelet Transform

Yang Dangfu Liu Shengjun Jiang Yuanhong Liu Xinru∗

(1.School of Mathematics and Statistics,Central South University,Changsha 410083,China；2.State Key Laboratory of High Performance Complex Manufacturing,Central South University,Changsha 410083,China)

Abstract Image fusion aims to construct images that are more appropriate and understandable for human and machine perception.In remote sensing applications,the fusion of the high-resolution panchromatic(PAN)image and the low-resolution multi-spectral(MS)image has always been a problem and has drawn much attention.In this paper,we propose a PAN and MS image fusion algorithm based on a wavelet transform.Firstly,after performing a wavelet transform on both images,the PAN image’s low-frequency components are fused into the MS image’s low-frequency components by using the edge intensity factor (EIF).Then,the high-frequency components of images are fused to obtain high-frequency features based on the maximum local standard deviation criterion (MLSTD).Finally,the high-resolution and multi-spectral fused images can be obtained by the wavelet inverse transform from the fused low-frequency and high-frequency components.Examples illustrate that the fused images are well equipped with desired features,and the proposed algorithm performs better than several classical methods.

Key words Image fusion Wavelet transform Edge intensity factor Local standard deviation

1 Introduction

As one of the most common information carriers,images can provide helpful information for humans.People can get more valuable information from higher quality,more informative images.The image fusion process is defined as gathering all the vital information and inclusion from multiple images into fewer images,usually a single one.This single image is more informative and accurate than any single source image,and it contains all the necessary information[1].Due to the limitation of the signal-to-noise ratio,remote sensing applications are impractical for remote sensing satellites to obtain high-quality surface images directly from high-altitude orbits with a single sensor.In practice,multiple sensors simultaneously image one area from satellite to obtain multiple images with the different necessary information.One image that is more appropriate and understandable for human and machine perception can be obtained through the image fusion process[2].

The two most commonly used image types in remote sensing applications are the high-resolution panchromatic (PAN) image and the low-resolution multi-spectral (MS) image.The MS image has the advantages of rich color and easy recognition of terrain types.However,its spatial resolution is too low to distinguish the target details,while the PAN image is the high-resolution grayscale image with rich details but without multi-spectral features.The fusion of the MS and PAN image is to gathering high-resolution information of the PAN image and multi-spectral features of the MS image into a single image,which can be used for further use.The image fusion algorithm proposed in this paper is mainly applied to the fusion of PAN and MS images in remote sensing.

Many methods have been proposed for the MS and PAN image fusion in remote sensing in the past two decades.In the early research,methods based on component substitution are most widely used.Standard methods like HIS[3],PCA[4],GS[5]can generally improve the spatial resolution of fused images.They first transform the MS image into new image space.If a specific component has a similar structure to the PAN image after transformation,then the PAN image is used to substitute the component.Then the corresponding inverse transform can be applied to obtain the final fused image.However,this type of method generally distort the color information.

In recent years,image fusion methods based on multi-resolution analysis,including pyramid decomposition[6],wavelet transform[7,8],and curvelet transform[9,10],have gradually become more mainstream.Among which the role of wavelet transform and curvelet transform has been widely approved in image processing.Through multi-resolution analysis,high-frequency spatial details of PAN images can be extracted and fused into each frequency band of MS images[11].Compared with methods based on component substitution,the methods based on multi-resolution analysis can preserve MS images’better spectral features in the fused images.However,such methods have spatial distortion defects,such as the ring or ladder phenomenon[12].

The proposed algorithm in this paper is a hybrid method that combines the advantages of the HIS-based method and the wavelet-transform-based method.The scheme is to constrain the color distortion while improving spatial resolution.Results show that the fused image of the PAN and MS images by our method has lower spatial and color distortion than the original methods.We will compare our results with several previous methods,including HIS[3],PCA[4],wavelet substitution[13],superposition wavelet[13],and curvelet transform[14]in section 4.

The rest of this paper is organized as follows:section 2 and section 3 describe the proposed image fusion algorithm;section 4 demonstrates the experiments and compares the results with those of previous methods;section 5 summarizes our work.

2 Algorithm Overview

Our goal is to fuse high-resolution PAN and multi-spectral MS images into one informative image with high-resolution and multi-spectral features.Before the fusion,the MS image is processed by HIS transform to obtain theH,I,andScomponents in HIS color space.We keep theHandScomponents constant and denote the intensity componentIasfMSI.The fusion operation is performed amongfMSI,the grayscale MS imagefMSGand PAN imagefP AN.Firstly,the high and low-frequency components of both MS and PAN images are extracted by a wavelet transform.Secondly,The two different components are fused separately by using different fusion schemes to obtain the wavelet intensity component.Thirdly,the corresponding intensity componentsI′′is obtained from the wavelet inverse transform.Finally,an inverse HIS transform is applied on the three componentsH,S,and the new intensity componentI′′to get the final fused image.

The overall pipeline is illustrated in Figure 1.MSI denotes theIcomponent of the MS image from the HIS transform.The low-frequency components from the wavelet transform of the PAN and MSI images areAP ANandAMSIrespectively.VP AN,HP AN,andDP AN(VMSI,HMSI,andDMSI)are the high-frequency components of the PAN(MSI)image from the wavelet transform.EIF ·AMSI,V ′′,H′′andD′′are the corresponding components after fusion,andMfdenotes the final fused image.Algorithm 1 summarizes the pseudocode for the improved image fusion algorithm based on the wavelet transform.

Figure 1 The overview of the proposed image fusion algorithm pipeline

The core of the algorithm is in steps 3.1 and 3.2,which will be detailed in the next section 3.

3 Image Fusion Schemes

As mentioned above,our fusion algorithm is based on the wavelet transform,and the fusion operations are performed in the frequency domain of the input images.By the wavelet transform,on the scale whose highest resolution is 2−J(J >0),the imagefcan be decomposed into a combination of low-frequency and high-frequency components[16]:

whereA2−Jfis the low-frequency component on scaleJ,the component,also known as the approximation image well preserves the contour information of the original image.The following three items respectively denote the high-frequency components on the scale fromJto 1 and record the high-frequency information from three different directions (horizontal,vertical,and diagonal directions) of the original image.We perform the wavelet transform onfMSIandfP ANand obtain the corresponding frequency sequences

With the high-frequency and low-frequency components,we apply different fusion schemes to them.We mainly introduce the edge intensity factor (EIF) for the low-frequency part,while for the high-frequency part,we use the maximum local standard deviation(MLSTD)criterion.

3.1 Low-frequency fusion based on EIF

To preserve the multi-spectral and high-resolution information of the MS and PAN images in the fused image,we start from the grayscale image of the MS imagefMSGand perform the wavelet decomposition to the scale whose highest resolution is 2−j,(J >0),and get

on which we define the EIF as follows:

wherefP AN(i,j) andfMSG(i,j) are the grayscale values at pixel (i,j) of the PAN and MSG images,respectively.The EIF reflects the intensity ratio between the two sub-imagesA2−JfP ANandA2−JfMSG.From Equation 3.3 along with the EIF,we define a new wavelet transformed intensity component as:

Then the region boundary information of the PAN approximation image is incorporated into the MSI approximation image.Because of the linear adjustment of the approximate image of the MSI,this method can not only improve the spatial resolution but also preserve the spectral features of the MS image[16].

3.2 High-frequency fusion based on MLSTD

The introduced EIF combine the approximation source images but not the detailed information.The high-frequency sub-images contain the region boundaries,edges,bright lines,and other significant detail information of the original images.To fuse the high-frequency sub-images,we firstly compute each pixel’s local standard deviation in sub-images and then use a simple maximum local standard deviation(MLSTD)strategy to determine the pixel value of the fused sub-images.The local standard deviation measures the pixel values deviations for each pixel in the local region.Take pixel(i,j)as an example:the local standard deviation of the pixel denoted asstd(i,j)can be computed as

wheref(i,j)is the pixel value at pixel(i,j),is the average pixel values of the neighbor pixels and the center pixel(i,j),totally 9 pixels in this case.

The MLSTD strategy is as follows:by comparing the local standard deviationstd(i,j) of the two fusing sub-images,the pixel with greaterstd(i,j)will be written into the fused sub-images.The MLSTD strategy can preserve both the detail information and texture information of the fusing sub-images[15].

4 Experiments

To verify the proposed algorithm we implemente the method in Matlab R2014a and compare the results with the above-mentioned PCA,HIS,SWT,AWT,CT methods.For convenience,we call the proposed method the IWT(Improved Wavelet Transform)method.

Figure 2 shows the fusion results of the MS and PAN chimp images from [16].In which the low-resolution multi-spectral MS chimp image has three channels:red,green,and blue.The image is in RGB space with a wide range of color information.The high-resolution PAN image is a grayscale image with clear texture and detailed information.The standard HIS and PCA methods are parameter-free.However,both the wavelet transform and curvelet transform methods have several parameters,such as the wavelet function selection and decomposition level setting.We chose Symlet Wavelets sym4 and apply the 1 level decomposition for the wavelet transform,and use the wrapping algorithm.The decomposition level is set to 2.

Figure 2 Fusion results for MS and PAN chimp images of different methods,where(a)and(b)are original MS and PAN chimp images,respectively

Human visual perception is a subjective evaluation method of fusion images.It is s one of the best ways to evaluate the image quality by human visual perception since the end-user of fused images is often humans[18].Figures 2(d) and 2(g) have noticeable color distortion,while Figures 2(c),2(e) and 2(f) are still of low-resolution.The best visual result is produced by our improved wavelet transform method in Figure 2(h) with low color distortion and high-resolution from visual observation.Similar comparisons are made on remote sensing images of the IKONOS satellite in Figure 3 and Figure 4[17].

Figure 3 Fusion results for MS and PAN mountain images of different methods,where(a)and(b)are original MS and PAN chimp images,respectively

Figure 4 Fusion results for MS and PAN village images of different methods,where(a)and(b)are original MS and PAN chimp images,respectively

Except for the visual comparison,we also do some quantitative calculations to measure the sharpness and color distortion of fused images.Firstly,on the sharpness of the image,which is an important evaluation criterion for image fusion,the average gradient of an image can reflect the image’s ability to express small details[19].Then average gradient of an image is defined as

whereMandNare the width and height of the image in pixels,f(i,j) is the pixel value of the image at pixel(i,j).The larger the average gradient,the higher the sharpness and the spatial resolution of the image.Figure 5 shows the average gradient of fusion results in Figure 2,Figure 3,and Figure 4.

Figure 5 Average gradient histogram on different images

Figure 6 Correlation coefficients histogram on different fused images with different algorithms

As reflected by the numerical values of three results of average gradients,the new algorithm designed in this paper has the best performance in terms of sharpness.Moreover,in the first set of experiments,the PAN image’s sharpness are greatly improved,so the proposed algorithm achieves the expected goal of improving the fusion image’s spatial resolution.

Besides,correlation coefficients are adopted as the evaluation criteria for spectral resolution analysis[20].By comparing different color channels,correlation coefficients of fused images and source images can measure the similarity and distortion of spectral features of the two images.Since the MS image has R,G,and B color channels,the correlation coefficients are computed between the source MS images and the fused images on each channel.The three values can measure the color distortion of the fused image on each color channel.To measure the color distortion on the whole image,we use the standard deviation of three correlation coefficients on R,G,and B channels.The smaller the standard deviation,the lower color distortion of the fused image.The correlation coefficient for each channel is computed as follows:

whereAandBare two input images with the same rows and columns in pixel.npixis the number of pixels for each image,A¯ andB¯ are the average color value for the color channel.We useCC0,CC1,andCC2to denote the correlation coefficients on R,G,and B channels,respectively,anddenotes the average ofCC0,CC1,andCC2.Then the standard deviation of correlation coefficients can be simply computed by

The computation results of the three fused images are listed in Table 1,Table 2,and Table 3.As mentioned above,CC0,CC1,andCC2denote correlation coefficients between the MS source image and the fused image in R,G,and B channels,respectively.is the average standard deviation ofCCi,i=0,1,2.As shown in the last columns of each table,TheSTDis larger in the PCA,IHS,and SWT algorithms,which implies higher color distortion.They are also consistent with the visual observations above.Among all three experiments,the AWT algorithm always performes best,and our IWT method follows.Besides,the SWT and AWT algorithms’results are stable on the three different experiments with 0.0134 and 0.00003 standard deviations,respectively.According to the standard deviation of correlation coefficients,our IWT algorithm is only inferior to the AWT algorithm.Nevertheless,note that our IWT algorithm performes better than the AWT algorithm on the fused images’sharpness.

Table 1 The correlation coefficients and standard deviations of the different algorithms for the results in Figure 2

Table 2 The correlation coefficients and standard deviations of different algorithms for the results in Figure 3

Table 3 The correlation coefficients and standard deviations of different algorithms for the results in Figure 4

Furthermore,to visually compare the correlation coefficients of three channels,the histograms ofCCi,i=0,1,2 are displayed in Figure 6.Figure 6(a)shows that the AWT algorithm achieves the best performance on all the three channels for the first fusion experiment.The correlation coefficients are all close to 1.0,and the deviations among the three values are little,so it can preserve the spectral information of the MS image best.Note that the closer is the three correlation coefficients to 1,the more similar are the source MS image and the fused image,which implies lower spatial resolution.The sharpness analysis also proves that the AWT algorithm has a lousy performance on fused image spatial resolution.As for the algorithms PCA,IHS,SWT and CT,their three channels’ correlation coefficients are pretty uneven,leading to high color distortion fused images.Furthermore,note that the PCA’s R channel correlation coefficient is close to 1.0 and higher than the other two’s,so the fused image’s R channel is brighter than G and B channels.In Figure 2(c),the red nose part of the chimpanzee is close to the source MS image’s,but the other parts of the image with green and blue colors are darker than the source MS image’s.In contrast,the HIS algorithm gets minimum correlation coefficients on the R channel than the others,so in Figure 2(d)the red nose part has noticeable color distortion,but looks good on the other parts.The same analysis can be done on the algorithms SWT and CT,which also have high color distortion.The proposed IWT algorithm has relatively stable correlation coefficients,which is only no better than the AWT algorithm.It obtains a pretty low color distortion fused image as shown in Figure 2(h).However,our IWT algorithm performed far better than the AWT algorithm on preserving the spatial resolution of the PAN images.

Compared to the first experiment,the second and third experiments in Figure 3 and Figure 4 are quite consistent on correlation coefficient of RGB channels.As shown in Figures 6(b) and 6(c),the AWT’s correlation coefficients are all closest to 1.0,thereby generate the lowest color distortions between Figures 3(f) and 3(a),as well as 4(f) and 4(a).However,the fused images both perform poorly on preserving high-resolution spatial characteristics of the corresponding PAN images 3(b) and 4(b).The SWT algorithm has the second-highest correlation coefficients,but the deviation among RGB channels is pretty high.Thereby the fused images 3(e)and 4(e)have high color distortion.The PCA,IHS,CT and our IWT algorithms’correlation coefficients are all below 0.85.Furthermore,according to the computed standard deviation in Table 2 and Table 3,our IWT has method has minimum color distortion among these algorithms.

Through the above analysis,it shows that our algorithm performs best in the experiment of Figure 2 on image sharpness but only no better than the CT algorithm in the experiments of Figure 3 and Figure 4.However,our algorithm has lower color distortion than the CT algorithm.As for color distortion analysis,although the AWT algorithm gets the minimum color distortion,it performs much worse on preserving the PAN image’s spatial high-resolution characteristics.Our algorithm’s standard deviations of correlation coefficients are only higher than the AWT’s.In conclusion,our proposed algorithm has better appropriate correlation coefficients and standard deviation of those coefficients.

5 Conclusions

In this paper,we proposed a wavelet-transform-based image fusion algorithm.The algorithm’s inputs are the MS image and the PAN image.The former type of image is low-resolution but has rich color information,while the latter type of image is single-spectral but has detailed texture.The fused image has both high-resolution and multi-spectral characteristics from the input MS and PAN images.Analysis of visual observation and numerical statistics suggested the efficiency and performance of our algorithm.Additionally,comparisons with several typical fusion algorithms demonstrated the advantages of our method.