Galaxy morphologies revealed with Subaru HSC and super-resolution techniques. II. Environmental dependence of galaxy mergers at z ∼ 2–5

Abstract

We super-resolve the seeing-limited Subaru Hyper Suprime-Cam (HSC) images for 32187 galaxies at |$z\sim 2$|–5 using three techniques, namely, the classical Richardson–Lucy (RL) point spread function (PSF) deconvolution, sparse modeling, and generative adversarial networks, to investigate the environmental dependence of galaxy mergers. These three techniques generate overall similar high spatial resolution images but with some slight differences in galaxy structures; for example, more residual noises are seen in the classical RL PSF deconvolution. To alleviate the disadvantages of each technique, we create combined images by averaging over the three types of super-resolution images, resulting in galaxy substructures resembling those seen in the Hubble Space Telescope images. Using the combined super-resolution images, we measure the relative galaxy major merger fraction corrected for the chance projection effect, |$f_{\rm merger}^{\rm rel,col}$|⁠, for galaxies in the |$\sim$|300 deg|$^2$| area data of the HSC Strategic Survey Program and the CFHT Large Area U-band Survey. Our |$f_{\rm merger}^{\rm rel,col}$| measurements at |$z\sim 3$| validate previous findings showing that |$f_{\rm merger}^{\rm rel,col}$| is higher in regions with a higher galaxy overdensity |$\delta$| at |$z\sim 2$|–3. Thanks to the large galaxy sample, we identify a nearly linear increase in |$f_{\rm merger}^{\rm rel,col}$| with increasing |$\delta$| at |$z\sim 4$|–5, providing the highest-z observational evidence that galaxy mergers are related to |$\delta$|⁠. In addition to our |$f_{\rm merger}^{\rm rel,col}$| measurements, we find that the galaxy merger fractions in the literature also broadly align with the linear |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation across a wide redshift range of |$z\sim 2$|–5. This alignment suggests that the linear |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation can serve as a valuable tool for quantitatively estimating the contributions of galaxy mergers to various environmental dependences. This super-resolution analysis can be readily applied to datasets from wide field-of-view space telescopes such as Euclid and Roman.

1 Introduction

Galaxy evolution is thought to accelerate within dense cosmic environments such as galaxy groups and galaxy clusters (Thomas et al. 2005). In the local Universe, old and star-formation-quenched elliptical galaxies are more abundant in massive galaxy clusters than in the field environments (e.g., Dressler 1980; Kodama et al. 1998; Goto et al. 2003; Tanaka et al. 2004). This strong environmental dependence on the star-formation activity and the galaxy morphology is likely to be a consequence of the accelerated galaxy evolution in galaxy overdense regions. Among the proposed environmental processes to accelerate the galaxy evolution (e.g., ram-pressure stripping: Gunn & Gott 1972; warm and hot gas removal: Larson et al. 1980; Balogh et al. 2000; high-speed galaxy encounters: Moore et al. 1996), galaxy mergers are thought to be one of the key physical mechanisms in forming massive elliptical galaxies (Mihos & Hernquist 1994; Naab et al. 2014; Tadaki et al. 2014). Galaxy mergers trigger nuclear starbursts followed by the emergence and feedback of active galactic nuclei (AGN), eventually resulting in the formation of massive elliptical galaxies (Hopkins et al. 2008; Alexander & Hickox 2012; Toft et al. 2014).

The frequency of galaxy mergers in galaxy overdense regions has been extensively investigated in a large number of studies. At |$z\sim 0$|–1, galaxy mergers and interactions occur more frequently in higher galaxy density regions due to small galaxy–galaxy separations (e.g., Lin et al. 2010; de Ravel et al. 2011; Laishram et al. 2024). However, the fraction of galaxy mergers in massive galaxy clusters tends to be lower than that in galaxy groups or the field environment (e.g., McIntosh et al. 2008; Tran et al. 2008; Perez et al. 2009; Alonso et al. 2012; Omori et al. 2023; Sureshkumar et al. 2024). Galaxy–galaxy interactions are suppressed due to a high-velocity dispersion of |$\sim$|500–1000 km s|$^{-1}$| in a deep gravitational potential well of massive galaxy clusters, as predicted in theoretical studies (e.g., Ghigna et al. 1998; Gottlöber et al. 2001; Jian et al. 2012). Considering the evolution of velocity dispersion, it is expected that the frequency of galaxy mergers is high in young and developing galaxy protoclusters at |$z\gtrsim 2$| compared to local massive galaxy clusters.

Despite the expectation, the relation between galaxy mergers and galaxy overdensity remains controversial at |$z\gtrsim 2$|⁠. Using high spatial resolution data of the Hubble Space Telescope (hereafter Hubble), some studies have indicated a high galaxy merger fraction in galaxy protoclusters (e.g., Lotz et al. 2013; Hine et al. 2016; Watson et al. 2019; Liu et al. 2023). On the other hand, there are reports suggesting that the frequency of galaxy mergers may not be correlated with galaxy overdensity (Delahaye et al. 2017; Monson et al. 2021; Momose et al. 2022). The discrepancy might be partially due to limited galaxy samples, typically comprising |$\sim$|100–1000 sources, in a study with the small field of view (FoV) of Hubble. Both wide survey area data and high spatial resolution images are needed to study the morphology of high-z compact sources (i.e., merger or isolated) with statistical samples of rare galaxy overdense regions.

To investigate galaxy merger features of rare high-z objects, we have undertaken morphological studies by applying super-resolution techniques to seeing-limited images of ground-based telescopes (Shibuya et al. 2022), specifically, the Subaru/Hyper Suprime-Cam (HSC)-Strategic Survey Program (SSP) data (Schlafly et al. 2012; Tonry et al. 2012; Magnier et al. 2013; Aihara et al. 2018a, 2018b; Miyazaki et al. 2018; Coupon et al. 2018; Furusawa et al. 2018; Huang et al. 2018; Kawanomoto et al. 2018; Komiyama et al. 2018). The HSC-SSP data cover a wide area, from several hundred to |$\sim$|1000 deg|$^2$|⁠, providing the capability to trace diverse galaxy populations and environments, including rare high-z objects of rest-frame ultraviolet (UV) luminous galaxies and galaxy protoclusters. The pilot study (Shibuya et al. 2022, hereafter Paper I) estimated the galaxy major merger fractions for galaxies at |$z\sim 4$|–7 at the bright end of UV luminosity functions (LFs) using the classical Richardson–Lucy (RL) point spread function (PSF) deconvolution (Richardson 1972; Lucy 1974). The classical RL PSF deconvolution has enhanced the spatial resolution of the seeing-limited HSC images, from |$\sim 1^{\prime \prime }$| to |$\lesssim$||${0{^{\prime \prime}_{.}}1}$|⁠, which is comparable to that of Hubble. The galaxy major merger fractions estimated using the super-resolution HSC images are found to be comparable between the UV-bright galaxies and control faint sources, suggesting that galaxy major mergers are not dominant physical mechanisms for the galaxy number density excess at the bright end of UV LFs (e.g., Ono et al. 2018; Harikane et al. 2022).

This is the second paper in a series investigating morphological properties of rare galaxy populations at high z with super-resolution techniques. In this paper, we study the environmental dependence of galaxy mergers at |$z\sim 2$|–5 by applying several super-resolution techniques for the latest product available, the internal S21A HSC-SSP data release. Apart from the classical RL PSF deconvolution used in Paper I, we introduce two additional super-resolution techniques: sparse modeling and machine learning. While these two super-resolution techniques have been used in various domains of astronomy (e.g., Honma et al. 2014; Schawinski et al. 2017; Villaescusa-Navarro et al. 2021; Shirasaki et al. 2021; Murata & Takeuchi 2022; see also review papers of, e.g., Huertas-Company & Lanusse 2023 and Moriwaki et al. 2023), the application to high-redshift galaxies has been relatively limited. This study has technical and scientific goals. The technical goal is to apply the three super-resolution techniques to high-z galaxies and to explore the advantages and disadvantages by comparing the performance of the super-resolution. The scientific goal is to measure the galaxy merger fraction as a function of galaxy overdensity using the super-resolution HSC images.

This study may also contribute to our understanding about physical mechanisms of environmental dependences found in high-z galaxy protoclusters. Over the last two to three decades, numerous galaxy protoclusters at |$z\gtrsim 2$| have been discovered (e.g., Steidel et al. 1998; Pentericci et al. 2000; Venemans et al. 2002; Shimasaku et al. 2004; Matsuda et al. 2005; Ouchi et al. 2005; Overzier et al. 2006; Papovich et al. 2010; Capak et al. 2011; Hatch et al. 2011; Trenti et al. 2012; Toshikawa et al. 2012, 2014, 2016, 2024; Muzzin et al. 2013; Wylezalek et al. 2013; Cucciati et al. 2014; Lemaux et al. 2014, 2018; Planck Collaboration 2015; Dey et al. 2016; Chiang et al. 2017; Oteo et al. 2018; Castellano et al. 2018; Jiang et al. 2018; Miller et al. 2018; Higuchi et al. 2019; Shi et al. 2019, 2020; Ando et al. 2020; Uchiyama et al. 2022; Ito et al. 2023; Forrest et al. 2023; Staab et al. 2023; Morishita et al. 2023; Hashimoto et al. 2023; see also review papers of Overzier 2016 and Alberts & Noble 2022, a white paper of Overzier & Kashikawa 2019, and summary tables in Chiang et al. 2013 and Harikane et al. 2019). Within high-z galaxy protoclusters, various environmental dependences have been identified. Although the galaxy mergers have been proposed as a possible physical mechanism to create the environmental dependences, the influence of galaxy mergers has not been well understood observationally. Establishing a useful relation between galaxy mergers and galaxy overdensity would allow us to discuss quantitatively the contribution of galaxy mergers to the high-z environmental dependences.

This paper is structured as follows. Section 2 provides an overview of galaxy catalogs and images used in this study. Section 3 explains the details of each super-resolution technique and methods to calculate the galaxy merger fraction and the galaxy overdensity. In section 4, we present super-resolution images and the relation between the galaxy merger fraction and the galaxy overdensity. Section 5 discusses whether the galaxy merger and overdensity relation changes with redshift, and effects of galaxy mergers on environmental dependences at high z. Section 6 summarizes our findings.

Throughout this paper, we assume a flat universe with the cosmological parameters of |$(\Omega _{\rm m}, \Omega _\lambda , h)=(0.3, 0.7, 0.7)$|⁠. All magnitudes are given in the AB system (Oke 1974; Oke & Gunn 1983). We refer to the Hubble F606W and F814W filters as |$V_{606}$| and |$I_{814}$|⁠, respectively.

2 Data

2.1 Galaxy catalogs

In this study, we use two catalogs of galaxies: dropout galaxies in Harikane et al. (2022) and photo-z galaxies in Desprez et al. (2023). These galaxy catalogs are constructed with the HSC-SSP data (Aihara et al. 2018a, 2018b, 2019). HSC is a |$\sim$|1.8 deg|$^2$| FoV optical camera installed on the Subaru Telescope. Using the wide FoV camera of HSC, the HSC-SSP survey covers three observational layers with different survey areas and depths: the UltraDeep, Deep, and Wide layers. These layers are observed through five optical broad-band filters of g, r, i, z, and y. Due to the data availability and the sample size for galaxy overdensity maps, we combine dropout galaxies at |$z\sim 4$|–5 and photo-z galaxies at |$z\sim 2$|–3, as explained below.

The dropout galaxies are selected with the HSC-SSP data obtained from 2014 March through 2018 January. The effective survey areas of the HSC-SSP data are |$\sim$|3, |$\sim$|18, and |$\sim$|288 deg|$^2$| for the UltraDeep, Deep, and Wide layers, respectively, which is |$\sim$|3 times larger in total than that of the data used in Paper I (see also Ono et al. 2018). Thanks to the wide survey area, a substantial number of |$\sim$|4 million galaxies are identified at |$z\sim 2$|–7. Among the sample, the catalogs of |$z\sim 4$|–7 dropout galaxies are publicly available on the Zenodo website.¹ In this study, we use g-dropouts at |$z\sim 4$| and r-dropouts at |$z\sim 5$|⁠, the numbers of which are large enough to obtain reliable galaxy overdensity maps. The selection of the dropout galaxies is based on the Lyman-break technique (Steidel et al. 1996; Giavalisco 2002). The color selection criteria are |$g-r\gt 1.0$|⁠, |$r-i\lt 1.0$|⁠, and |$g-r\gt 1.5(r-i)+0.8$| for g-dropouts, and |$r-i\gt 1.2$|⁠, |$i-z\lt 0.7$|⁠, and |$r-i\gt 1.5(i-z)+1.0$| for r-dropouts. See Harikane et al. (2022) for more details. We retrieve the masked catalogs in which sources are masked in regions affected by, e.g., halos of bright stars.

The photo-z galaxies are selected in the CFHT Large Area U-band Deep Survey (CLAUDS; Sawicki et al. 2019). CLAUDS is an imaging survey program with the u and |$u^{*}$| bands of an optical camera of CFHT MegaCam. The CLAUDS survey covers a |$\sim$|20 deg|$^2$| area in the HSC-SSP UltraDeep and Deep layers. The relative survey coverage of CLAUDS is |$\sim$|80% of the HSC-SSP Deep fields (see table 2 of Desprez et al. 2023). The u- and |$u^{*}$|-band data are useful to select |$z\sim 2$|–3 galaxies whose Lyman-break features fall in the wavelength range of |$\sim$|3600–4800 Å. Part of the CLAUDS fields is also observed in near-infrared (NIR) surveys of the VISTA Deep Extragalactic Observations (VIDEO; Jarvis et al. 2013) and UltraVISTA (McCracken et al. 2012). Photometric redshifts are obtained through spectral energy distribution (SED) fitting from the u to y bands (extending to the H or K bands for the fields with NIR data available). From the catalogs available on the CLAUDS website,² we select the SExtractor catalogs with the results of the SED fitting performed using the u to y bands for all the CLAUDS fields, which have better performance in the photo-z measurements than the other catalogs. The precision of the photo-z measurements is |$\sigma \lesssim 0.04$| down to an i-band magnitude of |$i\sim 25$|⁠. For further details, please refer to Desprez et al. (2023).

2.2 Images for the super-resolution analysis

We use the latest product available, the internal S21A HSC-SSP data release, as images for our super-resolution analysis. The S21A HSC-SSP data were obtained in observations from 2014 March through 2021 January. The HSC images have been reduced with the hscPipe 8.0–8.4 software (Bosch et al. 2018; see also Axelrod et al. 2010; Jurić et al. 2017; Ivezić et al. 2019). The pixel scale of the HSC image is |${0{^{\prime \prime}_{.}}168}$| per pixel. The typical seeing size of the HSC-SSP data ranges from |$\sim$||${0{^{\prime \prime}_{.}}7}$| to |$\sim$||${0{^{\prime \prime}_{.}}8}$| (Aihara et al. 2022). Table 1 presents the limiting magnitudes of the S21A HSC-SSP data. We retrieve the S21A HSC-SSP coadd cutout images by 36 pixels |$\times$| 36 pixels (⁠|$\sim 6^{\prime \prime }$|  |$\times$| 6|$^{\prime \prime }$|⁠) centered at the position of each galaxy. The size of the cutout images is sufficiently large to cover entire structures of |$z\sim 2$|–5 galaxies with a typical half-light radius of |$r_{\rm e} \sim {0{^{\prime \prime}_{.}}1}$|–|${0{^{\prime \prime}_{.}}2}$| (e.g., Shibuya et al. 2015). We also download PSF images from the HSC-SSP database³ for the analysis with the classical RL PSF deconvolution and sparse modeling. Before applying the super-resolution techniques, we change the original pixel scale of the HSC coadd and PSF images, and define a new pixel grid to match the pixel scale of the Hubble Advanced Camera for Survey (ACS; i.e., |${0{^{\prime \prime}_{.}}06}$| per pixel). Here we perform a linear interpolation of the flux distribution in the new pixel grid.

2.3 Selection of galaxies for the super-resolution analysis

Combining the two galaxy catalogs of dropout galaxies and photo-z galaxies, we construct a sample of |$z\gtrsim 2$| galaxies for the super-resolution analysis. From the CLAUDS photo-z galaxy catalog, we select galaxies whose photometric redshift falls in typical redshift ranges of |$z\sim 2$| BX/BM and |$z\sim 3$|  U-dropout galaxies. We define galaxies with the best-fitting photometric redshift Z_BEST of |$z=1.7\pm 0.3$| as BM galaxies, |$z=2.2\pm 0.3$| as BX galaxies, and |$z=3.1\pm 0.3$| as U-dropout galaxies. We do not apply the redshift selection for the dropout galaxies because these sources are selected in the Lyman-break technique. We consider |$z=3.8$| and 4.9 as the representative redshifts of g-dropout and r-dropout galaxies, respectively.

We select galaxies with a rest-frame UV continuum magnitude of |$m_{\rm UV}\lt 23.5$|⁠, which are bright enough for our super-resolution analysis. To trace the rest-frame UV continuum emission, we use magnitudes in the g band for BM, the r band for BX and U-dropout, the i band for g-dropout, and the z band for r-dropout galaxies. We exclude compact sources that could potentially represent AGN or stars: photo-z galaxies with object classification parameters of OBJ_TYPE  |$=$| 1 (QSOs) or 2 (stars), COMPACT  |$=$| 1 (compact) in the CLAUDS photo-z galaxy catalog, and dropout galaxies with the magnitude difference |$m_{\rm PSF}-m_{\rm CModel}\lt 0.15$| (Matsuoka et al. 2019), where |$m_{\rm PSF}$| and |$m_{\rm CModel}$| are the magnitudes measured with the PSF and CModel profiles, respectively (Aihara et al. 2018a). To further reduce contaminants, only sources flagged as MASK  |$=$| 0 and ST_TRAIL  |$=$| 0 are used in the CLAUDS photo-z galaxy catalog.

Table 2 presents the numbers of the selected galaxies at |$z\sim 2$|–5. The total number of the galaxies at |$z\sim 2$|–5 is 32187. Although the sample consists of two different types of galaxies, i.e., dropout galaxies and photo-z galaxies, previous studies have found that UV LFs are in good agreement between the two types of sources (e.g., Moutard et al. 2020). The agreement suggests that selection effects do not strongly impact the analysis of this study.

2.4 Hubble Space Telescope data

In addition to the HSC data, we use two Hubble datasets: COSMOS (Koekemoer et al. 2007) and CANDELS (Grogin et al. 2011; Koekemoer et al. 2011). These high-resolution data can serve as ground-truth images of galaxies.

The COSMOS data are used to evaluate the performance of the super-resolution techniques. Part of the HSC-SSP UltraDeep COSMOS field has been observed with the |$I_{814}$| filter of the Hubble ACS. With the Hubble COSMOS data, we create a list of galaxy mergers and isolated galaxies. Using a Hubble detection source catalog of Leauthaud et al. (2007) for COSMOS and the HSC-SSP photometric redshifts obtained with the MIZUKI photo-z software (Tanaka et al. 2018), we select galaxy mergers and isolated galaxies at |$z\gtrsim 2$|⁠. We apply the same selection criteria for the flux ratio and source separation as those for the photo-z and dropout galaxies, described in Paper I. Using the SExtractor software (Bertin & Arnouts 1996), we detect sources in the super-resolution HSC images. In the source catalogs constructed with SExtractor, galaxy major mergers are identified based on the presence of similar-flux double components. Specifically, we define galaxy close-pair systems as galaxy mergers when the systems have a double component with a flux ratio of |$\mu =f_2/f_1\ge 1/4$| and a source separation within |$d=r_{\rm min}-r_{\rm max} = 3$|–6.5 kpc. Here, |$f_1$| and |$f_2$| are the flux values for the bright primary and the faint secondary galaxy components, respectively. The number of the selected objects is |$\sim$|1000 for each class of galaxy mergers and isolated galaxies. These objects selected in COSMOS are used to calculate the detection rates of galaxy mergers and isolated galaxies in the super-resolution HSC images (subsection 4.1). In addition, we use these objects as a test sample to assess the generalization performance of a machine-learning model in sub-subsection 3.1.3.

On the other hand, the CANDELS data are used as the training data for a machine-learning network. The details of the training data are described in sub-subsection 3.1.3.

3 Analysis

3.1 Super-resolution techniques

We apply three super-resolution techniques: (1) the classical RL PSF deconvolution, (2) sparse modeling, and (3) machine learning. Figure 1 illustrates super-resolution HSC images of an example high-z galaxy. Each technique is explained in the following three subsections (sub-subsections 3.1.1–3.1.3). In subsection 4.1, we compare images generated in each super-resolution technique, and discuss the advantages and disadvantages.

3.1.1 The classical RL PSF deconvolution

The first technique is RL PSF deconvolution (Richardson 1972; Lucy 1974). The RL PSF deconvolution algorithm has been widely used in many applications for about five decades. Hence, we refer to the RL PSF deconvolution as the classical method in this paper.

The classical RL PSF deconvolution is a maximum-likelihood algorithm assuming that the noise of observed images |$\mathbf {y}$|⁠,

follows Poisson statistics p at a specific image pixel i,

where |$\mathbf {P}$| is a PSF matrix, and |$\mathbf {x}$| is a vector of a restored image. The classical RL PSF deconvolution has a super-resolution effect because the cutoff spatial frequency of the image |$\mathbf {x}$| increases during the maximum-likelihood estimation.

We analyze the HSC images with the classical RL PSF deconvolution in the same way as in Paper I. Specifically, we exploit the alternating direction method of multipliers (ADMM) algorithm (Boyd et al. 2011). For the convergence check during the ADMM iteration, we monitor the total absolute percentage error,

where |$*$| represents the convolution operator, i denotes a specific pixel, |$n_{\rm iter}$| signifies the current iteration number, and |$N_{\rm im}$| means the total number of image pixels. We stop the iteration when the super-resolution image |$\mathbf {x}$| satisfies a convergence criterion of |$|{d} E(n_{\rm iter})/ {d}n_{\rm iter}| \lt 1\times 10^{-3}$| at least 10 times consecutively, or |$n_{\rm iter}$| exceeds |$N_{\rm iter}=250$|⁠. See Paper I for the detailed analysis.

The bottom left panel of figure 1 shows an example image processed with the classical RL PSF deconvolution technique. As demonstrated in Paper I, the classical RL PSF deconvolution reproduces substructures of high-z galaxies that resemble those seen in the Hubble images. However, the classical RL PSF deconvolution tends to generate many residual noises in the restored images. In addition, the process of the RL PSF deconvolution can occasionally violate the Shannon–Nyquist sampling theorem (Nyquist 1928; Shannon & Weaver 1949; Magain et al. 1998). To deal with these limitations, we use two additional techniques, namely, sparse modeling and machine learning.

3.1.2 Sparse modeling

The second technique is sparse modeling. Sparse modeling is an algorithm to solve problems by imposing constraints related to the sparsity in specific domains. In image restoration studies, widely utilized constraints are the sparseness in pixel values and the smoothness between adjacent pixels. These constraints can enhance the image resolution less affected by noises than the classical RL PSF deconvolution.

In this study, we employ a method of sparse modeling proposed by Morii, Ikeda, and Maeda (2019). The method incorporates two additional constraints of sparseness and smoothness into the classical RL PSF deconvolution. The sparseness constraint is the Direclet function,

where |$\rho$| is an image normalized with the total flux, |$\rho =y(u)/\Sigma _u{y(u)}$|⁠, u is a pixel index, and |$\beta$| is a hyperparameter controlling the degree of sparseness. A smaller |$\beta$| value generates a sparser image.

The smoothness constraint is the total squared variation (TSV),

where μ is a hyperparameter controlling the degree of smoothness. A higher μ value generates a smoother image. The TSV is a squared version of the total variation (TV; Rudin et al. 1992) and has been introduced to resolve radio interferometer images taken with the Event Horizon Telescope (EHT; Kuramochi et al. 2018) and ALMA (Yamaguchi et al. 2020). While TV creates mosaic-like structures, TSV restores more realistic images for smooth objects.

By adding the sparseness and smoothness constraints to the classical RL PSF deconvolution, the cost function is defined as

where |$L_\rho (\rho ) = \sum _{v} Y(v) \log \left[ \sum _u t(v, u) \rho (u) \right]$|⁠. The joint distribution of |$Y(v)$| is a product of two functions: the Poisson and multinomial functions. The optimization problem aims to minimize the cost function with the hyperparameters of |$\beta$| and μ. See Morii, Ikeda, and Maeda (2019) and Ikeda et al. (2014) for details of the optimization process.

The bottom middle panel of figure 1 displays an image analyzed with the sparse modeling technique. We determine the hyperparameters of sparseness |$\beta$| and smoothness μ through a grid search of |$\beta$| and μ by visually inspecting restored images for several randomly chosen sources. The top panel of figure 2 shows images at different hyperparameters of |$\beta$|⁠. Other example galaxies are presented in figure 14 in the Appendix. Here we only present the dependence on |$\beta$| because we find that |$\mu =0$| offers the optimal results in a grid search of the hyperparameters. The parameter of |$\beta =0.96$| produces an image that most closely resembles that of Hubble. Hence, we adopt |$\beta =0.96$| and |$\mu =0$| as the hyperparameters for the super-resolution analysis.

3.1.3 Generative adversarial network

The third technique is based on machine learning, specifically, the generative adversarial network (GAN; Goodfellow et al. 2014). The GAN is a machine-learning algorithm designed for data generation tasks. The GAN consists of two networks: the generator and the discriminator. The generator yields fake images by learning the characteristics of numerous input real images, while the discriminator distinguishes real images from generated (fake) images. After successful training, the GAN is able to generate synthetic images that are closely similar to real images.

In this study, we employ the conditional GAN (cGAN; Mirza & Osindero 2014). The cGAN framework trains the generator and discriminator incorporating additional conditions, such as class labels and descriptive tags. For our task of generating high-resolution images from low-resolution inputs, we use a specific cGAN known as pix2pix, which sets conditions on images (Isola et al. 2016, 2017). By learning a mapping from input images to output images, the pix2pix cGAN performs image-to-image translation tasks. Through training with a large dataset of image pairs consisting of low-resolution and high-resolution images, the network can effectively convert low-resolution inputs to high-resolution outputs. For our super-resolution analysis, we adopt the HSC (Hubble) images as low-resolution (high-resolution) images.

To build our training samples, we select galaxies in the HSC-SSP fields of UltraDeep COSMOS, UDS, and Wide AEGIS, which are partially observed in the Hubble CANDELS survey. Galaxies for the training sample are extracted from the 3D-HST galaxy catalog (Skelton et al. 2014). We use galaxies brighter than the |$15\sigma$| limiting magnitudes of the CANDELS fields. Table 3 summarizes the magnitude limits used for the galaxy selection in each HSC-SSP layer. Due to the lack of Deep-depth HSC-SSP images in the Hubble CANDELS fields, we substitute images from the previous S18A UltraDeep release, exhibiting similar depths to those of the S21A Deep layer. To augment the training sample, we utilize images from two Hubble ACS bands, |$V_{606}$| and |$I_{814}$|⁠, as high-resolution images. Correspondingly, we employ the HSC r- and i-band images, covering wavelengths similar to those of |$V_{606}$| and |$I_{814}$|⁠, respectively, as low-resolution images. These images are paired as r–|$V_{606}$| and i–|$I_{814}$|⁠. To avoid redundantly inputting images with similar galaxy structures, we rotate the r and |$V_{606}$| images by |$90^{\circ }$| from the original orientation. Table 3 presents the number of galaxies in the training sample. The training sample does not include the sources in the test sample (subsection 2.4), which is used to assess the generalization performance of GAN. The total number of galaxies in the training sample is |$N_{\rm train}\sim 20000$|–35000, which is roughly one order of magnitude larger than the dataset in previous studies (⁠|$N_{\rm train}\sim 600$|–4000; e.g., Schawinski et al. 2017; Gan et al. 2021). Although the number of galaxies for GAN may seem large, these galaxies are identified in small areas of the CANDELS fields, covering only |$\sim$|0.16 deg|$^2$| (Skelton et al. 2014), which is significantly smaller than the |$\sim$|300 deg|$^2$| of the HSC-SSP survey. By combining the HSC-SSP survey data and the super-resolution, we can study the relation between the galaxy mergers and the galaxy overdensity with large galaxy samples.

We use a publicly available PyTorch code of pix2pix cGAN.⁴ We set the default architectures of PatchGAN and resnet_9blocks as the discriminator and the generator, respectively. During the training, we augment the input data by flipping images randomly along the horizontal axis, which doubles the size of the training sample. The training time is |$\sim$|3–5 d for each HSC-SSP layer using GPU NVIDIA Quadro P5000.

To determine the optimal epochs for generating high-resolution images, we visually inspect images at various epochs for several randomly chosen sources during the training process. The bottom panels of figure 2 present images at different epochs of the GAN training process. Other example galaxies are presented in figure 15 in the Appendix. We find that the most favorable epochs are 230 for UltraDeep, 250 for Deep, and 270 for Wide. As shown in figure 2, similarly good images are output at different epochs (e.g., Epochs = 170 and 210). However, note that the scientific goal of this study is to measure the relative galaxy merger fraction in galaxy overdense regions and the field environment. Even if we fail to determine the best hyperparameters and ideal training epochs, we are able to achieve this scientific goal. By conducting the super-resolution analysis with consistent hyperparameters and training epochs, we can compare the galaxy merger fractions at different levels of galaxy overdensity under the same conditions.

The bottom right panel of figure 1 shows the example image generated with GAN.

3.2 Calculation of galaxy overdensity

To investigate the environmental dependence of galaxy mergers, we calculate the galaxy overdensity |$\delta$| in the same manner as in Toshikawa et al. (2016, 2018). The galaxy overdensity is defined as the excess in the local surface number density of galaxies,

where |$N_{\rm gal}$| and |$\overline{N_{\rm gal}}$| are the number of galaxies within a galaxy search aperture and its average, respectively. The radius of the galaxy search aperture is set to 0.75 physical Mpc, which corresponds to |${1{^{\prime }_{.}}5}$|⁠, |${1{^{\prime }_{.}}6}$|⁠, |${1{^{\prime }_{.}}8}$|⁠, and |${1{^{\prime }_{.}}9}$| for galaxies at |$z\sim 2$|⁠, |$z\sim 3$|⁠, |$z\sim 4$|⁠, and |$z\sim 5$|⁠, respectively. The galaxy search apertures are distributed in a grid pattern with |$1^{\prime }$| intervals. At each grid, we calculate |$\delta$| with the photo-z and dropout galaxies with |$m_{\rm UV}\lesssim 25$|–26. To estimate the area of sky regions affected by halos of bright stars in the galaxy search apertures, we use the random catalogs provided on the HSC-SSP database. The random catalogs contain randomly distributed data points over the HSC-SSP fields at a specific number density, which enables us to estimate effective survey areas and to identify problematic regions (Aihara et al. 2019). We use only galaxy search apertures whose masked area is narrower than 5% for the calculation of |$\delta$|⁠. Apertures with a |$\gt $|50% masked area are excluded from the analysis of the galaxy mergers and galaxy overdensity. After drawing the galaxy overdensity maps, we manually remove peculiar overdense regions that are likely to be caused by, e.g., unmasked light near bright stars. Figure 3 shows an example of the galaxy overdensity maps.

4 Results

4.1 Comparison of the super-resolution techniques

We compare the performance of the super-resolution techniques. Figure 4 shows high-resolution images obtained in the three super-resolution techniques for galaxies at |$z\sim 2$|–5. Basically, all the super-resolution techniques reproduce the galaxy shapes that resemble those seen in the Hubble images. The super-resolution techniques clearly reveal substructures of galaxies, e.g., individual components within galaxy mergers, at a scale smaller than the full width at half maximum (FWHM) of the HSC PSF (i.e., |$\sim$||${0{^{\prime \prime}_{.}}6}$|–|${1{^{\prime \prime}_{.}}0}$|⁠). Although the three techniques generate overall similar high spatial resolution images, there are advantages and disadvantages in each super-resolution technique. The classical RL PSF deconvolution excels in resolving galaxy substructures with close separations of |$\sim$||${0{^{\prime \prime}_{.}}1}$|–|${0{^{\prime \prime}_{.}}2}$|⁠. The restored images indicate that the classical RL PSF deconvolution has the highest resolving power in the super-resolution techniques. However, many residual noises are seen in the images of the classical RL PSF deconvolution. The sparse modeling significantly reduces the noises thanks to the sparseness and smoothness constraints, while tending to resolve close galaxy pairs less clearly than the classical RL PSF deconvolution. The GAN machine-learning technique generates realistic images with sky background noises similar to those in the Hubble images. Another advantage of GAN is the short amount of computing time needed to generate images, |$\lesssim$|0.01 s per image. Despite these merits, GAN requires a large dataset of HSC–Hubble image pairs to efficiently train the network.

To check the performance of each high-resolution technique quantitatively, we calculate the true positive rate (TPR) and the false positive rate (FPR) in identifying galaxy mergers. The TPR is defined as

where |$N_{\rm true}$| is the number of objects that are correctly selected as galaxy mergers in our super-resolution analysis (i.e., classical, sparse, GAN, or combined), and |$N_{\rm real-merger}$| is the total number of real galaxy mergers selected with the Hubble images. The FPR is defined as

where |$N_{\rm false}$| is the number of isolated galaxies that are incorrectly selected as galaxy mergers in our super-resolution analysis, and |$N_{\rm isolated}$| is the total number of isolated galaxies selected with the Hubble images. To obtain |$N_{\rm true}$| and |$N_{\rm false}$|⁠, we select galaxy mergers and isolated galaxies with the super-resolution images in the same manner as in subsection 2.2.

Figure 5 shows TPR and FPR as a function of magnitude. For all the super-resolution techniques, we find that the TPR (FPR) tends to increase (decrease) gradually toward bright magnitudes. The TPR (FPR) values in the UltraDeep fields are typically higher (lower) than those in the Wide fields. In all of the super-resolution techniques, the classical RL PSF deconvolution has the highest TPR and FPR values, which are consistent with the highest resolving power and the most noisy images, respectively. On the other hand, sparse modeling and GAN suppress FPR compared to the classical RL PSF deconvolution, especially in the Wide fields. However, sparse modeling and GAN show low TPR values. These TPR and FPR measurements confirm the qualitative evaluation of image characteristics in each super-resolution technique.

To obtain average features of restored galaxy images and to alleviate disadvantages associated with the individual techniques, we generate a combined image by averaging over the three types of super-resolution images. The top central panel of figure 1 shows an example of the combined images. Such an approach has been applied to data for the black hole shadows taken with EHT (Event Horizon Telescope Collaboration 2019a, 2019b). Before combining the images, we normalize the pixel values of each image to match the variations in the flux scale among the super-resolution images. To achieve this, we measure the total flux around the central regions of 25 pixels |$\times$| 25 pixels (⁠|${4{^{\prime \prime}_{.}}2} \times {4{^{\prime \prime}_{.}}2}$|⁠), which sufficiently cover the main components of galaxies. In the measurements, we exclude pixels with a pixel value with zero and negative flux because images obtained in the sparse modeling technique have many zero-value pixels. By summing the three types of pixel value-normalized super-resolution images, we obtain the average images.

As shown in figure 5, the combined images have the ability to identify galaxy mergers with a relatively high TPR of |$\gtrsim$|70%–80% and a low FPR of |$\lesssim$|5%–10% at bright magnitudes |$m\lesssim 23.5$|⁠. As our sample galaxies are bright, i.e., |$m\lt 23.5$|⁠, the galaxy merger fraction can be estimated at high TPR and low FPR values. We adopt the combined images as the representative super-resolution images for the galaxy merger fractions.

To make the analysis simple and to reduce potential systematic errors, we do not correct for TPR and FPR to calculate the galaxy merger fractions in this study. In Paper I, we correct for these TPR and FPR values in order to measure the absolute values of the galaxy merger fraction. The correction must be applied because the main goal of Paper I is to compare the merger fractions between bright and faint galaxies. On the other hand, the primary focus of this paper is to examine the relative trends of the galaxy merger fraction in different environments. Measuring the galaxy merger fractions for galaxies at a fixed redshift and comparable magnitudes ensures a consistent evaluation of the environmental dependence on galaxy mergers. The effect of magnitude difference on the galaxy merger fraction is discussed in subsection 4.3.

4.2 Relation between galaxy merger fraction and galaxy overdensity

We apply the super-resolution techniques to the galaxies at |$z\sim 2$|–5, and estimate the galaxy merger fraction as a function of galaxy overdensity. In this study, we measure the relative galaxy merger fraction with respect to the field environment, corrected for the galaxy chance projection effect, |$f_{\rm merger}^{\rm rel,cor}$|⁠. Figure 6 shows the super-resolution images of example galaxies at |$z\sim 2$|–5. By selecting galaxy mergers with the combined images of the super-resolution techniques, we first calculate the galaxy merger fraction, |$f_{\rm merger} = N_{\rm merger} / N$|⁠, where |$N_{\rm merger}$| and N are the numbers of galaxy mergers and sample galaxies, respectively. These numbers are obtained for the |$z\sim 2$|–5 galaxies with |$m_{\rm UV}\lt 23.5$| selected in subsection 2.3.

Next, we correct for the galaxy chance projection effect. Although we do not correct for TPR and FPR (subsection 4.1), we need to take into account the chance projection effect of foreground/background sources. Our method to identify galaxy mergers relies on the projected separation, lacking redshift information for individual galaxy components. In higher-|$\delta$| regions, there is a higher likelihood of chance identifications of foreground/background sources near isolated galaxies. We correct for the chance projection effect by calculating

where |$N_{\rm proj}$| is the expected number of chance projection sources in a galaxy sample. The expected number of chance projection sources per galaxy is estimated by multiplying the area of the galaxy merger search annulus |$S=\pi (r_{\rm max}^2-r_{\rm min}^2)$| by the average surface number density of galaxies n in the flux interval |$0.25 f_1 \le f \lt f_1$|⁠. To calculate n, we extract extended sources (i.e., galaxies) in the HSC-SSP UltraDeep COSMOS field by setting the extendedness_value flag to 1 (Aihara et al. 2018b). Then, we substitute |$N_{\rm proj}=n\times S$| for equation (10).

Finally, we shift the galaxy merger fractions so that |$f_{\rm merger}^{\rm col}$| nearest |$\delta =0$| is aligned to a specific value, serving as the measurement for the field environment. We choose |$f_{\rm merger}^{\rm col}=0.2$| as the galaxy merger fraction for the field environment, which is just for visualization purposes. The shifted |$f_{\rm merger}^{\rm col}$| is referred to as the relative galaxy merger fraction, |$f_{\rm merger}^{\rm rel,cor}$|⁠.

Figure 7 presents |$f_{\rm merger}^{\rm rel,cor}$| as a function of |$\delta$|⁠. The |$1\sigma$| uncertainties of |$f_{\rm merger}^{\rm rel,col}$| are calculated by taking into account Poisson confidence limits (Gehrels 1986) on the numbers of the sources. For |$z\sim 4$|–5, the low, intermediate, and high overdensity bins are defined as |$\delta _{\rm low}\lt 2$|⁠, |$\delta _{\rm mid}=2$|–4, and |$\delta _{\rm high}=4$|–10, respectively. For |$z\sim 2$|–3, the samples are divided by a small |$\delta$| value of |$\delta =0.5$|⁠, i.e., |$\delta _{\rm low}\lt 0.5$|⁠, |$\delta _{\rm mid}=0.5$|–4 due to relatively small |$\delta$| coverages. Table 4 lists |$f_{\rm merger}^{\rm rel,cor}$| in each galaxy overdensity bin. We also plot previous studies for |$z\sim 2$|–3 galaxy protoclusters whose |$f_{\rm merger}$| and |$\delta$| are available: Hine et al. (2016) and Monson et al. (2021) for the SSA22 galaxy protocluster at |$z=3.09$| and Liu et al. (2023) for two galaxy protoclusters of BOSS 1244 and BOSS 1542 at |$z=2.24$|⁠. The galaxy merger fractions for the previous studies are shifted in the same way as our measurements.

Despite the relatively narrow |$\delta$| coverage, our |$f_{\rm merger}^{\rm rel,col}$| measurements at |$z\sim 3$| validate the results of the previous studies suggesting that |$f_{\rm merger}^{\rm rel,col}$| increases with increasing |$\delta$| at |$z\sim 2$|–3 (Hine et al. 2016; Liu et al. 2023). In addition, we find a similar increasing trend of |$f_{\rm merger}^{\rm rel,col}$| at higher redshifts of |$z\sim 4$|–5. To our knowledge, this represents the highest-z evidence that galaxy mergers occur more frequently in denser regions, identified with a statistical galaxy sample. The combination of the wide-area HSC-SSP data and the super-resolution techniques has allowed us to reveal the relation between |$f_{\rm merger}^{\rm rel,col}$| and |$\delta$| at the highest redshifts. Interestingly, |$f_{\rm merger}^{\rm rel, col}$| at |$z\sim 4$|–5 appears to increase almost linearly with increasing |$\delta$|⁠. The |$f_{\rm merger}^{\rm rel,col}$| value at |$z\sim 4$|–5 is fitted with the following linear function:

The uncertainty in the parenthesis in equation (11) represents the |$1.5\sigma$| typical scatter of the |$z\sim 4$|–5 data points (the magenta shaded region in figure 7). The reason for the slight difference between the intercept of equation (11) (i.e., 0.19) and |$f_{\rm merger}^{\rm rel,col}=0.2$| is that the average |$\delta$| value within the |$\delta$| bin of the field environment is slightly offset from |$\delta =0$| (table 4). We find that |$f_{\rm merger}^{\rm rel,col}$| measured in this study and the literature, including Monson et al. (2021) who suggest that |$f_{\rm merger}^{\rm rel,col}$| does not correlate with |$\delta$|⁠, broadly follows the linear |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation within the uncertainty. The lack of |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| correlation of Monson et al. (2021) might result from a small sample of galaxies (⁠|$N_{\rm galaxy}=63$|⁠).

In contrast, the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation for our |$z\sim 2$| sample is weak or almost flat. The weak or missing |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation at |$z\sim 2$| is perhaps due to the narrower area of the CLAUDS survey than the HSC-SSP survey (table 2). A wider survey for |$z\sim 2$| might be required to confirm the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation at |$z\sim 2$|⁠.

In subsection 5.1, we conduct detailed comparisons between |$f_{\rm merger}^{\rm rel,col}$| measured in this study and the literature at |$z\sim 0$|–1 over a wide |$\delta$| range of |$\delta \sim 0$|–20 to investigate whether the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation evolves with redshift or not.

4.3 Is the galaxy merger–overdensity relation real?

We examine whether the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| trend is real or not. Our |$f_{\rm merger}^{\rm rel,col}$| results are based on the application of the super-resolution techniques that computationally enhance the spatial resolution of the seeing-limited HSC images. The |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| trend might be artificially produced during the image processing analysis because of some possible systematics. Here we check whether the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation is affected by differences in (1) the super-resolution techniques, (2) the magnitude between the |$\delta$| bins, and (3) the depth of the survey fields.

1. Difference in the super-resolution techniques. We test whether the difference in the super-resolution techniques affects the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation. Figure 8 presents |$f_{\rm merger}^{\rm rel,col}$| obtained with the three super-resolution techniques. At |$z\sim 2$| and |$z\sim 3$|⁠, we confirm that the classical RL PSF deconvolution, sparse modeling, and GAN-based |$f_{\rm merger}^{\rm rel,col}$| are in good agreement with those of the combined images. Similarly, at |$z\sim 4$|–5, we also find that |$f_{\rm merger}^{\rm rel,col}$| measured in the three super-resolution techniques follows the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation given the uncertainties. These agreements indicate that the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation does not strongly depend on the super-resolution techniques.

2. Difference in the magnitude between the |$\delta$| bins. We examine the difference in magnitude between the |$\delta$| bins. Typically, galaxy mergers are identified with a higher TPR value for brighter galaxies (figure 5). If bright galaxies are located selectively in high-|$\delta$| regions, it might potentially result in an artificial increasing |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| trend due to the high TPR value. Figure 9 shows the histograms of magnitude between the |$\delta$| bins and the median values of magnitudes. Within the |$1\sigma$| uncertainties, the average magnitudes are comparable between the |$\delta$| bins. In addition, the magnitude distribution appears similar between the |$\delta$| bins. To quantify the difference in the magnitude distribution, we perform Kolmogorov–Smirnov (KS) tests. Table 5 summarizes the p-values of the KS tests. We find that the p-values exceed the commonly used significance level of 5% for almost all the distributions. The KS tests cannot reject a null hypothesis that two samples are drawn from a statistically identical distribution. Among the distributions, one combination of the low- and mid-|$\delta$| bins, (⁠|$\delta _{\rm low}, \delta _{\rm mid}$|⁠), at |$z\sim 4$|–5 shows |$p\lt 5$|%. To reduce the magnitude difference, we measure |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| in a relatively narrow magnitude range of |$m=23$|–23.5. With this narrow m range, the KS test for the (⁠|$\delta _{\rm low}, \delta _{\rm mid}$|⁠) combination shows |$p\gt 5$|% (table 5). We find that the new |$f_{\rm merger}^{\rm rel,col}$| values measured in the narrow m range still follow the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation given the uncertainties (figure 10). The comparable median magnitudes and the p-values suggest that the difference in magnitude between the |$\delta$| bins does not significantly influence the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation.

3. Difference in the depth of the survey fields. We investigate whether the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation is affected by the depths of the different survey fields. Rare high-|$\delta$| regions are typically identified in the HSC Wide fields whose FPR is higher than those in the HSC UltraDeep and Deep fields (figure 5). There is a possibility that |$f_{\rm merger}^{\rm rel,col}$| is systematically higher in the higher-|$\delta$| bins due to the high FPR value of the HSC Wide fields. To examine the effect, we measure |$f_{\rm merger}^{\rm rel,col}$| using only the galaxy sample in the HSC Wide fields (figure 10).⁵We find that |$f_{\rm merger}^{\rm rel,col}$| obtained in only the HSC Wide fields agrees with the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation within the uncertainty. The agreement suggests that the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation is not a result of the difference in the depth of the survey fields.

These tests indicate that the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation is unlikely to be caused by the differences in the super-resolution techniques, magnitudes between the |$\delta$| bins, or the depth of the survey fields. Therefore, we conclude that the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation is real.

5 Discussion

5.1 Comparisons with previous studies at low redshifts; redshift evolution of the galaxy merger fraction–galaxy overdensity relation

We compare |$f_{\rm merger}^{\rm rel,col}$| in this study with results from previous studies at low redshifts to investigate whether the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation evolves or remains unchanged over cosmic time. As described below, the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation does not strongly evolve in a redshift range of |$z\sim 2$|–5, while the galaxy merger fractions are possibly suppressed in massive galaxy clusters at |$z\lesssim 1$|⁠. In subsection 4.2, we have found that the galaxy merger fraction broadly follows the linear |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation at high redshifts of |$z\sim 2$|–5. Here we include previous studies at |$z\lesssim 1$| and a high galaxy overdensity of |$\delta \gtrsim 5$| as comparison samples. Table 6 summarizes observational studies examining the relation between galaxy mergers and galaxy environments. The Appendix provides details about the previous studies, e.g., galaxy samples and methods to estimate the galaxy overdensity for each work. For the plot, we select previous studies whose galaxy merger fractions are available for both field and overdense environments and that have been measured with the galaxy close-pair method. Although the methods to identify galaxy mergers are slightly different from ours (e.g., the separation between galaxy merger components), the comparisons yield insights into understanding the general trend of the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation.

Figure 11 presents |$f_{\rm merger}^{\rm rel,col}$| as a function of |$\delta$| including three previous studies of Lotz et al. (2013), Laishram et al. (2024), and Lin et al. (2010). Lotz et al. (2013) estimated the frequency of double nuclei or close companions in a |$\delta \sim 20$| massive protocluster at |$z\sim 1.6$|⁠. Laishram et al. (2024) investigated the morphological disturbance of galaxies in extensive filamentary structures at |$z\sim 1.5$|⁠. Lin et al. (2010) used a spectroscopic sample of |$\sim$|20000 galaxies, and evaluated the galaxy merger fraction at two average redshifts of |$z=1.08$| and |$z=0.88$|⁠. These studies suggest that |$f_{\rm merger}^{\rm rel,col}$| increases with increasing |$\delta$|⁠. At |$\delta \sim 10$| and 20, we find that the galaxy merger fractions at the high redshift of |$z\sim 1.5$|–1.6 follows the extrapolation of the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation within the uncertainty. However, the two |$f_{\rm merger}$| measurements of Lin et al. (2010) at |$z\sim 1$| show significant deviations from the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation. The deviation from the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation appears to be more significant at lower redshifts.

In addition to |$f_{\rm merger}^{\rm rel,col}$|⁠, we measure the enhancement ratio of the galaxy merger fraction, |$R_{\rm merger}$|⁠, to reduce the impact of the redshift evolution of |$f_{\rm merger}$| in the field environment (e.g., Lotz et al. 2011; Man et al. 2016; Mundy et al. 2017; Mantha et al. 2018; Conselice et al. 2022). The enhancement ratio of the galaxy merger fraction is defined as

where |$f_{\rm merger}^{\rm rel,col}(\delta \sim 0)$| indicates the galaxy merger fraction at |$\delta \sim 0$|⁠.

Figure 12 presents the enhancement ratio of the galaxy merger fraction as a function of |$\delta$|⁠. From equation (11), the linear |$R_{\rm merger}$|–|$\delta$| relation (the magenta line and magenta shaded region) is calculated as

This |$R_{\rm merger}$| function is roughly consistent with the environmental dependence predicted by an N-body simulation of Fakhouri and Ma (2009) suggesting that the halo merger rate at |$z\sim 2$| is |$\sim$|1.2–1.3 times enhanced at |$\delta =3$| compared to the field environment. In figure 12, we add three enhancement ratios of the galaxy merger fraction measured using 10644 galaxies at |$z\sim 0$|–1 in de Ravel et al. (2011). As in the figure of |$f_{\rm merger}^{\rm rel,col}$| (figure 11), almost all of the |$f_{\rm merger}^{\rm rel,col}$| measurements broadly agree with the |$R_{\rm merger}$|–|$\delta$| relation at |$\delta \sim 0$|–10 although some data points slightly deviate from the error region (e.g., a data point in Liu et al. 2023). In particular, at a high galaxy overdensity of |$\delta \sim 20$|⁠, |$R_{\rm merger}$| at |$z\sim 1$|–2 shows a good agreement with the |$R_{\rm merger}$|–|$\delta$| relation. However, |$R_{\rm merger}$| at |$z\lt 1$| might still be below the |$R_{\rm merger}$|–|$\delta$| relation.

In summary, the environmental dependence of galaxy mergers does not strongly evolve over cosmic time of |$z\sim 2$|–5 according to the diagrams of |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| and |$R_{\rm merger}$|–|$\delta$|⁠. However, in massive galaxy clusters at |$z\lesssim 1$|⁠, the galaxy merger fraction appears to be suppressed compared to the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation, and possibly even below the |$R_{\rm merger}$|–|$\delta$| relation. This trend is consistent with predictions of theoretical studies with N-body simulations (e.g., Gottlöber et al. 2001), and semi-analytic models (e.g., Jian et al. 2012). These theoretical studies predict that |$f_{\rm merger}$| tends to be high in galaxy group/cluster environments at |$z\gt 1$|⁠, while |$f_{\rm merger}$| rapidly decreases in massive galaxy clusters at |$z\lt 1$|⁠. The decrease in |$f_{\rm merger}$| is attributed to high-velocity dispersions of massive galaxy clusters, where high-velocity galaxy encounters hinder galaxy mergers and interactions (Ostriker 1980; Makino & Hut 1997). In addition, we have found that this |$f_{\rm merger}$| suppression at high |$\delta$| takes place at |$z\sim 1$|⁠, which is roughly consistent with the predicted redshift when |$f_{\rm merger}$| is suppressed in simulations (Gottlöber et al. 2001). Our |$f_{\rm merger}$|–|$\delta$| and |$R_{\rm merger}$|–|$\delta$| relations might be useful and provide additional constraints on cosmological simulations for galaxy overdense regions at high z (e.g., Sembolini et al. 2013; Chiang et al. 2013; Muldrew et al. 2015; Contini et al. 2016; Bahé et al. 2017; Barnes et al. 2017; Henden et al. 2018; Cui et al. 2018; Tremmel et al. 2019; Yajima et al. 2022; Nakazato et al. 2024).

5.2 Role of galaxy mergers in galaxy protoclusters

We discuss the role of galaxy mergers in the environmental dependence of galaxy populations. As presented in subsection 5.1 and figure 12, we have found that the enhancement ratio of the galaxy merger fraction, |$R_{\rm merger}$|⁠, can be roughly predicted from the galaxy overdensity at least in the redshift range of |$z\sim 2$|–5. Using the |$R_{\rm merger}$|–|$\delta$| relation, we are able to evaluate the contribution level of galaxy mergers to establishing various types of high-z environmental dependence. As a demonstration, we focus on one of the widely studied populations for the environmental dependences: AGN. Because galaxy mergers and interactions are thought to trigger AGN activity (e.g., Keel et al. 1985; Sanders et al. 1988; Di Matteo et al. 2005; Urrutia et al. 2008; Koss et al. 2010; Silverman et al. 2011; Ellison et al. 2011; Satyapal et al. 2014; Goulding et al. 2018; see also a summary table of Villforth 2023), the AGN fraction, |$f_{\rm AGN}$|⁠, is expected to be enhanced in high-z galaxy protoclusters, as found in the literature (e.g., Lehmer et al. 2009, 2013; Digby-North et al. 2010; Hashiguchi et al. 2023; Toba et al. 2024). In this analysis, we use a summary table of Macuga et al. (2019), which compiles measurements of |$f_{\rm AGN}$| for galaxy protoclusters and the field environment at |$z\sim 2$|–3.

As in the equation of |$R_{\rm merger}$| (equation 12), the enhancement ratio of |$f_{\rm AGN}$| is defined as

where |$f_{\rm AGN}(\delta \sim 0)$| is the AGN fraction in the field environment. Using the |$f_{\rm AGN}$| and |$f_{\rm AGN}(\delta \sim 0)$| values in the summary table of Macuga et al. (2019), we calculate |$R_{\rm AGN}$| for six types of |$z\sim 2$|–3 galaxy protoclusters that show the enhancement of AGN abundance.

Figure 13 presents the enhancement ratios of the galaxy merger fraction and the AGN fraction. We also show the fractional contribution of galaxy mergers relative to AGN, |$R_{\rm merger}/R_{\rm AGN}$|⁠. The fractional contribution |$R_{\rm merger}/R_{\rm AGN}$| varies among the galaxy protoclusters, ranging from |$\sim$|10% to |$\gtrsim$|100%. It is notable that the majority of galaxy protoclusters have |$R_{\rm merger}/R_{\rm AGN}\gtrsim 30$|%, except for BX/MD galaxies in QSO HS 1700|$+$|643 with |$R_{\rm merger}/R_{\rm AGN}\sim 10$|%. The high |$R_{\rm merger}/R_{\rm AGN}$| fractions suggest that galaxy mergers play an important role in enhancing the AGN abundances in galaxy protoclusters at |$z\sim 2$|–3. In contrast, in galaxy protoclusters with low |$R_{\rm merger}/R_{\rm AGN}$|⁠, other physical mechanisms might considerably contribute to the AGN enhancement, such as smooth gas accretion to supermassive black holes (e.g., Kereš et al. 2005; Dekel et al. 2009; van de Voort et al. 2011). The difference in |$R_{\rm merger}/R_{\rm AGN}$| could also originate from cosmic variance due to the limited sample.

It should be noted that we simply compare the enhancement ratios of |$R_{\rm merger}$| and |$R_{\rm AGN}$| in this discussion. An agreement of |$R_{\rm merger}$| and |$R_{\rm AGN}$| does not necessarily imply a direct connection between galaxy mergers and the AGN enhancement in galaxy overdense regions. It is important to take into account several factors related to the AGN triggering processes, e.g., the time delay between galaxy mergers and the onset of gas accretion on to black holes (e.g., Springel et al. 2005; Hopkins 2012; Choi et al. 2023). The purpose of this discussion is to demonstrate how the |$R_{\rm merger}$|–|$\delta$| relation can be used to estimate quantitatively the contributions of physical mechanisms triggering the AGN activity. In addition, one might think that the super-resolution of AGN host galaxies provides more direct evidence of the physical connection between galaxy mergers and the AGN activity. However, we find it difficult to investigate the morphology of AGN host galaxies with the current super-resolution techniques, because faint structures of host galaxies are strongly affected by the light of central bright point sources. This is why we have analyzed only extended objects, i.e., galaxies, in this study (subsection 2.3). Improvements of the super-resolution techniques for AGN host galaxies are left for future work.

A similar analysis about the galaxy merger contribution can be conducted for various kinds of environmental dependences at high z, for example, star-formation rate (e.g., Elbaz et al. 2007; Koyama et al. 2013; Shimakawa et al. 2018; Kubo et al. 2019; Lemaux et al. 2022), sub-millimeter galaxies (e.g., Tamura et al. 2009; Umehata et al. 2015; see also Uchiyama et al. 2018), Ly|$\alpha$| blobs (e.g., Matsuda et al. 2004; Prescott et al. 2008; Yang et al. 2010; Cai et al. 2017), and bright cluster galaxies (e.g., Tran et al. 2008; Ito et al. 2019). Our galaxy merger–overdensity relations should provide hints for unveiling the physical origins of these high-z environmental dependences.

6 Summary and conclusions

We have super-resolved the seeing-limited Subaru HSC images with three techniques: classical RL PSF deconvolution (sub-subsection 3.1.1), sparse modeling (sub-subsection 3.1.2), and GAN (sub-subsection 3.1.3). By applying the super-resolution techniques to 32187 galaxies at |$z\sim 2$|–5 selected from survey areas of |$\sim$|300 deg|$^2$| in HSC-SSP for |$z\sim 4$|–5 and |$\sim$|20 deg|$^2$| in CLAUDS for |$z\sim 2$|–3, we have examined the environmental dependence of galaxy mergers at |$z\sim 2$|–5.

The major findings are summarized as follows:

1. The three super-resolution techniques generate overall similar high spatial resolution HSC images (figure 1). In the qualitative and quantitative comparisons, we find that there are advantages and disadvantages in each super-resolution technique; for example, many noises tend to be seen in the classical RL PSF deconvolution, while the resolving powers of the sparse modeling and GAN are relatively low (figures 4 and 5). To alleviate the disadvantages of each super-resolution technique, we combine the three types of super-resolution images. The combined super-resolution images show galaxy substructures that closely resemble those seen in the Hubble images.

2. Using the combined super-resolution images, we measure the relative galaxy merger fraction corrected for the chance projection effect, |$f_{\rm merger}^{\rm rel,col}$|⁠, at |$z\sim 2$|–5 as a function of galaxy overdensity |$\delta$|⁠. With the wide-area survey data and the super-resolution images, our |$f_{\rm merger}^{\rm rel,col}$| measurements at |$z\sim 3$| validate the results of previous studies showing that |$f_{\rm merger}^{\rm rel,col}$| is higher in higher-|$\delta$| regions at |$z\sim 2$|–3. In addition, we identify a trend that |$f_{\rm merger}^{\rm rel,col}$| increases almost linearly with increasing |$\delta$| at |$z\sim 4$|–5 (figure 7). This is the highest-z observational evidence that the galaxy merger fraction is related to |$\delta$|⁠. The enhancement ratio of the galaxy merger fraction, |$R_{\rm merger}$| (equation 12), shows a similar linear trend to the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation. In contrast to |$z\gtrsim 3$|⁠, we have not found a large |$f_{\rm merger}^{\rm rel,col}$| increase with increasing |$\delta$| at |$z\sim 2$|⁠; wider survey data may be needed to confirm the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| relation at |$z\sim 2$|⁠.

3. We find that the |$f_{\rm merger}^{\rm rel,col}$| and |$R_{\rm merger}$| values measured in this study for |$z\sim 2$|–5 and in the literature for |$z\sim 2$|–3 broadly align with the linear |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| and |$R_{\rm merger}$|–|$\delta$| relations (figures 11 and 12). These alignments indicate that the environmental dependence of galaxy mergers does not change greatly, at least in a redshift range of |$z\sim 2$|–5, allowing us to quantitatively estimate the contributions of galaxy mergers to various environmental dependences. In contrast, in massive galaxy clusters at |$z\lesssim 1$|⁠, the galaxy merger fraction is possibly suppressed compared to the |$f_{\rm merger}^{\rm rel,col}$|–|$\delta$| and |$R_{\rm merger}$|–|$\delta$| relations. This suppression is consistent with theoretical studies predicting that |$f_{\rm merger}$| decreases in massive galaxy clusters at |$z\lt 1$| due to the high-velocity dispersion of low-z massive galaxy clusters, hindering galaxy–galaxy interactions.

4. Using the |$R_{\rm merger}$|–|$\delta$| relation, we estimate |$R_{\rm merger}$| for |$z\sim 2$|–3 galaxy protoclusters with high AGN abundances to investigate how galaxy mergers contribute to the enhancement ratio of the AGN fraction, |$R_{\rm AGN}$| (equation 14). We find that the fraction of |$R_{\rm merger}$| relative to |$R_{\rm AGN}$| largely depends on the galaxy protoclusters, ranging from |$R_{\rm merger}/R_{\rm AGN}\sim 10$|% to |$\gtrsim$|100%, but |$R_{\rm merger}/R_{\rm AGN}\gtrsim 30$|% for almost all of the galaxy protoclusters examined in this study (figure 13). The high |$R_{\rm merger}/R_{\rm AGN}$| fraction suggests that galaxy mergers play an important role in enhancing AGN abundances in galaxy protocluster regions. In contrast, in galaxy protoclusters with a relatively low |$R_{\rm merger}/R_{\rm AGN}$| fraction, other physical mechanisms may predominantly drive the AGN enhancement, such as smooth cold gas accretion to galaxy overdense regions.

This super-resolution analysis has been made possible with a combination of wide-area Subaru HSC and high spatial resolution Hubble data. This method can be easily applied to combinations of space telescopes, e.g., Euclid–JWST and Roman–JWST. These combinations would enable us to investigate morphological properties of rare galaxy populations at higher redshifts and on smaller scales than in the study with Subaru–Hubble.

Acknowledgments

We thank the anonymous referee for valuable comments and suggestions that improved the manuscript. We acknowledge Rieko Momose, Kiyoaki Christopher Omori, Yoshiaki Ono, Taira Oogi, and Masayuki Yamaguchi for useful comments. This work was supported by JSPS KAKENHI Grant Numbers 20K14508, 22H01266, 22K14076, and 24H00002.

The HSC collaboration includes the astronomical communities of Japan and Taiwan, and Princeton University. The HSC instrumentation and software were developed by the National Astronomical Observatory of Japan (NAOJ), the Kavli Institute for the Physics and Mathematics of the Universe (Kavli IPMU), the University of Tokyo, the High Energy Accelerator Research Organization (KEK), the Academia Sinica Institute for Astronomy and Astrophysics in Taiwan (ASIAA), and Princeton University. Funding was contributed by the FIRST program from the Japanese Cabinet Office, the Ministry of Education, Culture, Sports, Science and Technology (MEXT), the Japan Society for the Promotion of Science (JSPS), the Japan Science and Technology Agency (JST), the Toray Science Foundation, NAOJ, Kavli IPMU, KEK, ASIAA, and Princeton University.

This paper makes use of software developed for the Large Synoptic Survey Telescope. We thank the LSST Project for making their code available as free software at |$\langle$|http://dm.lsst.org|$\rangle$|⁠.

This paper is based on data collected at the Subaru Telescope and retrieved from the HSC data archive system, which is operated by the Subaru Telescope and Astronomy Data Center (ADC) at NAOJ. Data analysis was in part carried out with the cooperation of the Center for Computational Astrophysics (CfCA), NAOJ.

The Pan-STARRS1 Surveys (PS1) have been made possible through contributions of the Institute for Astronomy, the University of Hawaii, the Pan-STARRS Project Office, the Max-Planck Society and its participating institutes, the Max Planck Institute for Astronomy, Heidelberg and the Max Planck Institute for Extraterrestrial Physics, Garching, The Johns Hopkins University, Durham University, the University of Edinburgh, Queen’s University Belfast, the Harvard-Smithsonian Center for Astrophysics, the Las Cumbres Observatory Global Telescope Network Incorporated, the National Central University of Taiwan, the Space Telescope Science Institute, the National Aeronautics and Space Administration under Grant No. NNX08AR22G issued through the Planetary Science Division of the NASA Science Mission Directorate, the National Science Foundation under Grant No. AST-1238877, the University of Maryland, and Eotvos Lorand University (ELTE) and the Los Alamos National Laboratory.

This research is based on data collected at the Subaru Telescope, which is operated by the National Astronomical Observatory of Japan. We are honored and grateful for the opportunity of observing the Universe from Maunakea, which has cultural, historical, and natural significance in Hawaii.

These data were obtained and processed as part of the CFHT Large Area U-band Deep Survey (CLAUDS), which is a collaboration between astronomers from Canada, France, and China described in Sawicki et al. (2019). CLAUDS is based on observations obtained with MegaPrime/MegaCam, a joint project of CFHT and CEA/DAPNIA, at the CFHT, which is operated by the National Research Council (NRC) of Canada, the Institut National des Sciences de l’Univers of the Centre National de la Recherche Scientifique (CNRS) of France, and the University of Hawaii. CLAUDS uses data obtained in part through the Telescope Access Program (TAP), which has been funded by the National Astronomical Observatories, the Chinese Academy of Sciences, and the Special Fund for Astronomy from the Ministry of Finance of China. CLAUDS uses data products from TERAPIX and the Canadian Astronomy Data Centre (CADC) and was carried out using resources from Compute Canada and the Canadian Advanced Network For Astrophysical Research (CANFAR).

This research made use of SExtractor (Bertin & Arnouts 1996), SAOImage DS9 (Joye & Mandel 2003), Numpy (Harris et al. 2020), Matplotlib (Hunter 2007), Scipy (Virtanen et al. 2020), Astropy (Astropy Collaboration 2013, 2018),⁶ and Ned Wright’s Javascript Cosmology Calculator (Wright 2006).⁷

Appendix 1 Other example galaxies

Figures 14 and 15 show other example galaxies with different hyperparameters of sparse modeling and different epochs of GAN, respectively.

Appendix 2 Previous observational studies on the relation between galaxy mergers and galaxy environments

This section presents details of previous observational studies investigating the relation between galaxy mergers and galaxy environments.

Table 6 summarizes the redshift and the number of galaxies, the methods to identify galaxy mergers, and whether |$f_{\rm merger}$| is related to |$\delta$|⁠.

A.2.1 Hine et al. (2016)

Hine et al. (2016) investigate the morphology of 23 Lyman-break galaxies (LBGs) in the SSA22 galaxy protocluster at |$z\sim 3.1$| and 86 coeval LBGs in the field environment. By visually classifying the LBGs using the Hubble ACS |$I_{814}$|-band images, which trace the rest-frame UV light at |$\lambda \sim 2000$| Å, the study reveals a galaxy merger fraction of |$0.48\pm 0.10$|⁠, surpassing the |$0.30\pm 0.05$| observed in the field environment. This finding suggests an enhanced galaxy merger fraction in galaxy protocluster environments. For the plot in figure 7, we adopt |$\delta =6$| as the galaxy overdensity of the SSA22 protocluster (Steidel et al. 2000) and |$f_{\rm merger}$| of the “combined field” as the galaxy merger fraction in the field environment (table 1 of Hine et al. 2016).

A.2.2 Monson et al. (2021)

Monson et al. (2021) measure the galaxy merger fractions for eight LBGs and six AGN in the SSA22 galaxy protocluster and 49 field LBGs at |$z\sim 3.1$| using the Hubble WFC3 F160W images, which trace the rest-frame optical light at |$\lambda \sim 4000$| Å. Morphological analyses with Sérsic profile fitting, the |$GM_{20}$| coefficients, and visual classifications suggest no significant differences in |$f_{\rm merger}$| between the SSA22 galaxy protocluster and field LBGs. As with Hine et al. (2016), we adopt |$\delta =6$| as the galaxy overdensity of the SSA22 galaxy protocluster (Steidel et al. 2000) to plot |$f_{\rm merger}$| in figure 7. We use the |$f_{\rm merger}$| values for the “Total Field” and “Protocluster” as |$f_{\rm merger}$| in the field and protocluster environments, respectively (table 5 of Monson et al. 2021).

A.2.3 Mei et al. (2023)

Mei et al. (2023) visually classify galaxies in 16 spectroscopically confirmed galaxy clusters at |$z\sim 1.4$|–2.8 from the Clusters Around Radio-Loud AGN (CARLA) survey (Wylezalek et al. 2013, 2014) and field galaxies from the CANDELS survey. The total number of galaxies in the CARLA galaxy protoclusters is 271. The study finds a higher galaxy merger fraction of |$f_{\rm merger}=0.26\pm 0.03$| for the galaxy clusters than |$f_{\rm merger}=0.17\pm 0.05$| for the CANDELS fields, but no significant dependence of |$f_{\rm merger}$| on the local environment within the CARLA sample.

A.2.4 Momose et al. (2022)

Momose et al. (2022) examine the relation between galaxy mergers at |$z\sim 2$|–2.5 and the underlying 3D dark matter density reconstructed from galaxies and Ly|$\alpha$| forest spectroscopic data. To measure |$f_{\rm merger}$|⁠, the study cross-matches 1344 spectroscopically confirmed galaxies with close-pair objects in galaxy merger catalogs (Silva et al. 2018; Shah et al. 2020). The analysis suggests that |$f_{\rm merger}$| does not clearly correlate with the 3D dark matter density.

A.2.5 Liu et al. (2023)

Liu et al. (2023) focus on two massive galaxy protoclusters, BOSS 1244 and BOSS 1542, at |$z=2.24$|⁠. The galaxy overdensities are |$\delta =5.6$| and |$\delta =4.9$| for the BOSS 1244 and BOSS 1542 galaxy protoclusters, respectively. Using the Hubble WFC3 F160W data, the study investigates morphologies of 122 H|$\alpha$| emitters in the galaxy protoclusters and 455 |$z\sim 2$|–3 field galaxies from the CANDELS survey. Galaxy mergers are identified using the close-pair technique with |$\mu \ge 1/4$| and |$d=5$|–30 kpc. The galaxy merger fractions are |$f_{\rm merger}=0.22\pm 0.05$| in BOSS 1244 and |$f_{\rm merger}=0.33\pm 0.06$| in BOSS 1542, which are 1.8 and 2.8 times higher than those in the general fields, respectively.

A.2.6 Delahaye et al. (2017)

Delahaye et al. (2017) compile photometric and spectroscopic samples of 55 galaxies in four galaxy clusters at |$z=1.59$|–1.71 from the SpARCS survey (Muzzin et al. 2009) and 91 field galaxies from the UKIDSS Deep Survey (Lawrence et al. 2007). The visual inspection is conducted with Hubble NIR imaging data to identify galaxy mergers. The study finds that the galaxy merger fraction is |$f_{\rm merger}=0.110^{+0.070}_{-0.056}$| for the galaxy cluster members, compared to |$f_{\rm merger}=0.247^{+0.053}_{-0.046}$| for the field galaxies, ruling out the possibility of a mild or strong enhancement of the galaxy merger activity in galaxy clusters compared to the field.

A.2.7 Coogan et al. (2018)

Coogan et al. (2018) observe 11 galaxies in the galaxy cluster Cl J1449|$+$|0856 at |$z=1.99$| using ALMA and JVLA. Visual inspections of the Hubble images, CO maps, and CO line profiles suggest that 5 out of the 11 galaxies are mergers or interacting systems, indicating a large number of mergers in the galaxy cluster core compared to the field environment.

A.2.8 Sazonova et al. (2020)

Sazonova et al. (2020) study morphological properties for 498 galaxies in four galaxy clusters at |$z=1.2$|–1.8 using the Hubble WFC3 images. The study uses galaxies in CANDELS as a control sample. Based on the morphological parameters of A and |$GM_{20}$| and close-pair signatures, the researchers calculate the disturbance fractions and close neighbor fractions. The study finds that these fractions in the galaxy clusters are typically higher than those in the field.

A.2.9 Lotz et al. (2013)

Lotz et al. (2013) research a galaxy protocluster, IRC-0218A, at |$z=1.62$| using the Hubble WFC3 F125W and F160W data. The galaxy overdensity of IRC-0218A is |$\delta =20$|⁠. By identifying double nuclei or close pairs with |$d\lt 20$| kpc, the study measures |$f_{\rm merger}$| for 14 galaxies in the galaxy protocluster and 130 galaxies at |$z\sim 1.6$| in the CANDELS fields. The |$f_{\rm merger}$| measurements indicate that galaxy mergers in the galaxy protocluster are |$\sim$|3–10 times more abundant than in the field environment. For the plots in figures 11 and 12, we use the |$f_{\rm merg}{\rm (cor)}$| values for the “Cluster all” and “Field |$z\sim 1.6$|” samples (table 1 of Lotz et al. 2013).

A.2.10 Watson et al. (2019)

Watson et al. (2019) investigate two galaxy clusters, IRC 0222A at |$z\sim 1.6$| and IRC 0222B at |$z\sim 1.9$|⁠, identified through Spitzer mid-infrared observations. Using the Hubble F105W-, F125W-, and F160W-band images, the study morphologically classifies 61 galaxies in the galaxy clusters and 888 field galaxies at |$z\sim 1.5$|–2.5 in fields of COSMOS (Scoville et al. 2007) and CDFS (Giacconi et al. 2002). To identify galaxy mergers, the researchers combine visual inspections, the close-pair technique, and selection criteria with grism spectroscopic redshifts. The study finds galaxy merger fractions of |$f_{\rm merger}=0.110^{+0.082}_{-0.032}$| in IRC 0222A and |$f_{\rm merger}=0.180^{+0.078}_{-0.045}$| in IRC 0222B, which are higher than the measurement in the field, |$f_{\rm merger}=0.050^{+0.011}_{-0.008}$|⁠.

A.2.11 Laishram et al. (2024)

Laishram et al. (2024) examine the morphological features of [O ii] emitters in extensive filamentary structures at |$z\sim 1.5$| discovered with a narrow-band filter of Subaru HSC. The filamentary structures have a highly overdense core with a galaxy number density |$\sim$|11 times higher than the field average. By investigating the morphology of 368 galaxies with JWST NIRCam and the |$GM_{20}$| indices, the study finds that galaxies in the dense core show a higher frequency of disturbances (⁠|$50\%\pm 9\%$|⁠) compared to those in the outskirts (⁠|$41\%\pm 9\%$|⁠) and the field (⁠|$21\%\pm 5\%$|⁠), indicating more frequent mergers and interactions in denser environments.

A.2.12 Lin et al. (2010)

Lin et al. (2010) explore galaxy mergers at |$z=0.75$|–1.2 using a sample of approximately |$\sim$|20000 galaxies with reliable spectroscopic redshifts in the DEEP2 data (Davis et al. 2003). The study identifies galaxy mergers in selection criteria of |$d=10$|–50 kpc |$h^{-1}$| and a relative velocity difference, |$|\Delta V|\lt 500$| km s|$^{-1}$|⁠. The galaxy overdensity is estimated by the third-nearest-neighbor method. The measurements of the galaxy merger fraction in low-, medium-, and high-|$\delta$| bins suggest that the galaxy companion rate increases with galaxy overdensity, but after correcting for merger probability and timescales, wet galaxy mergers display weak environmental influence. In contrast, dry and mixed galaxy mergers are more effective in galaxy overdense regions. For the plots in figures 11 and 12, we adopt |$N_{\rm c}$| for the “All” type galaxy sample (table 1 of Lin et al. 2010), which is most similar to our |$f_{\rm merger}$| definition in the merger types.

A.2.13 Kampczyk et al. (2013)

Kampczyk et al. (2013) examine the relation between galaxy mergers and galaxy environments at |$z=0.2$|–1 using the zCOSMOS 10k-bright spectroscopic sample (Lilly et al. 2007, 2009). The study identifies 153 close kinematic pairs out of 3667 galaxies in selection criteria of |$d\lt 100$|⁠, 50, or 30 kpc |$h^{-1}$|⁠, |$|\Delta V|\lt 500$| km s|$^{-1}$|⁠, and |$\mu \ge 1/4$|⁠. The galaxy overdensity is estimated by counting the number of galaxies within radius-fixed apertures and source velocity differences |$|\Delta V|\lt 1000$| km s|$^{-1}$|⁠. By correcting for projection effects, the study reveals a three times higher pair fraction in the top density quartile compared to the lowest, indicating a higher galaxy merger rate in denser galaxy environments.

A.2.14 De Ravel et al. (2011)

De Ravel et al. (2011) analyze 10644 galaxies at |$z=0.2$|–1 from the zCOSMOS spectroscopic survey. The research identifies 263 close pairs with selection criteria of |$d\lt 100$|⁠, 50, 30, or 20 kpc |$h^{-1}$|⁠, |$|\Delta V|\lt 500$| km s|$^{-1}$|⁠, and |$\mu \ge 1/4$|⁠. The galaxy overdensity is estimated by counting the number of galaxies within radius-fixed apertures. The pair fraction and merger rate show an increase with local galaxy density, suggesting that dense galaxy environments favor major merger events. Although there are no |$f_{\rm merger}$|–|$\delta$| data available in the paper, based on the description of “...for an environment 10 times denser, one finds 3.0, 2.6, and 2.1 times more pairs for separations of |$r = 100$|⁠, 50, and 30 h|$^{-1}$| kpc respectively”, we plot |$R_{\rm merger}$| in figure 12.

A.2.15 Kocevski et al. (2011)

Kocevski et al. (2011) investigate morphologies of galaxies in three clusters and three groups of the Cl1604 galaxy supercluster at |$z\sim 0.9$| using the Hubble ACS images. The study classifies 131 galaxies in the core regions within the virial radii of the six overdense systems, |$R\lt R_{\rm vir}$|⁠, and 70 galaxies at the outer regions, |$R\gt R_{\rm vir}$|⁠. By combining visual inspections with results of the |$GM_{20}$| classification, the study finds that the fractions of galaxies with disturbed morphologies are typically higher at |$R\lt R_{\rm vir}$| than at |$R\gt R_{\rm vir}$|⁠, indicating that galaxy mergers are more prevalent in higher galaxy density regions.

A.2.16 Asano et al. (2020)

Asano et al. (2020) measure the galaxy merger fractions in four environments within and around the galaxy cluster CL1604-D at |$z=0.923$|⁠: the field, SW group, NE group, and cluster core. By selecting mergers/interacting galaxies in the same way as in Kocevski et al. (2011), the study finds that the galaxy merger fraction varies across the regions. The galaxy mergers are scarce in the field and the cluster core (⁠|$f_{\rm merger}\sim 8$|%). In contrast, merger/interacting galaxies are more prevalent in the galaxy group environments, the SW and NE groups (⁠|$f_{\rm merger}\sim 20$|% and 50%, respectively).

A.2.17 Paulino-Afonso et al. (2019)

Paulino-Afonso et al. (2019) examine galaxy structures for 490 galaxies in and around a galaxy superstructure at |$z\sim 0.84$| in the COSMOS field. With the Hubble ACS |$I_{814}$|-band images in COSMOS, the study measures the morphological parameters of |$CAS$| and |$GM_{20}$| and uses visual classification catalogs of Willett et al. (2017) to investigate the environment impacts of galaxy structures. The galaxy overdensity is estimated based on the 3D position of the spectroscopically confirmed galaxies. The study finds that the asymmetry parameter A and the fraction of irregular galaxies are nearly constant with the galaxy overdensity. However, increasing and decreasing trends are found if the sample is divided into star-forming galaxies (SFGs) and quiescent galaxies (QGs). The fraction of QGs (SFGs) with irregular morphologies increases (decreases) from the field/filament-like regions to the highest galaxy density environment.

A.2.18 Van Dokkum et al. (1999)

Van Dokkum et al. (1999) conduct a morphological analysis of galaxies in a luminous X-ray cluster, MS 1054-03 at |$z = 0.83$|⁠. Examining |$V_{606}$| and |$I_{814}$| images of the Hubble WFPC2 for 81 spectroscopically confirmed cluster members, the researchers identify 13 ongoing mergers. This means that the galaxy merger fraction is |$f_{\rm merger}=17$|% for the |$L\gtrsim L^{*}$| cluster population, which is higher than those for the field environments and lower-z rich galaxy clusters. These galaxy mergers are preferentially found in the outskirts of the MS 1054-03 galaxy cluster. In addition to MS 1054-03, they measure |$f_{\rm merger}$| for a lower-z galaxy cluster, CL 1358|$+$|62 at |$z=0.33$| (van Dokkum et al. 1998). The galaxy merger fraction in CL 1358|$+$|62 is |$f_{\rm merger}\sim 0.005$|⁠, much lower than the |$f_{\rm merger}$| value in MS 1054-03.

A.2.19 Omori et al. (2023)

Omori et al. (2023) classify 302148 galaxies at |$z\sim 0.01$|–0.35 in the Subaru HSC-SSP survey using the Zoobot deep learning model pre-trained on the Galaxy Zoo DeCALS data. The Zoobot model is fine-tuned with synthetic HSC images from the TNG simulation. The study identifies that at scales of 0.5–8 |$h^{-1}$| Mpc, merger galaxies prefer lower-density environments, while non-mergers favor higher-density environments. However, below these scales, the TNG simulation data show a reversal in the trend, with higher-density environments favoring galaxy mergers.

A.2.20 Sureshkumar et al. (2024)

Sureshkumar et al. (2024) determine whether galaxy mergers at |$z=0.1$|–0.15 occur more frequently in dense or less dense environments. Using data from the GAMA survey, the study classifies 23855 galaxies into mergers and non-mergers based on the convolutional neural network and morphological parameters of |$GM_{20}$|⁠. While no significant difference is found in the clustering strengths of mergers and non-mergers using the two-point correlation function, an anti-correlation is observed between the likelihood of a galaxy being a merger and the galaxy environment using the marked correlation function. The results suggest that galaxy mergers prefer underdense galaxy environments.

A.2.21 McIntosh et al. (2008)

McIntosh et al. (2008) analyze 5376 member galaxies in galaxy groups and clusters at |$z\le 0.12$|⁠. By identifying close pairs with |$\le$|30 kpc projected separations and asymmetric structures in the Sérsic-fit residuals of the SDSS images, the study isolates 38 mergers. Investigating the galaxy merger frequency as a function of dark matter halo mass, the researchers find that galaxy mergers are more likely to occur in large galaxy groups than in galaxy clusters.

A.2.22 Liu et al. (2009)

Liu et al. (2009) identify 18 major dry mergers out of 515 early-type brightest cluster galaxies (BCGs) at |$z=0.03$|–0.12. The galaxy major mergers are selected as galaxy pairs with projected separations of |$\lt $|30 kpc and with signatures of interaction in SDSS residual images. The study finds that the galaxy major merger fraction in the BCGs increases with cluster richness.

A.2.23 Alonso et al. (2012)

Alonso et al. (2012) examine 660 galaxy pairs in galaxy groups at |$z\lt 0.1$| from SDSS-DR7. The galaxy pairs are categorized into “Merging”, “Tidal”, and “Non-disturbed” pairs based on levels of galaxy interactions. The galaxy pairs are concentrated towards group centers, with the disturbed pairs often containing the brightest galaxies in the galaxy groups. To analyze the influence of the group global environment, the densities of the galaxy pairs are quantified using the distance to the fifth-nearest neighbor. Galaxy pairs have an excess of low-density group global environments compared to the control sample. This trend is more pronounced in the merging pairs than in the tidal pairs and non-disturbed pairs.

A.2.24 Perez et al. (2009)

Perez et al. (2009) use galaxies at |$z=0.01$|–0.1 in the SDSS DR4 catalog to investigate the relation between galaxy mergers and galaxy environments. The study defines close pairs with a projected separation of |$\lt $|100 kpc |$h^{-1}$| and a relative velocity difference of |$\Delta V\lt 350$| km s|$^{-1}$| as galaxy pairs. Based on visual inspections for the sample of galaxy pairs, the researchers also identify merging systems with morphological disturbances and strong signals of interactions. The galaxy overdensity is calculated by using the projected distance to the fifth-nearest neighbor of galaxies. The study analyzes the cumulative fractions of galaxy mergers and isolated galaxies in different galaxy overdensities. The cumulative fractions of galaxies suggest that the numbers of close galaxy pairs and merging systems increase in intermediate-density environments.

A.2.25 Ellison et al. (2010)

Ellison et al. (2010) examine galaxy mergers at |$z\lt 0.1$| in various galaxy environments using 5784 close galaxy pairs from the SDSS DR4 data. The analysis of the galaxy merger fraction indicates that galaxy mergers occur across all the galaxy densities. The low galaxy density areas favor triggered star formation, while high galaxy density regions show galaxy mergers without star formation. Measurements of the asymmetry index reveal a high galaxy merger fraction in cluster centers.

A.2.26 Darg et al. (2010)

Darg et al. (2010) compare the galaxy local densities of galaxy pairs and control galaxies at |$z=0.005$|–0.1 from SDSS DR6 to understand galaxy merger properties and galaxy environmental effects. The galaxy pairs are identified using the Galaxy Zoo classification catalog. The study finds no significant difference in the environmental distribution between the galaxy mergers and the control galaxies. Elliptical galaxies in galaxy mergers tend to occupy slightly denser galaxy environments than the controls, while spiral galaxies show a minimal difference.

A.2.27 Moss (2006)

Moss (2006) measures the galaxy merger fraction for 680 galaxies in eight galaxy clusters at |$z\sim 0$|⁠. The galaxies are likely to reside in intermediate-density infall regions according to the velocity distribution of the galaxies. Visual inspections are conducted to classify the galaxies based on the presence of disturbance signatures. The analysis shows that |$\sim$|50%–70% of the galaxies in the infall population are interacting or merging systems. The galaxy merger fraction is |$\sim$|4–5 times higher than |$\sim$|13% for the field galaxies located outside the virial radius of the galaxy clusters.

Footnotes

References

Author notes