Predicting drug–target binding affinity through molecule representation block based on multi-head attention and skip connection

Abstract

Introduction

Proteins are involved in many cellular processes, and lots of diseases are caused by abnormal protein levels [1]. Thus, a large number of drugs have been developed to target proteins. According to statistics, over 70% of 53 Food and Drug Administration (FDA)-approved drugs in 2020 act on proteins for drug effects [2]. In general, identifying novel drug–target interactions (DTIs) and elucidating their biological mechanisms play a vital role in drug discovery and development [3–6]. Although experimental methods are reliable and have been widely applied in drug discovery and repurposing [7–9], it is extremely time-consuming and laborious to screen hit compounds from a large-scale chemical space solely by experimental methods. Computational methods as a supplement to experimental methods have aroused a huge rise in the field of biomedicine [10–15]. Some researchers employed knowledge graph, random walk with restart and Bayesian machine learning to predict DTIs, and they regard DTI prediction as a binary classification problem [16–18]. Different from DTI prediction, drug–target binding affinity prediction can further reveal the strength of DTIs and could be regarded as a regression problem [19–21]. Here, we concentrate on predicting binding affinities of drug–target pairs.

The following methods are two early representatives of utilizing traditional machine learning models to predict drug–target binding affinity. Pahikkala et al. [22] predicted binding affinities of drug–target pairs based on Kronecker regularized least-squares (KronRLS). Following standards of kernel learning methods, they first defined an objective function where the kernel function indicated the similarity of two drug–target pairs in the Hilbert space. Then, they regarded the problem of learning a prediction function as finding a minimizer of the objective function with the aim of binding affinities prediction. Afterward, He et al. [23] developed a supervised machine learning method named SimBoost to make predictions for binding affinities between drugs and targets. Depending on drug–drug similarity, target–target similarity and known binding affinities, they calculated four properties of each drug (target) as type 1 features by statistical analysis. Through utilizing the k-nearest neighbor method, they extracted six properties as type 2 features from the drug–drug (target–target) similarity network. Relying on the DTI network, they extracted five properties as type 3 features via matrix factorization. For each drug (target), they integrated type 1 and type 2 features as its final features. To obtain final features of drug–target pairs, they integrated features of drugs and targets, as well as type 3 features of drug–target pairs. After training gradient boosting machine based on final features of drug–target pairs, SimBoost can be applied to predict binding affinities of unknown drug–target pairs.

Recently, due to the breakthroughs in deep learning and the huge increase in computing power, deep learning-based models have gradually been applied to make predictions for drug–target binding affinity [24–32], and some of them have achieved favorable performance [29–32]. Adopting a widespread deep learning framework, convolutional neural network (CNN), Öztürk et al. [24] devised a novel model called DeepDTA to predict drug–target binding affinities. Firstly, they converted FASTA sequences for proteins and Simplified Molecular Input Line Entry System (SMILES) sequences for drugs into the embedding space via the embedding layer. Then, two CNN modules were utilized to learn representations from embedded protein FASTA and drug SMILES sequences, respectively. Each CNN module in DeepDTA is composed of a stack of 1D-convolutional layers with increasing number of filters and one max-pooling layer. Through feeding concatenated representations for protein FASTA and drug SMILES sequences into fully connected layers, drug–target binding affinities could be finally predicted. Depending on a similar deep learning framework to DeepDTA, Öztürk et al. [25] designed another CNN-based model named WideDTA to predict drug–target binding affinity. Originally, they introduced four types of data, including protein FASTA sequences, protein domains and motifs, drug SMILES sequences and drug maximum common substructures, and exploited the Keras-based embedding layer to respectively denote these data with 128-dimensional vectors. Then, the CNN module with two 1D-convolutional layers and a max pooling layer was built and employed to respectively extract features for above-mentioned four types of data. Features extracted from these CNN modules were concatenated and delivered into three fully connected layers to predict drug–target binding affinities. Combining effective feature embedding with powerful deep learning methods, Wan et al. [26] proposed a general and scalable computational framework named DeepCPI. They first exploited a word-embedding technique namely Word2vec, which is popularly applied in different natural language processing (NLP) tasks [33–35] to learn low-dimensional representations of protein FASTA sequences. Secondly, the latent semantic indexing technique [36], one representative method in the field of NLP, was utilized to obtain low-dimensional representations of drug structures. Then, they built a multimodal deep neural network (DNN) by adding the multimodal variant to a Vanilla DNN. Eventually, extracted low-dimensional feature representations of proteins and drugs were fed into the multimodal DNN to predict drug–target binding affinities. Integrating different kinds of DNNs, Lin et al. [27] released the model of Deep representation learning framework for Graphs and Sequences (DeepGS), which was an end-to-end deep learning model and could extract local chemical contexts from protein FASTA and drug SMILES sequences as well as drug molecule structures. Specifically, motivated by Word2Vec [37], they developed methods of Prot2Vec and Smi2Vec to encode all symbols in protein FASTA and drug SMILES sequences, respectively. Each sequence of proteins and drugs could be transformed into a matrix, where a row represented a symbol of sequences. Afterward, they extracted features from protein matrices by employing a CNN and captured features from drug matrices through utilizing a bi-directional gated recurrent unit (BiGRU). Simultaneously, they used a graph attention network (GAT)-based module to gain topology information of drug molecule structures. As a result, they got latent representations for proteins and two types of latent representations for drugs. Last, the concatenation of three kinds of latent representations was input to three consecutive fully connected layers, and a predicted binding affinity value was output. Taking the sequence and structure information of proteins and drugs into consideration, Pu et al. [28] put forward a two-stage DNN ensemble model named DeepFusionDTA to detect binding affinities of drug–target pairs. Above all, through integrating a Dilated-CNN module with three stacked dilated convolution layers [38], and a bi-directional long short-term memory network (BiLSTM), the authors designed a dual-channel sequence analysis module (SeqM) to extract sequence features. Meanwhile, they specially employed a single Dilated-CNN module (StruM) to analyze structure information. Secondly, for proteins (drugs), the authors respectively applied SeqM and StruM to obtain features of their FASTA (SMILES) sequences and secondary structures (fingerprints), and merged the two types of features into protein (drug) feature maps. After concatenating protein and drug feature maps as representations of drug–target pairs, they constructed fully connected layers to decrease the representation dimensionality for each drug–target pair. Conclusively, relying on decreased-dimensionality representations of drug–target pairs, they utilized bagging strategy to integrate outputs of multiple parallel lightGBM models as prediction values for drug–target binding affinity.

As known, the attention mechanism has achieved a huge success in the field of NLP and computer vision (CV) [39]. Therefore, some researchers attempted exploring the potential of attention mechanism in predicting drug–target binding affinity [29–32]. Shin et al. [29] proposed molecule transformer DTI (MT-DTI) on foundation of the attention mechanism to predict drug–target binding affinity. They encoded protein FASTA sequences into vectors via multi-layer CNNs and generated vectors of drug SMILES sequences through utilizing multi-layered bidirectional Transformer encoders where each Transformer encoder is made up of a self-attention layer and a feed-forward layer. Next, protein and drug vectors were concatenated and fed into the multi-layered feed-forward network to obtain binding affinities of drug–target pairs. Relying on the two-sided attention mechanism, Abbasi et al. [30] proposed a method of DeepCDA to predict binding affinities of drug–target pairs. At first, protein FASTA and drug SMILES sequences were, respectively, processed by the encoding layer to get vectors for proteins and drugs. Hereafter, they exploited the CNN block consisting of three 1D-convolutional layers, two dropout layers and two max-pooling layers to extract features for proteins and drugs. To capture more information from the sequence perspective, they constructed the long short-term memory (LSTM) block with multiple LSTM units to further extract features for proteins and drugs. The feature representations of proteins and drugs were merged in the two-sided attention mechanism layer, and the merged representations were fed into the fully connected layer to predict binding affinities. Combining the attention mechanism, the graph neural network (GNN) and the CNN, Nguyen et al. [31] released a new deep learning-based method named GraphDTA to make drug–target binding affinity prediction. For proteins, they first utilized an embedding layer to encode protein FASTA sequences. Then, a CNN module with three 1D-convolutional layers and a max pooling layer was adopted to get representation vectors of encoded protein FASTA sequences. For drugs, they converted SMILES sequence of each drug to its corresponding molecule graph by using the open-source chemical informatics software RDKit [40], so that GraphDTA could directly capture the interaction information between atoms. Hereafter, they respectively employed four GNN variants, including a graph convolutional network (GCN), a GAT, a graph isomorphism network and a combined GAT–GCN architecture to obtain representation vectors of drug [41–43]. After fusing protein representations with four graph-based drug representations, they utilized two fully connected layers in series to obtain binding affinities of drug–target pairs. Later, Zeng et al. [32] proposed a multiple attention blocks-based model (MATT_DTI) to make predictions for drug–target binding affinity. MATT_DTI includes three parts: protein representation learning, drug representation learning and DTI learning. In protein representation learning, the embedding layer was applied to encode protein FASTA sequences and the CNN module, including three CNN layers, was utilized to extract features for encoded proteins. In drug representation learning, they designed a relation-aware self-attention module to further encode drug SMILES sequences processed by an embedding layer. The same CNN module as in protein representation learning was employed to extract features for encoded drugs. In DTI learning, they exploited the multi-head attention to acquire interaction features of drug and protein representations. The interaction features were finally fed into three factorization machine supported neural network layers to make predictions for drug–target binding affinity.

Generally, although some computational models of predicting drug–target binding affinity have appeared, and the part of them have achieved good prediction performance, there is no denying that existing computational models still have much room for improvement in prediction accuracy, robustness and generalization ability. Most deep learning models for drug–target binding affinity prediction lack interpretability analysis, which reduces their credibility in practical applications to some extent. What’s more, many studies do not offer application examples in real-life situations. For the sake of solving above problems, we developed a novel deep learning model called Molecule Representation Block-based Drug–Target binding Affinity prediction (MRBDTA). MRBDTA consists of three parts including embedding and positional encoding, molecule representation block (Trans block) and DTI learning. First of all, the original protein FASTA and drug SMILES sequences are encoded during embedding and positional encoding process. Then, two Trans blocks are employed to extract features from encoded proteins and drugs, respectively. Finally, the interaction learning module is utilized to integrate features of proteins and drugs extracted by the Trans block and predict binding affinities between proteins and drugs. To evaluate the performance of our deep learning model, we tested MRBDTA on two benchmark datasets under evaluation metrics of mean squared error (MSE), concordance index (CI) and regression toward the mean (⁠$r_m^2$⁠) and implemented analysis experiments, interpretability analysis and case studies. The test results manifest that MRBDTA achieves better prediction performance and stronger stability compared with 11 state-of-the-art computational models. Besides, we conducted analysis experiments by replacing Trans block with single Trans encoder and removing skip connection in Trans block and verified the effectiveness of Trans block and skip connection in improving the prediction accuracy, stability and reliability of MRBDTA. Then, for 1N6R (PDB ID) formed from reacting Ras-related protein Rab-5A with GppNHp, and 3AQV (PDB ID) obtained from the chemical reaction after mixing AMP-activated protein kinase and Dorsomorphin, we carried out interpretability analysis based on multi-head attention mechanism to illustrate that MRBDTA can correctly capture part of interaction sites between proteins and drugs. In case studies, we firstly applied MRBDTA to predict binding affinities (KIBA score and K_d in nM) between 3137 FDA-approved drugs and severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) replication-related proteins. In top 50 drugs with better predicted affinities, compared with MT-DTI and MATT_DTI, MRBDTA can predict more antiviral drugs and their rankings are higher. Secondly, by comparing experimentally measured binding affinities between 3C-like proteinase and 185 drugs with those predicted by MRBDTA, we observed that most binding affinities predicted by MRBDTA are satisfactory. All in all, MRBDTA is a promising computational tool in predicting binding affinities between drugs and targets and has great potential to assist in drug discovery and repurposing in the future.

Results

Performance evaluation

Based on the Davis and KIBA datasets including known binding affinity values between proteins and drugs, we evaluated the performance of MRBDTA. Using the same method as literature [32], we divided samples in the Davis and KIBA datasets into training set and test set in accordance with the ratio of 5 to 1, respectively. Here, the training set was applied to 5-fold cross-validation experiment to search for optimal parameter settings of MRBDTA, such as embedding size, the number of Trans block and heads in multi-head attention, hidden size, the number of epochs, dropout rate and learning rate. Table 1 gives the parameter settings for MRBDTA in our experiments.

In our study, MSE, CI and $r_m^2$ are utilized to evaluate the performance of various computational models. In regression prediction tasks, MSE is a common metric of measuring the deviation between true values and predicted values, and the smaller MSE indicates the higher prediction accuracy of a model. CI is proposed in the literature [44] and can be applied to measure the probability of the concordance between truth values and predicted values. The larger CI manifests the better prediction performance of a model. It is well known that $r_m^2$ has been extensively applied in quantitative structure–activity relationship (QASR) models and can reflect the external predictive potential of a model [45]. An acceptable model should have an $r_m^2$ value greater than 0.5, and the larger $r_m^2$ reflects the better generalization performance of a model.

As aforementioned, the training set was equally divided into five subsets in the 5-fold cross-validation experiment. Then, one subset was left out in turn and we trained MRBDTA based on other four subsets with the above-obtained optimal parameters. Next, we utilized the trained MRBDTA to predict binding affinities in the test set. Based on the predicted and true affinities of drug–target pairs in the test set, we could calculate the value of MSE, CI and $r_m^2$⁠, respectively. The above experiment process was repeated five times because each of five subsets in the training set was left out in turn. Table 2 shows final results on the test set of two benchmark datasets. Finally, we were able to get the average value and standard deviation (SD) of MSE, CI and $r_m^2$⁠. As displayed in Table 3, the average MSE, CI and $r_m^2$ of MRBDTA on the test set in the Davis dataset are 0.216, 0.901 and 0.716, respectively. Besides, the SD of MSE, CI and $r_m^2$ is 0.006, 0.004 and 0.008, respectively. Analogously, in Table 4, we have given the average MSE (0.146), CI (0.892) and $r_m^2$ (0.778) of MRBDTA on the test set in the KIBA dataset. Besides, the SD of MSE, CI and $r_m^2$ is 0.001, 0.002 and 0.005, respectively. Compared with 11 state-of-the-art computational models (KronRLS [22], SimBoost [23], DeepDTA [24], MT-DTI [29], WideDTA [25], DeepCPI [26], DeepCDA [30], GraphDTA [31], DeepGS [27], MATT_DTI [32] and DeepFusionDTA [28]), MRBDTA achieves almost the best performance in both Davis and KIBA datasets (see Tables 3 and 4). Concretely, in the Davis dataset, the MSE, CI and $r_m^2$ of MRBDTA are the best. In the KIBA dataset, the CI and $r_m^2$ of MRBDTA also gain the best results, and the MSE of MRBDTA (0.146) is second only to the MSE of GraphDTA (0.139) [31]. In addition, the low SD of MSE, CI and $r_m^2$ in both Davis and KIBA datasets reflect the strong stability of MRBDTA (See Table 3 and Table 4).

Analysis experiments

The main contributions of MRBDTA lie in developing Trans block to extract molecule features through improving the encoder of transformer and introducing skip connection at encoder level in Trans block. Therefore, in order to verify the effectiveness of Trans block and skip connection, respectively, we implemented analysis experiments by replacing Trans block with single Trans encoder (MRBDTA_STE) and removing skip connection in Trans block (MRBDTA_RSC) based on Davis and KIBA datasets under evaluation metrics of MSE, CI and $r_m^2$⁠. Besides, through comparing MRBDTA_STE and MRBDTA_RSC with MRBDTA, respectively, we have observed that MRBDTA achieved the best performance among the three models (See Table 5, Supplementary Tables 1 and 2 available online at http://bib.oxfordjournals.org/). Further, the results of analysis experiments manifest that Trans block and skip connection can indeed improve the prediction accuracy, stability and reliability of MRBDTA.

The interpretability based on multi-head attention mechanism

Another important contribution of MRBDTA lies in that multi-head attention mechanism could enhance the ability to capture interaction sites between proteins and drugs and hence benefit the biological interpretability of MRBDTA. Therefore, depending on multi-head attention mechanism, we calculated the attention weight value for each amino acid residue of a protein, and the attention weight value means contribution of the amino acid residue to DTIs associated with this protein. According to the attention weight value, we ranked all amino acid residues in a protein. In addition, potential and captured interaction sites were introduced to analyze the reliability of MRBDTA. Usually, in the protein data bank (PDB), potential interaction sites between a drug and a protein are defined by a distance in the range of <5.0 Angstrom between a drug and all amino acid residues of a protein [30]. Additionally, for a drug–protein pair, we selected the amino acid residues with higher attention weight values as captured interaction sites by MRBDTA. Here, the number of selected amino acid residues is equal to the number of potential interaction sites in the drug–protein pair. We also visualized potential and captured interaction sites between proteins and drugs based on the 3D View tool of PDB.

Figure 1 shows the potential and captured interaction sites for two complexes, namely 1N6R and 3AQV. According to PDB, Ras-related protein Rab-5A reacts with GppNHp to form 1N6R, and 3AQV is obtained from the chemical reaction after mixing AMP-activated protein kinase and Dorsomorphin. Concretely, the potential and captured interaction sites marked with red in 1N6R are displayed in Figure 1A and B, respectively. There are 38 amino acid residues viewed as potential interaction sites in 1N6R on the basis of PDB (See Supplementary File 1 available online at http://bib.oxfordjournals.org/). According to the attention weight value, we obtained top 38 amino acid residues with higher attention weight values in 1N6R as captured interaction sites by MRBDTA (See Supplementary File 1 available online at http://bib.oxfordjournals.org/). Here, we regarded these captured interaction sites, which were included in potential interaction sites or near potential interaction sites (the absolute value of the PDB residue number difference between a captured interaction site and a potential interaction site was less than or equal to 10) as correctly captured interaction sites by MRBDTA. As illustrated in Figure 1B, the 21 circled amino acid residues were correctly captured interaction sites in 1N6R (See Supplementary File 1 available online at http://bib.oxfordjournals.org/). Moreover, the potential and captured interaction sites marked with red in 3AQV are shown in Figure 1C and D, respectively. Relying on PDB, there are 22 amino acid residues seen as potential interaction sites in 3AQV (See Supplementary File 1 available online at http://bib.oxfordjournals.org/). Here, we achieved top 22 amino acid residues with higher attention weight values in 3AQV as captured interaction sites by MRBDTA (See Supplementary File 1 available online at http://bib.oxfordjournals.org/). The nine circled amino acid residues were viewed as correctly captured interaction sites in 3AQV (see Figure 1D and Supplementary File 1 available online at http://bib.oxfordjournals.org/).

Figure 1

The visualization of interaction sites between proteins and drugs. The potential and captured interaction sites marked with red in 1N6R are shown in (A) and (B), respectively. As depicted in (A), there are 38 potential interaction sites in 1N6R. In (B), among the 38 captured interaction sites by MRBDTA, the circled 21 captured interaction sites were regarded as correctly captured interaction sites in 1N6R. The potential and captured interaction sites marked with red in 3AQV are exhibited in (C) and (D), respectively. As displayed in (C), there are 22 potential interaction sites in 3AQV. In (D), the circled 9 of the 22 captured interaction sites were considered as correctly captured interaction sites in 3AQV.

Open in new tabDownload slide

All in all, the above two examples indicate that our model can correctly capture part of interaction sites between proteins and drugs, which partly explains the reliable performance of MRBDTA.

Case studies

In recent years, the infection of SARS-CoV-2 has spread rapidly around the world, posing a huge threat to the human life. Due to a severe shortage of therapeutic drugs for patients with SARS-CoV-2, and long cycle for developing a new drug, drug repurposing, a novel strategy for drug discovery, has been utilized to find effective drugs for patients with SARS-CoV-2 from FDA-approved drugs [46, 47]. In this case study, we firstly applied trained MRBDTA to predict binding affinities between 3137 FDA-approved drugs and SARS-CoV-2 replication-related proteins. Secondly, we compared experimentally measured binding affinities between 3C-like proteinase and 185 drugs with those predicted by trained MRBDTA. The purpose of this case study is to provide an application example of MRBDTA in real-life situations and verify reliable prediction performance of MRBDTA in drug design for SARS-CoV-2. Besides, we expect the predicted results of MRBDTA are capable of providing scientists with some ideas in developing novel drugs against SARS-CoV-2 and assist in treating patients with SARS-CoV-2.

The FASTA sequences of SARS-CoV-2 replication-related proteins containing 3C-like proteinase (accession YP_009725301.1), RNA-dependent RNA polymerase (accession YP_009725307.1), helicase (accession YP_009725308.1), 3′-to-5′ exonuclease (accession YP_009725309.1), endoRNAse (accession YP_009725310.1) and 2′-O-ribose methyltransferase (accession YP_009725311.1) have been acquired from National Center for Biotechnology Information (NCBI) database. Relying on literature [32], we obtained SMILES sequences of 3137 FDA-approved drugs. The binding affinities predicted by MRBDTA between 3137 FDA-approved drugs and 6 SARS-CoV-2 replication-related proteins based on KIBA score and K_d in nM can be found in Supplementary File 2 and Supplementary File 3 available online at http://bib.oxfordjournals.org/, respectively. For each SARS-CoV-2 replication-related protein, we first ranked 3137 FDA-approved drugs according to predicted binding affinities. Then, through comparing MRBDTA with MT-DTI [29] and MATT_DTI [32], we observed that among top 50 drugs with better predicted affinities, MRBDTA can predict more antiviral drugs and their rankings are higher (See Tables 6 and 7). Specifically, for six SARS-CoV-2 replication-related proteins, there are respectively 22 and 13 antiviral drugs in top 50 drugs predicted by MRBDTA under two different types of binding affinities, while there are merely 6 and 8 antiviral drugs in top 50 drugs predicted by MT-DTI based on the binding affinity of K_d in nM and MATT_DTI based on the binding affinity of KIBA score, respectively. It should be pointed out that since the codes of MT-DTI and MATT_DTI are unavailable, we only could adopt the prediction results of MT-DTI and MATT_DTI in their studies, respectively. What’s more, there are 14 and 12 antiviral drugs in top 30 drugs predicted by MRBDTA, while there are no antiviral drugs in top 30 drugs predicted MT-DTI and only 3 antiviral drugs are included in top 30 drugs predicted by MATT_DTI. Moreover, the best ranking of antiviral drug predicted by MRBDTA is 2nd, while the best ranking of antiviral drug predicted by MT-DTI and MATT_DTI are 32nd and 21st, respectively. To sum up, these results indicate that MRBDTA shows better performance than MT-DTI and MATT_DTI in this important application.

In order to further verify reliable prediction performance of MRBDTA in drug design for SARS-CoV-2, we first obtained the sequence of 3C-like proteinase, structure files of 185 drugs and experimentally measured binding affinities between 3C-like proteinase and 185 drugs from literature [48, 49]. Then, depending on structure files of the 185 drugs, we utilized the software RDKit [40] to get SMILES sequences of the 185 drugs. Since the type of true binding affinities for drug–target pairs in the Davis dataset is the same as experimentally measured binding affinities of these 185 drug–target pairs, we trained MRBDTA on the basis of the Davis dataset. It is worth nothing that 3C-like proteinase is not included in the training data. The binding affinities predicted by trained MRBDTA between 3C-like proteinase and 185 drugs can be found in Supplementary File 4 available online at http://bib.oxfordjournals.org/. Relying on predicted and experimentally measured binding affinities of the 185 drug–target pairs, we obtained the MSE (1.451) of MRBDTA. Further, the low MSE of MRBDTA on the 185 drug–target pairs manifests that most binding affinities predicted by MRBDTA are satisfactory.

Discussion

The identification of drug–target binding affinities is a crucial step in drug discovery. Lately, computational methods have gradually emerged to predict binding affinities, and some pharmaceutical companies have benefited from these methods to some extent [50–52]. An excellent computational model of predicting drug–target binding affinity can not only shorten the cycle of drug discovery but also reduce the cost of drug discovery. In this research, we put forward a deep learning model named MRBDTA to make predictions for drug–target binding affinity. The implementation process can be divided into three parts. In embedding and positional encoding, we convert the original protein FASTA and drug SMILES sequences into the embedding layer space and add position information to the converted data. Then, we adopt two Trans blocks to extract features for converted proteins and drugs, respectively. In DTI learning, we integrate features of proteins and drugs extracted by the Trans block and predict binding affinities between proteins and drugs. To evaluate the performance of our deep learning method, we tested MRBDTA on two benchmark datasets and carried out analysis experiments, interpretability analysis and case studies. In the Davis and KIBA datasets, the MSE, CI and $r_m^2$ of MRBDTA on test data almost exceed all the 11 state-of-the-art computational models. The low SD of MSE, CI and $r_m^2$ in both Davis and KIBA datasets reveals the strong stability of MRBDTA. Besides, we conducted analysis experiments via replacing Trans block with single Trans encoder and removing skip connection in Trans block and confirmed that Trans block and skip connection could effectively improve the prediction accuracy, stability and reliability of MRBDTA. Then, for 1N6R and 3AQV, depending on multi-head attention mechanism, we performed the interpretability analysis to illustrate that the ability of correctly capturing part of interaction sites between a protein and a drug is one important reason for the excellent performance of MRBDTA. In case studies, we firstly utilized MRBDTA to predict binding affinities between FDA-approved drugs and SARS-CoV-2 replication-related proteins. As a result, in top 50 drugs with better-predicted affinities, compared with MT-DTI and MATT_DTI, our model can predict more antiviral drugs and their rankings are higher. Secondly, the trained MRBDTA was applied to predict binding affinities between 3C-like proteinase and 185 drugs. We found that most binding affinities predicted by MRBDTA were close to experimentally measured binding affinities of the 185 drug–target pairs. This further comprehensively manifests the excellent performance of MRBDTA.

MRBDTA shows better performance over the state-of-the-art computational models because of the following three factors. First, through optimizing the encoder of transformer, we developed a novel module called Trans block to extract molecule features. Trans block can fully take advantage of multi-head attention mechanism to capture more detailed information about molecule sequences from a wider perspective. Second, in Trans block, we introduced skip connection at encoder level to avoid the loss of global molecule features when acquiring the local molecule features, so as to ensure comprehensive capture of molecule features. Third, MRBDTA has the ability to correctly capture part of interaction sites between proteins and drugs, which reflects the interpretability of MRBDTA.

However, MRBDTA still has much room for improvement. Firstly, because MRBDTA is a large model with a lot of parameters, it may suffer from slow convergence when applied to larger datasets. In the future, we will further optimize MRBDTA and reduce the number of parameters in MRBDTA as much as possible when ensuring the performance of MRBDTA. Secondly, all proteins used in this study are less than 1500 amino acids in length. Therefore, when MRBDTA is applied to a protein with the length greater than 1500 amino acids, its performance is unknown. Over time, with the increase of trained data including long-sequence proteins, the generalization performance of MRBDTA will be further promoted. Last but not least, we clarify the biological interpretation of MRBDTA from the perspective of the protein sequence, but a protein and a drug react in three-dimensional space. As more and more protein tertiary structures are reported, and excellent protein structure prediction models such as AlphaFold2 [53], RoseTTAFold [54], trRosetta [55] and MMpred [56] gradually emerge, available protein tertiary structure data will further increase. In future research, we will work on introducing protein tertiary structure data to further improve the interpretability of MRBDTA.

The future direction of drug–target binding affinity prediction is mainly reflected in three aspects. First, drug and protein tertiary structure data will be widely utilized in predicting drug–target binding affinity. Second, researchers will introduce more different deep learning models to predict drug–target binding affinity, and some deep learning models may achieve excellent performance in this field. Third, the biological interpretability of computational models for drug–target binding affinity prediction will be further enhanced.

Materials and methods

Benchmark datasets

Molecule Representation Block-based Drug-Target binding Affinity prediction

As demonstrated in Figure 2, based on the embedding layer, the positional encoding, the encoder module of transformer [59], the skip connection and the feed-forward layer, we proposed a novel deep learning model MRBDTA to predict drug–target binding affinities. MRBDTA is composed of three parts: embedding and positional encoding, Trans block and DTI learning. In detail, the original protein FASTA and drug SMILES sequences are encoded during embedding and positional encoding process. Next, two Trans blocks are exploited to extract features from encoded proteins and drugs, respectively. Eventually, the interaction learning module is employed to integrate features of proteins and drugs extracted by Trans block and predict binding affinities between proteins and drugs.

Figure 2

Illustration of MRBDTA proposed in this study. The architecture of MRBDTA is shown in (A). Especially, the architecture of MRBDTA consists of three parts: embedding and positional encoding, Trans block and DTI learning. (B) The implementation steps of scaled dot-product attention with mask operation. The principle of multi-head attention mechanism is given in (C). Here, multi-head attention is composed of four attention layers running in parallel. (D) and (E) describe the details of Trans encoder and L-Trans encoder in MRBDTA, respectively.

Open in new tabDownload slide

Embedding and positional encoding

Trans block

The internal details of Trans block are listed in Figure 2A. Concretely, Trans block includes one L-Trans encoder, two parallel Trans encoders, the skip connection drawn as a red line and the concat operation. The input of Trans block is the output

Figure 2D and E describes the details of Trans encoder and L-Trans encoder, respectively. Trans encoder in our model is the same as the encoder in transformer. Specifically, Trans encoder contains two sub-layers (a multi-head attention layer and a position-wise feed-forward layer), the residual connection namely Add operation and the layer normalization called Norm operation. In the above paragraph, we have elaborated on the multi-head attention layer. The position-wise feed-forward layer is made up of two linear transformations with a ReLU activation in between [60]. What’s more, we applied a residual connection around each of the two sub-layers, followed by layer normalization [61–63]. Unlike Trans encoder, L-Trans encoder, which could extremely enhance the robustness of our deep learning model, was constructed through adding a linear layer at the beginning of Trans encoder.

In summary, relying on Trans encoder and L-Trans encoder, we introduced the skip connection and the concat operation to develop Trans block. It should be noted that the skip connection is created through adding the input of the L-Trans encoder to the output of the L-Trans encoder at encoder level. As the most important component of MRBDTA, Trans block can extract effective features from encoded proteins and drugs, respectively. Besides, in DTI learning, the extracted features will be utilized to make predictions for drug–target binding affinity.

DTI learning

As depicted in Figure 2A, the interaction learning module contains the concat operation, two feed-forward layers and a linear layer. The feed-forward layer is orderly made up of a linear layer, a layer normalization namely Norm operation, a dropout layer and a ReLU activation [64]. Here, we first summed columns of $X^{PA}\in {\mathbb{R}}^{l\times 2e}$ and $X^{CA}\in {\mathbb{R}}^{z\times 2u}$⁠, which are outputs of two Trans blocks (one is for a protein P, the other is for a drug C) to get $X^{PSA}\in {\mathbb{R}}^{1\times 2e}$ and $X^{CSA}\in {\mathbb{R}}^{1\times 2u}$⁠, respectively. Then, $X^A\in {\mathbb{R}}^{1\times 4e}$ obtained from concatenating $X^{PSA}\in {\mathbb{R}}^{1\times 2e}$ and $X^{CSA}\in {\mathbb{R}}^{1\times 2u}$ was orderly fed into two feed-forward layers and a linear layer to achieve the output $Y^A$ of the interaction learning module. $Y^A$ is regarded as the predicted binding affinity value between a protein P and a drug C. What’s more, the fusion for features of a protein P and a drug C extracted by the Trans block is realized in the interaction learning module. From the perspective of molecular biology, the interaction learning module can be regarded as the simulation for the chemical reaction of a protein P and a drug C.

Funding

National Natural Science Foundation of China (under grant no. 61972399 to X.C.).

Data and code availability

The supporting data for this study and the implementation codes of MRBDTA are available online at https://github.com/LiZhang30/MRBDTA.

Author Biographies

Li Zhang, is a PhD student of School of Information and Control Engineering, China University of Mining and Technology. His research interests include bioinformatics, drug discovery, neural networks and deep learning.

Chun-Chun Wang, is a PhD student of School of Information and Control Engineering, China University of Mining and Technology. His research interests include bioinformatics, complex network algorithm and machine learning.

Xing Chen, PhD, is a professor of China University of Mining and Technology. He is the associate dean of Artificial Intelligence Research Institute, China University of Mining and Technology. He is also the founding director of Institute of Bioinformatics, China University of Mining and Technology and Big Data Research Center, China University of Mining and Technology. His research interests include complex disease-related non-coding RNA biomarker prediction, computational models for drug discovery and early detection of human complex disease based on big data and artificial intelligence algorithms.

References