Fast and accurate protein intrinsic disorder prediction by using a pretrained language model

Abstract

Determining intrinsically disordered regions of proteins is essential for elucidating protein biological functions and the mechanisms of their associated diseases. As the gap between the number of experimentally determined protein structures and the number of protein sequences continues to grow exponentially, there is a need for developing an accurate and computationally efficient disorder predictor. However, current single-sequence-based methods are of low accuracy, while evolutionary profile-based methods are computationally intensive. Here, we proposed a fast and accurate protein disorder predictor LMDisorder that employed embedding generated by unsupervised pretrained language models as features. We showed that LMDisorder performs best in all single-sequence-based methods and is comparable or better than another language-model-based technique in four independent test sets, respectively. Furthermore, LMDisorder showed equivalent or even better performance than the state-of-the-art profile-based technique SPOT-Disorder2. In addition, the high computation efficiency of LMDisorder enabled proteome-scale analysis of human, showing that proteins with high predicted disorder content were associated with specific biological functions. The datasets, the source codes, and the trained model are available at https://github.com/biomed-AI/LMDisorder.

Introduction

Proteins were thought to have a stable tertiary structure when performing their functions. However, Romero et al. [1] suggested that about 15 000 proteins or long protein regions in SwissProt [2] do not have a well-defined 3D structure. These intrinsically disordered proteins or protein regions (IDPs/IDRs) enriched our understanding of the sequence-structure–function relation of proteins. For example, lacking a rigid structure, IDPs/IDRs can transform between a group of transient and mutually transforming structural states [3]. Some functions directly require structural flexibility such as linkers between structured domains whereas other IDPs can undergo disorder-to-order transitions when interacting with specific bioactive molecules and serve as specific chaperones for certain proteins. These binding regions or molecular recognition features were found in modular proteins participating in signaling and regulation [4, 5]. The plasticity and flexibility of these functional modules, allow IDPs/IDRs to adopt varying conformations to interact with different partners in signal transmission, assembly and regulation, resulting in their involvement in many human diseases, including cardiovascular disease, cancer, various genetic diseases and neurodegenerative diseases [6]. Multi-structure states of IDPs [3, 7] may be evolutionarily advantageous, as suggested by higher degree of intrinsic disorder in natural proteins than in random sequences [8]. Thus, identifying IDPs/IDRs is crucial to comprehend and resolve the causes and impacts of these unstructured states [9].

Several experimental techniques have been employed to identify IDPs/IDRs including X-ray crystallography, nuclear magnetic resonance (NMR) and circular dichroism (CD) [7, 10]. However, due to the high-cost, laborious and time-consuming nature of these experiments, computational tools for predicting IDRs/IDPs in protein sequences have been an active area of research for the past 20 years.

IDP prediction was initially based on small-scale machine learning models [11], using neural networks and support vector machines. As more data and powerful tools emerged, the deep learning models have come to the forefront, in methods such as ESpritz [12], SPOT-Disorder [13], NetSurfP-2.0 [14], AUCpreD [15], SPINE-D [16] and SPOT-Disorder2 [17]. Depending on the input, these methods for protein disorder prediction can be divided into two categories: single-sequence and sequence profile from multiple sequence alignment. The single-sequence-based methods leverage the composition and connectivity of the single protein sequence along with its derived information such as statistical potentials and amino acid propensities [18–20]. These methods allow fast computation but are less accurate than those using evolutionary sequence profiles [21, 22] because sequence conservations are important indicators for structure and function. Examples of profile-based methods are SPINE-D [16], MDFp2 [23], AUCpreD [15], SPOT-Disorder [13] and SPOT-Disorder2 [17]. However, the exponential increase in the library size of protein sequences makes profile generation computationally more and more intensive. Moreover, most proteins (>90%) do not pertain to large sequence clusters [24] and, as a result, their sequence profiles may not be that useful.

Recently, the unsupervised pretrained language models have been applied to extract features from protein sequences [25–28], which has shown very promising results in the downstream forecast, such as tertiary contact, ontology-based protein function, secondary structure, contact map and mutational effect [28–33]. These breakthroughs inspire us to exploit the language models for disorder prediction. A recent study [30] evaluated and discussed the performance of different models for protein representation, indicating that ProtTrans achieved the best performance in most of the tasks. However, ProtTrans has not yet been used for protein disorder prediction. In this work, we employed ProtTrans language model to predict the intrinsic disorder regions of protein.

Here, we present LMDisorder, an alignment-free sequence-based prediction of protein intrinsic disorder. Using a recently released pretrained language model [34], LMDisorder can generate informative sequence representations fast and make an accurate prediction. Specifically, we first leveraged the pretrained language model ProtTrans to produce the sequence embedding. Then, transformer networks were employed to capture the sequence patterns including long-range dependencies between the residues, followed by a fully connected layer for predicting protein disorder probabilities. LMDisorder was shown to be superior to or as good as other profile-based methods including SPOT-Disorder2 on different datasets with its computational efficiency as efficient as single-sequence-based techniques. The high computation efficiency of LMDisorder enabled proteome-scale analysis of human, showing that proteins with high predicted disorder content were associated with specific biological functions.

MATERIALS AND METHODS

Datasets

In our experiments, the datasets were obtained from previous studies [17, 34], as shown in Table 1. Briefly, we obtained 4229 protein sequences (DM4229), which contain 72 fully disordered chains from DisProt v5.0 [35] and 4157 high-resolution X-ray crystallography structures from PDB (prior to the 5 August 2003). These proteins were randomly divided into a training set (2700 proteins), a validation set (300 proteins) and a Testing set (1229 proteins, DM1229). According to BLASTClust [36], the sequence similarity of these proteins is <25%. Furthermore, we also employed four independent test datasets SL329, DisProt228, Mobi9230 and DisProt452. The SL329 dataset is selected from the SL477 protein set, which is released by Sirota et al. [37]. The overlap between DM4229 and SL477 was removed to obtain an independent test set containing 329 non-redundant sequences (SL329). DisProt228 is the subset from DisProt Complement [22] and consists of the proteins in DisProt database v7.0. Mobi9230 was obtained from MobiDB [38] with experimental annotation, which derived from manually curated database and primary data (e.g., from PDB structures). We constructed a novel dataset DisProt452, which consists of the proteins deposited to the DisProt database from June 2020 to December 2022. After removing the long proteins (>700 residues) and the homologous proteins in our training set and validation set (25% sequence identity cutoff with BLASTClust [36]), we obtained 452 proteins and built a novel independent test dataset DisProt452. Meanwhile, we respectively created the subsets of SL329 and DM1229, named SL250 and Test1185, all of which are the same datasets employed previously by SPOT-Disorder and SPOT-Disorder2 [17]. In Table 1, We have detailed the number of proteins, ordered and disordered residues as well as the percentage of disorder in each dataset. These independent test datasets contain the latest data, such as the newly annotated proteins in DisProt452, which were deposited to the DisProt database from June 2020 to December 2022.

Protein sequence features

Language model representation. LMDisorder utilizes the recent protein language model ProtT5-XL-U50 [34] (denoted as ProtTrans) for feature extraction. This is a transformer-based auto-encoder named T5 [39], which was pretrained in a self-supervised manner on UniRef50 [40], learning to predict masked amino acids. We extracted the hidden states from the last layer of the ProtTrans encoder as sequence features, which is an |$n\times 1024$| matrix (⁠|$n$| is the sequence length). We have also examined another pretrained model, ESM-1b [29] (denoted as ESM), which was also pretrained on UniRef50 with transformer. Sequence features by ESM are an |$n\times 1280$| matrix. The computational costs of inference for both ProtTrans and ESM are low, and the feature extraction process of our whole benchmark datasets (~4800 sequences) can be completed within 10 minutes on an Nvidia GeForce RTX 3090 GPU.

Evolutionary information (for ablation study). We have also tested evolutionary-derived features (position-specific scoring matrix, PSSM and hidden Markov models, HMM) for feature ablation studies. PSSM was obtained by performing PSI-BLAST [36] search against UniRef90 [40] with three iterations and an E-value of 0.001. HHblits was employed to generate HMM profile [41] by aligning a query sequence against UniClust30 [42] with default parameters. Each residue was encoded into a 20-dimensional feature vector in PSSM and HMM.

Predicted structural properties (for ablation study). Putative structural properties were generated by utilizing SPIDER3 [43], whose inputs include protein sequence, PSSM profiles and HMM profiles. We extracted four types of structural features from the outputs of SPIDER3: (1) Solvent accessible surface area. (2) The sine and cosine values of the four protein backbone torsion angles (θ, ϕ, ψ and τ). (3) Half sphere exposures, which are the numbers of spatially neighboring residues in the top and bottom half of the contacting sphere: the boundary of the two hemispheres was determined by the Cα-Cα and Cα-Cβ direction vectors. (4) Predicted probabilities of the three secondary structures (α-helix, β-sheet and random coil). Each residue was encoded into a 14-dimensional vector using SPIDER3. Values in PSSM, HMM and SPIDER were further normalized to scores between 0 to 1 using the min–max normalization:

where v is the original feature value, and Min and Max are the smallest and biggest values of this feature type observed in the training set.

The architecture of LMDisorder

The overall architecture of LMDisorder is shown in Figure 1. First, the protein sequence is inputted into the pretrained language model ProtTrans to yield the sequence embedding, which is augmented by Gaussian noise to avoid overfitting. Then, the transformer networks are employed to capture the sequence patterns including long-range dependencies between the residues. This is followed by a fully connected layer adopted to predict protein disorder probabilities.

Transformer networks

We stack N identical transformer [44] layers to learn the sequence representations. Each transformer layer has two sub-layers, a multi-head self-attention module, and a position-wise feed-forward network. A residual connection [45] is employed around each of the two sub-layers, followed by the layer normalization [46]. Let |$\boldsymbol{H}\in{\mathbb{R}}^{n\times d}$| denotes the input of the self-attention module, where n is the sequence length and d is the hidden dimension. The input of the |${l}^{\text{th}}$| layer |$\boldsymbol{{H}}^{(l)}$| is projected by three matrices |$\boldsymbol{W}_Q\in{\mathbb{R}}^{d\times{d}_K}$|⁠, |$\boldsymbol{W}_K\in{\mathbb{R}}^{d\times{d}_K}$| and |$\boldsymbol{W}_V\in{\mathbb{R}}^{d\times{d}_V}$| to get the corresponding query, key, and value representations |$\boldsymbol{Q},\boldsymbol{K},\boldsymbol{V}$| as below:

The scaled dot-product self-attention is then calculated according to the following equation:

where |$\sqrt{d_K}$| is a scaling factor and |$\boldsymbol{A}$| is a matrix capturing the similarities between queries and keys. Instead of performing a single attention function, multi-head attention was employed to jointly attend to information from different representation subspaces at different positions. We linearly projected the queries, keys and values h times, perform the attention function in parallel, and finally concatenate them together. In this work, |${d}_K={d}_V=d/h$|⁠.

Fully connected layer and noise augmentation

The output of the last transformer layer is input to a fully connected layer to predict the protein disorder probabilities for all n amino acid residues of the protein:

where |$\boldsymbol{H}^{\left(N+1\right)}\in{\mathbb{R}}^{n\times d}$| is the output of the |${N}^{\text{th}}$| transformer layer; |$\boldsymbol{W}\in{\mathbb{R}}^{d\times 1}$| is a learnable weight matrix; |$b\in \mathbb{R}$| is a bias term, and |$\boldsymbol{Y}^{\prime}\in{\mathbb{R}}^{n\times 1}$| is the predictions for the n residues. The sigmoid function normalizes the output of the fully connected layer into binding probabilities from 0 to 1. In addition, to avoid overfitting, the sequence features from ProtTrans are augmented by Gaussian noise before being fed to the transformer networks while training:

where |$\boldsymbol{H}^{(0)}$| is the sequence features from ProtTrans, |$\boldsymbol{X}$| is a matrix of random values from the standard normal distribution with the same size as |$\boldsymbol{H}^{(0)}$|⁠, and ε is a hyperparameter to regulate the augmentation.

Implementation details

We utilized the training set to train LMDisorder, and the validation set to evaluate its generalization on unseen proteins. The training process lasted at most 50 epochs and we chose the model in the epoch with the best performance on the validation set as the final model. The validation performance was optimized by choosing the best feature combination and searching all hyperparameters through a grid search. Specifically, we employed a 2-layer transformer network with 128 hidden units and the following set of hyperparameters: h = 4, ε = 0.05, and a batch size of 12. The dropout rate was set to 0.3 to avoid overfitting. We utilized the Adam optimizer [47] with a learning rate of 3 × 10⁻⁴ for model optimization on the binary cross-entropy loss. We implemented the proposed model with Pytorch 1.7.1 [48].

Evaluation metrics

Similar to previous studies [17, 34], we employed the area under the receiver operating characteristic curve (⁠|${\text{AUC}}_{\text{ROC}}$|⁠), precision (Pr), sensitivity (Se), specificity (Sp), the area under the precision-recall curve (⁠|${\text{AUC}}_{\text{PR}}$|⁠), Matthews correlation coefficient (MCC), and the weighted score Sw (Sw = sensitivity + specificity – 1) to evaluate the performance of the method developed:

where TN, TP, FN and FP denote true negatives, true positives, false negatives and false positives, respectively.

RESULTS AND DISCUSSIONS

Feature importance and ablation analysis

The results of LMDisorder on the validation set and two independent test sets are shown in Table 2. It can be seen that the fractions of disordered residues are very different in different datasets. The fraction of disordered residues is only 9.0% of all residues in the validation set, compared to 9.5% in DM1229 and 56.5% in SL329. As |${\text{AUC}}_{\text{ROC}}$| is a balanced metric, similar performance for two data sets differed significantly in disorder content, indicates the robustness of our method. The |${\text{AUC}}_{\text{ROC}}$| of LMDisorder on validation set and DM1229 datasets are 0.940 and 0.920, and the |${\text{AUC}}_{\text{PR}}$| are 0.732 and 0.705, respectively. Although the disordered residues dominate the SL329 set, LMDisorder still achieved a similar performance with an |${\text{AUC}}_{\text{ROC}}$| of 0.911 and |${\text{AUC}}_{\text{PR}}$| of 0.909, respectively.

To examine the importance of various features for disorder prediction, we performed feature ablation experiments. As shown in Figure 2 and Supplementary Table S1, the model with either ProtTrans or ESM language model as sequence features performs significantly better than the profile-based models including PSSM+HMM and PSSM+HMM + SPIDER. Among them, the |${\text{AUC}}_{\text{ROC}}$| of the model using ProtTrans on validation set, DM1229 and SL250 are 0.940, 0.920 and 0.905, respectively, which is better than the model using ESM (0.922, 0.912 and 0.885). By comparison, the models using PSSM+HMM and PSSM+HMM + SPIDER features yield |${\text{AUC}}_{\text{ROC}}$| values of only 0.847, 0.840, 0.806 and 0.893, 0.886, 0.855, respectively. We noted that combining ProtTrans and PSSM+HMM + SPIDER features did not improve the model, with the |${\text{AUC}}_{\text{ROC}}$| value of 0.938, 0.918 and 0.905 for the validation set, DM1229 and SL250 sets, respectively, indicating that the pretrained model ProtTrans may have captured the evolution information and structure information of proteins. Furthermore, we found that combining ProtTrans with ESM only led to minor changes. The |${\text{AUC}}_{\text{ROC}}$| on the three datasets was 0.936, 0.922 and 0.906, respectively. Furthermore, to examine the effect of Gaussian noise, we tested the model without Gaussian noise. As shown in Supplementary Table S1, when we removed Gaussian noise, the |${\text{AUC}}_{\text{ROC}}$| and |${\text{AUC}}_{\text{PR}}$| were 0.917 and 0.692, decreasing by 2.4 and 5.5%, respectively. These results demonstrated the usefulness of Gaussian noise.

Comparison with single-sequence-based methods

We compared LMDisorder with seven single-sequence-based methods (ESpritz-N, ESpritz-D, ESpritz-X, IUPred2A-short [49], IUPred2A-long and Spot-Disorder-Single [34]) on DM1229 and SL329. In addition, we compared to NetSurfP-3.0 [50], which is based on ESM-1b language model. The single-sequence-based methods employ sequence-derived features such as one-hot features. In this work, we obtained abundant protein representation from ProtTrans language model, which is found here effective for protein disorder prediction. Figure 3 compared the ROC curves and Supplementary Figure S1 and S2 compared the precision–recall curves produced by these methods on DM1229 and SL329, respectively, along with performance indicators listed in Table 3. Among all the methods compared, LMDisorder achieved the highest |${\text{AUC}}_{\text{ROC}}$| and MCC values in both datasets, while NetSurfP-3.0 achieved the second best. LMDisorder achieved the best results on SL329, with |${\text{AUC}}_{\text{ROC}}$| of 0.911, AUC_PR of 0.910 and MCC of 0.655, which is 1.0% (⁠|${\text{AUC}}_{\text{ROC}}$|⁠), 1.4% (AUC_PR) and 5.5% (MCC) higher than NetSurfP-3.0, respectively. LMDisorder and NetSurfP-3.0 achieved a comparable level on DM1229, where the |${\text{AUC}}_{\text{ROC}}$|⁠, AUC_PR and MCC of LMDisorder are 0.920, 0.705 and 0.621 respectively and the same indicators of NetSurfP-3.0 are 0.918, 0.732 and 0.620. It is worth noting that SPOT-Disorder-Single achieved the third best results on two datasets. For DM1229 dataset, the |${\text{AUC}}_{\text{ROC}}$|⁠, AUC_PR and MCC of SPOT-Disorder-Single are 0.868, 0.599 and 0.518, which are 5.6% (⁠|${\text{AUC}}_{\text{ROC}}$|⁠), 15% (AUC_PR) and 17% (MCC) lower than LMDisorder, respectively. And on SL329, the |${\text{AUC}}_{\text{ROC}}$|⁠, AUC_PR and MCC of SPOT-Disorder-Single are 2.6% (⁠|${\text{AUC}}_{\text{ROC}}$|⁠), 2.6% (AUC_PR) and 7.8% (MCC) lower than LMDisorder, respectively. For a more comprehensive comparison, we compared LMDisorder with other single-sequence-based methods on Mobi9230, and found our model consistently performed the best (Supplementary Table S3). The |${\text{AUC}}_{\text{ROC}}$| and AUC_PR of LMDisorder are 0.863 and 0.8 respectively, while the second best method (NetSurfP-3.0) are 0.846 (⁠|${\text{AUC}}_{\text{ROC}})$| and 0.8 (AUC_PR). Additionally, we constructed a novel independent dataset DisProt452 and tested LMDisorder with other methods in Supplementary Table S4. The results have shown that LMDisorder still has the best performance. The |${\text{AUC}}_{\text{ROC}}$| and AUC_PR of LMDisorder are 0.875 and 0.854, respectively, while the second best method (spot-disorder-single) are 0.840 (⁠|${\text{AUC}}_{\text{ROC}}$|⁠) and 0.817 (AUC_PR). The consistent performance on multiple data sets further confirmed the robustness of our model.

Comparison with state-of-the-art profile-based methods

We compared LMDisorder with 12 profile-based methods [51] for DisProt228, including s2D [34], AUCpreD, JRONN [52], ESpritz-D, MFDp2 [53], DISOPRED [54], MobiDB-lite [55], ESpritz-N, SPINE-D, MFDp [23], SPOT-Disorder and SPOT-Disorder2. Among these, SPOT-Disorder2 is a state-of-the-art profile-based method, which had the best performance in CAID prediction [56]. The profile-based methods [16, 17] usually use the evolutionary sequence profiles obtained from multiple sequence alignment [21, 22] created mostly from HHBlits [41] and PSI-Blast [36]. In general, profile-based methods are more accurate than single-sequence-based, although these methods will take more time to calculate the features. As shown in Table 4, LMDisorder improved over 11 Profile-based methods, while achieving similar performance as SPOT-Disorder2. The |${\text{AUC}}_{\text{ROC}}$|⁠, |${\text{AUC}}_{\text{PR}}$|⁠, MCC and Sw of LMDisorder are 0.800, 0.700, 0.471 and 0.484, respectively, compared to 0.809, 0.716 and 0.499, respectively, given by SPOT-Disorder2.

We further compared the top three methods (SPOT-Disorder, LMDisorder, SPOT-Disorder2) in Table 4 on additional two larger datasets. Table 5 shows the comparison of LMDisorder with SPOT-Disorder and SPOT-Disorder2 on Test1185 and SL250. LMDisorder is able to have a better performance on these two test sets. On Test1185, the |${\text{AUC}}_{\text{ROC}}$| and |${\text{AUC}}_{\text{PR}}$| of LMDisorder are 0.920 and 0.709, respectively, which are 3.3 and 9.1% higher than those of SPOT-Disorder and 0.6 and 2% higher than those of SPOT-Disorder2. Similarly, the |${\text{AUC}}_{\text{ROC}}$| and |${\text{AUC}}_{\text{PR}}$| of LMDisorder on SL250 are 0.905 and 0.890, which are 1.3 and 1.7% higher than those of SPOT-Disorder and 0.4 and 0.1% higher than those of SPOT-Disorder2. Thus, the performance of the proposed LMDisorder has matched the performance of the current state-of-the-art profile-based method at a fraction of computational time.

IDPs are associated with specific biological functions

Since LMDisorder is much faster than profile-based methods, we used LMDisorder to predict IDRs/IDPs in all human proteins in Uniprot [57]. Previous studies have shown that IDRs/IDPs play essential roles in many biological processes. Thus, we investigated whether the proteins with high disorder content predicted by LMDisorder were associated with these functions. Specifically, we selected the proteins with a fraction of disorder contents above 90% and conducted Gene Ontology (GO) [58] enrichment analysis using gProfiler [59]. Then we randomly selected the same number of proteins for 200 enrichment functional analyses and picked the lowest FDR (P-value adjusted by Benjamini-Hochberg False Discovery Rate) value (⁠|$1.0\times{10}^{-25}$|⁠) as the threshold. We found that these genes significantly enriched 5, 62 and 5 GO items in the molecular function (MF), biological process (BP) and cellular components (CC) category, respectively. Figure 4 shows the top-5 GO items of the three categories and the details are in Supplementary Table S6. The proteins with high disorder content enriched many important biological functions and many of them were supported by the literature. For instance, there are 12 pathways related to transcription [51, 60, 61], three pathways related to protein binding [62], and three pathways related to regulatory [6]. This indicates that LMDisorder provides biologically meaningful predictions, which may facilitate investigating the functions of the proteins without rich function annotations. The above-described enrichment of some functions is consistent with previous analysis, including cell differentiation, development and regulation [63, 64]. We also discovered some new functions associated with strong intrinsic disorder (Supplementary Table S6), including RNA biosynthetic process (⁠|$\text{FDR}=1.6\times{10}^{-53}$|⁠), DNA-templated (⁠|$\text{FDR}=5.2\times{10}^{-54})$| and RNA polymerase (⁠|$\text{FDR}=5.2\times{10}^{-54})$|⁠. Further studies are required to confirm these findings. Meanwhile, we have added the proteome level analysis for some proteins in Supplementary Table S5. Firstly, we selected proteins with less than 25% sequence similarity (using BLASTClust [36]) to the training set. And then the disorder ratio for each of the proteins was predicted. Finally, the proteins were ranked according to the prediction results and the top ones among them were picked. It is found that the proteins with high disorder contents have some special functions, such as DNA-binding, Transcription regulation and Cell division, which is consistent with the previous Gene Ontology enrichment analysis.

DISCUSSION

In this paper, we have developed a language-model-based method called LMDisorder to predict IDRs/IDPs in protein sequences. IDPs and IDRs are abundant in all species [65, 66], and relevant in many diseases [67, 68]. The existing profile-based methods can provide highly accurate prediction. However, they are computationally too intensive to perform genome-scale prediction due to the computational requirement for profile calculations. Here, we showed that using language model embedding as input features we can achieve speed and accuracy at the same time. For a protein with 500 residues, it takes about 15 minutes on an Nvidia GeForce RTX 3090 GPU for SPOT-Disorder2 but only one second on the same computer for LMDisorder with essentially the same performance. Meanwhile, we also tested other single-sequence-based methods, most finished within 1 to 5 seconds, because of no requirement of the time-consuming generation of sequence profiles. And compared with NetSurfP-3.0, which is the state-of-the-art language-model-based method, LMDisorder reached the equivalent or even better performance. This further proves the effectiveness of LMDisorder. In addition, the accuracy of our method remained essentially the same with the change of the protein length (Supplementary Figure S3), further indicating the robustness of our method.

One interesting observation is that the proteins with high disorder content are associated with specific biological functions. Through the enrichment analysis, we found that these proteins significantly enrich 72 GO items and many of them were supported by previous literature [6, 51, 60, 61, 69, 70]. In addition, we also discovered some new strong intrinsic-disorder-associated functions, which may be of interest for further studies.

Another observation is that combining ProtTrans and ESM did not improve over ProtTrans alone. This is a bit surprising because a recent study indicates that their combination led to an improved prediction of protein secondary structure and other tertiary structural properties [31]. This is perhaps due to the fact that disordered regions, unlike structured regions, are less conserved, and thus, evolutionary information from different language models may have captured different aspects of structural characteristics. Disorder predictions, on the other hand, are less sensitive to small intrinsic difference in the language models. In the future, we consider using the structural information predicted by AlphaFold2 [71, 72] or ESMFold [73], and then building models using graph networks [74] for further improving the performance.

In summary, this work developed a sequence-based, alignment-free method LMDisorder for protein disorder prediction. LMDisorder combines the advantages of current sequence-based and profile-based methods and greatly improves the prediction speed while ensuring high-prediction performance. The technique could be useful for highly accurate proteome-scale analysis.

DATA AVAILABILITY

The datasets, the source codes and the trained model are available on https://github.com/biomed-AI/LMDisorder.

FUNDING

This study has been supported by the National Key R&D Program of China [2022YFF1203100], National Natural Science Foundation of China [12126610], and by the Supercomputing facilities of Shenzhen Bay Laboratory.

Author Biographies

Yidong Song is a Ph.D. student in the School of Computer Science and Engineering at Sun Yat-sen University. His research interests include deep learning, graph neural network, protein disorder prediction and protein function prediction.

Qianmu Yuan is a Ph.D. student in the School of Computer Science and Engineering at Sun Yat-sen University. His research interests include deep learning, graph neural network and protein disorder prediction.

Sheng Chen is a Ph.D. student in the School of Computer Science and Engineering at Sun Yat-sen University. His research interests include deep learning, graph neural network and protein structure prediction.

Ken Chen is a Ph.D. student in the School of Computer Science and Engineering at Sun Yat-sen University. His research interests include deep learning, graph neural network and protein function prediction.

Yaoqi Zhou is a senior principal investigator in the Institute of Systems and Physical Biology, Shenzhen Bay Laboratory, Shenzhen, China. His research interests include protein disorder prediction, protein design and protein/RNA structure prediction.

Yuedong Yang is a professor in the School of Computer Science and Engineering at Sun Yat-sen University. Currently he focuses on integrating HPC and AI techniques for multi-scale biomedical research.

REFERENCES