HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) All Versions of this Article: M400215-MCP200v1 4/6/762    most recent Submit a response Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Glossary Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Weatherly, D. B. Articles by Orlando, R. Search for Related Content PubMed PubMed Citation Articles by Weatherly, D. B. Articles by Orlando, R.

---

Research

A Heuristic Method for Assigning a False-discovery Rate for Protein Identifications from Mascot Database Search Results ^*

D. Brent Weatherly^,, James A. Atwood, III^¶,, Todd A. Minning, Cameron Cavola^¶, Rick L. Tarleton and Ron Orlando^¶,||

From the Center for Tropical and Emerging Global Diseases and ^¶ Complex Carbohydrate Research Center, University of Georgia, Athens, Georgia 30602

	ABSTRACT

TOP ABSTRACT EXPERIMENTAL PROCEDURES RESULTS DISCUSSION REFERENCES

 
MS/MS and database searching has emerged as a valuable technology for rapidly analyzing protein expression, localization, and post-translational modifications. The probability-based search engine Mascot has found widespread use as a tool to correlate tandem mass spectra with peptides in a sequence database. Although the Mascot scoring algorithm provides a probability-based model for peptide identification, the independent peptide scores do not correlate with the significance of the proteins to which they match. Herein, we describe a heuristic method for organizing proteins identified at a specified false-discovery rate using Mascot-matched peptides. We call this method PROVALT, and it uses peptide matches from a random database to calculate false-discovery rates for protein identifications and reduces a complex list of peptide matches to a nonredundant list of homologous protein groups. This method was evaluated using Mascot-identified peptides from a Trypanosoma cruzi epimastigote whole-cell lysate, which was separated by multidimensional LC and analyzed by MS/MS. PROVALT was then compared with the two traditional methods of protein identification when using Mascot, the single peptide score and cumulative protein score methods, and was shown to be superior to both in regards to the number of proteins identified and the inclusion of lower scoring nonrandom peptide matches.

All software designed for matching MS/MS spectra to peptide sequences function in a similar manner. The experimental peptide masses are first compared with theoretical peptide masses derived from in silico enzymatic digestion of the specified protein database. The subsets of theoretical peptides with similar masses (within a user-defined mass tolerance) to the experimental peptides are fragmented in silico according to specific cleavage rules. These theoretical fragment ion masses are then compared with the masses of fragment ions from the experimental MS/MS spectra. Although all search engines will match a theoretical peptide sequence to an experimental MS/MS spectrum, every match is not always correct. Thus, all search algorithms attempt to assign scores indicating the degree of similarity between the experimental and theoretical MS/MS spectra.

Probability-based scoring algorithms attempt to accurately reflect the probability of a match being random by gathering information about the database itself and using this information in the score calculation. Of the probability-based search engines described in the literature, Mascot (14) is one of the most widely used. The Mascot probability model is based on the MOWSE (22) algorithm. The MOWSE algorithm functions by creating a matrix of weighting factors for the specified enzymatic cleavage, and sequence entries are clustered into cells formed by a distribution of intact protein and peptide fragment molecular masses. Therefore, each cell contains the frequency at which peptide molecular masses occur for a distribution of protein masses. The frequency factors are normalized and used to calculate the final score. Mascot reports a probability-based ion score for each peptide match, which indicates the statistical significance of that MS/MS spectral assignment. Following clustering of peptides to proteins, Mascot combines individual ion scores and reports a cumulative protein score. Although the Mascot algorithm (www.matrixscience.com/help/results_help.html) establishes a threshold ion score under which there is less than 95% confidence in any individual peptide match, the algorithm attributes no such statistical significance to the cumulative protein scores.

Published proteomics initiatives using Mascot have established a variety of criteria necessary to distinguish between correct and random protein assignments. A general approach has been to report a protein identified if at least one peptide from that protein is matched at or above the threshold ion score (23–25). We refer to this as the single peptide score method (SPM).¹ Another approach, the cumulative protein score method (CPM), has been to eliminate peptide matches below a given ion score and utilize an empirically derived protein score to identify proteins (26, 27). These strategies hinge on the ability of Mascot to accurately reflect true probabilities associated with matching MS/MS spectra to peptide sequences. With this in mind, evaluations of the Mascot scoring algorithm have demonstrated that the ion score thresholds established for a match to be considered significant accurately reflect a 95% confidence level when searching data from peptide mass fingerprints (28) and tandem mass spectra generated on high precision Q-TOF instruments (17). However, in large scale proteomics projects, when thousands of peptides are identified, this level of confidence may be unsatisfactory, resulting in several hundred false-positive peptide and/or protein identifications. To discriminate between correct and random peptide assignments from a Mascot search, several methods have been proposed. These include statistical models to evaluate peptide assignments using information gathered during the database search (29, 30) and filters that eliminate random matches based on the properties of the assigned peptides and experimental conditions (31–33). These methods, although allowing the removal of a large portion of the false-positive peptide identifications, either ignore potentially valid lower scoring peptide matches or do not account for the false-discovery rate (FDR) associated with the resulting protein identifications.

For most proteomics projects, the eventual goal is to establish the repertoire of expressed proteins in the sample of interest (34). As noted by Nesvizhskii et al. (35, 36), this is not a straightforward process when analyzing proteins at the peptide level. The statistical significance associated with matching MS/MS spectra to peptide sequences does not correlate with the likelihood that the consequent proteins are identified (35–37). This is ascribed to real peptide matches clustering to individual proteins, whereas random matches occur with an equal distribution across the entire database (35, 36). To calculate an overall probability or expectation value for a protein assignment, Nesvizhskii et al. (35, 36), Sadygov et al. (37), and Fenyo and Beavis (30) have proposed combining individual probabilities or expectation values for potential peptide matches. Such methods allow the specification of expectation values (30) or probabilities (35–37) to the proteins rather than peptide assignments and enable the inclusion of potentially valid low scoring peptide matches in the final protein assignment (35–37). However, these methods require the recalculation of expectation values or probabilities of peptide assignments made by the database search software and provide no information about the overall proportion of incorrect matches in a subset of identified proteins.

Herein we detail PROVALT, a tool that organizes large proteomic datasets and calculates protein FDRs (PRO-FDR) using peptides identified by Mascot. PROVALT extracts peptide matches from multiple Mascot results files, eliminates peptide redundancy, and clusters peptides to their corresponding proteins. In addition, homologous proteins for which no distinguishing peptide has been identified are organized into protein groups. Pertinent information such as sample origin and experimental conditions are linked to each peptide and protein match as well. We also report the implementation of a statistical model within PROVALT, an algorithm that determines PRO-FDRs when using Mascot. This model is based on the implementation of a random database (32, 38, 39) and Mascot peptide probability scores to calculate expected PRO-FDRs for a minimal number of expressed proteins identified by Mascot-matched peptides. As in prior reports, the random database served as the null hypothesis. Identifications resulting from the null hypothesis are considered to be random, and the scores assigned to the random matches should follow a quasi-normal distribution with a false-positive rate related to each score. PROVALT compares the score distributions obtained from searching the normal and random databases to calculate the FDRs associated with each score threshold. The goal is to set score thresholds that identify as many real proteins as possible while encountering a minimal number of false-positive protein identifications. It is important to note the difference between the false-positive rate and FDR as it applies to protein identification. The false-positive rate is the rate at which truly null protein identifications are treated as significant, whereas the FDR is the rate at which significant protein identifications are actually null (40–42). For example, Peng et al. (38) used a random database to identify 7,537 peptides at a FDR of 1%. Thus, 75 of the identified peptides were likely random matches.

Rather than calculate error rates at the peptide level, we demonstrate that the PRO-FDR calculations employed by PROVALT provide a reasonable balance between the number of correct and incorrect protein assignments. Here we evaluate this tool using a dataset of 50,000 MS/MS spectra generated from 100 LC-MS/MS analyses of peptides from Trypanosoma cruzi epimastigote whole-cell lysates, which were separated by multidimensional LC. Using this dataset, we evaluate the two predominant modes of protein identification when using Mascot, the SPM and CPM. We then compare the PRO-FDRs resulting from these traditional methods with the PRO-FDRs generated from the PROVALT analysis.

	EXPERIMENTAL PROCEDURES

TOP ABSTRACT EXPERIMENTAL PROCEDURES RESULTS DISCUSSION REFERENCES

 
Cell Culture and Peptide Preparation—
T. cruzi (Brazil) epimastigotes were grown as described previously (43). Proteins were extracted from 5 x 10⁷ parasites using Tri-Reagent (Sigma) per the manufacturer’s instructions. Proteins were reduced (8 M urea, 200 mM Tris-HCl, 40 mM DTT, pH 8.5) for 1 h at 55 °C followed by carboxyamidomethylation with iodoacetamide (80 mM) for 30 min at room temperature. The protein solution was diluted to 6 M urea by the addition of H₂O and digested overnight at 37 °C with endoproteinase Lys-C (1:100; Sigma). Following dilution to 1 M urea with 30 mM ammonium bicarbonate, proteins were digested overnight at 37 °C with sequencing-grade porcine trypsin (1:50; Promega, Madison, WI). The digest was lyophilized to dryness, reconstituted in 500 µl of 0.1% TFA, and filtered prior to separation.

First-dimension Reverse Phase Chromatography (RP)—
First-dimension separation was performed on an Agilent 1100 series work station (Palo Alto, CA) configured with a 4.6 x 150-mm Jupiter C₁₈ column (Phenomenex, Torrance, CA). Buffer A was H₂O/0.1% TFA, and buffer B was ACN/0.1% TFA. The peptides were loaded onto the column (flow rate 0.75 ml/min), desalted for 10 min at 5% buffer B, and then sequentially eluted during a 40-min linear gradient from 5 to 45% buffer B. Ten fractions were collected (Table I), frozen, and lyophilized to dryness.

RPLC-MS/MS—
Each fraction was analyzed independently using a Waters capillary LC (Milford, MA) interfaced directly to a Q-TOF-2 tandem mass spectrometer (Micromass, London, UK). Mobile phase A and B were H₂O/0.1% formic acid and ACN/0.1% formic acid, respectively. Each fraction was reconstituted in 5 µl of mobile phase A and loaded onto the column (PEPMAP, 15 cm x 180 µm; LC-Packings, Sunnyvale, CA) at a flow rate of 1 µl/min. To maximize separation, RP gradients were designed to correspond with the percentage of organic at which each peptide fraction eluted from the first-dimension RP (Table I).

MS/MS Parameters—
The instrument was set to acquire a 1-s MS scan from 500 to 1,700 Da, during which up to four precursor ions were selected for MS/MS analysis from 50 to 2,000 Da for 2 s. Precursor selection was based upon a threshold of 5 counts per second for peptides with charge states of 1–4, and dynamic exclusion was enabled for 8 min. Raw mass spectra were processed into peak-list format prior to database searching.

Protein Sequence Database—
Two sequence databases were constructed for our analyses. First, we created a representative database (normal) consisting of the 25,000 T. cruzi gene annotations provided by the Trypanosoma cruzi Sequencing Consortium as well as possible contaminating proteins from Bos taurus, Equus caballus, Homo sapien, and proteases. A randomized database was constructed by reversing the sequences in the normal database. The random database was used to establish accurate scoring thresholds for protein identification in the normal database.

Database Searching—
Mascot (version 1.8) searches were performed with the following parameters: a specified trypsin enzymatic cleavage with one possible missed cleavage, peptide tolerance of 50 parts-per-million, fragment ion tolerance of 0.2 Da, variable modifications caused by carbamylation (+43 Da), and carboxyamidomethylation (+57 Da).

PROVALT, Protein Organization and FDR Calculations—
The PROVALT method for identifying proteins at a specified PRO-FDR is outlined in Fig. 1. Protein identification begins by extracting all Mascot-matched peptides and corresponding ion scores from the normal and random Mascot search results. The results from each dataset are combined and filtered yielding nonredundant lists of peptides. The peptides are grouped as functions of score thus forming score bins (B_i) containing all peptides equal to or exceeding each Mascot ion score (i), where i represents a specific ion score S ranging from M to N.

(Eq. 1)

Peptides in each score bin are then clustered to their corresponding proteins. Proteins are selected based on the degree of peptide coverage c.

(Eq. 2)

Starting with C, a histogram is formed based on the frequency of protein identifications within each B_i for both the random and normal peptide matches (Fig. 2). Protein identification FDRs are then calculated for each score bin as follows.

(Eq. 3)

(Eq. 4)

and thus the minimal Mascot ion score threshold necessary to achieve the specified PRO-FDR for the given degree of peptide coverage (c). Peptides meeting these criteria are stored, and the remaining peptides that were not matched to proteins are then grouped as a function of ion score (S), thus forming new score bins (B_i) containing all peptides equal to or exceeding each Mascot ion score (i). For the next degree of coverage (c = C –1), the distribution of ion scores will have a minimum (M) that is dependent on score threshold (S_c) determined for the previous degree of peptide coverage as follows.

(Eq. 5)

Previously unmatched peptides in each bin are again clustered to their corresponding proteins along with the peptides matched at the previous levels. For the next degree of peptide coverage (C –1), another histogram is formed, and the PRO-FDR is then calculated for each score bin. S_c (for c = C –1) is calculated, and the peptides meeting these criteria are stored. This process is repeated until c = 1. For this dataset, the ion score thresholds for each coverage level are displayed in Table II.

View larger version (28K): [in this window] [in a new window]   FIG. 1. The PROVALT method for protein identification and statistical validation. Protein identification begins with the extraction of peptide matches from both normal and random Mascot database search results. Peptides are binned as a function of ion score and clustered to proteins to yield a minimal protein list. Proteins are selected based on degree of peptide coverage. The number of identified proteins between the random and normal database searches is compared as a function of peptide ion score to determine the minimal ion score and peptide coverage level necessary to achieve a 1% PRO-FDR. Matching proteins and peptides are output, and the process is iteratively repeated for each coverage level.

View larger version (20K): [in this window] [in a new window]   FIG. 2. Distribution of FDRs as a function of minimal ion score for proteins identified by six or more peptides. PROVALT constructs a histogram that compares the number of proteins identified in both the normal and random database searches. This comparison is iteratively performed for each peptide coverage level. The above histogram compares the frequency of protein identifications for proteins containing at least six or more peptides at or above specific Mascot ion scores. Using Eq. 3, the expected percentage of false discoveries is calculated for each peptide bin, and a 1% FDR is achieved if a protein is identified by six or more peptides with at least a minimal Mascot ion score of 11.

	RESULTS

TOP ABSTRACT EXPERIMENTAL PROCEDURES RESULTS DISCUSSION REFERENCES

Mascot Parsing Tools—
The integration of the Mascot parser allows the extraction of relevant peptide information from multiple Mascot results files. The parsing tool functions with both dat and html versions of Mascot results. The inclusion of an html parser allows users to access the nonlicensed version of Mascot (www.matrixscience.com), save multiple results files as html, and extract the peptide matches for later protein identification. The extracted information includes sample source, peptide sequence, ion score, precursor mass error, and fragment mass errors. The information is stored along with the peptide sequences and is later integrated with the protein identification. Following parsing, PROVALT eliminates redundant peptide identifications (but tracks the information associated with each valid occurrence), matches peptides to proteins, and then determines a list of unique protein groups. Although redundant peptides are removed and only the highest probability peptide match is used for the subsequent protein identification, all peptide matches are reported. Peptide matches are considered redundant if they have identical sequences plus any modifications or resulted from precursor ions of multiple charge states. If a peptide was identified with and without a modification, PROVALT does not consider these redundant. All matches to a given peptide regardless of precursor ion charge state are reduced to a single identification, and the highest ion score (S) is used in the PRO-FDR calculation. Although spectra from multiple precursor ion charge states may exhibit distinct fragmentation patterns, these peptides are not considered as independent peptide matches by PROVALT and are not allowed to contribute to the peptide coverage (c) (44). We have adopted this approach because of the fact that during database searching precursor ions of multiple charge states will be converted to their singly charged precursor masses prior to comparison with the in silico derived peptides. The subset of in silico derived peptides passing within the precursor ion mass tolerance will essentially be identical for the singly charged and multiply charged species. Thus identical assignments made from peptides with different precursor ion charge states are not unequivocally independent events. Conversely, peptide assignments with alternate modifications are matched through different parent mass filters and are treated as independent.

Peptide Clustering Tools—
Correlating peptides to proteins in a large scale proteomics project is not a straightforward process; Nesvizhskii and Aebersold (36) and Rappsilber and Mann (45) noted that eukaryotic protein databases contain many homologous proteins that are often indistinguishable by a single peptide identification. Therefore, if a group of peptides corresponds to more than one protein, it is impossible to determine which protein is actually expressed without performing additional experiments. In a manner similar to the organization described in Nesvizhskii and Aebersold (36), PROVALT accounts for the presence of protein families by determining protein groups that represent all peptides used to identify a group of homologous proteins. The protein groups were determined as follows: If protein "A" and protein "B" both contained the peptide sequences "x" and "y," then the identification of peptides x and y would not allow proteins A and B to be distinguished from each other. Thus protein group 1 would be defined as peptides x and y and would contain both proteins A and B. If protein "C" contained the peptide sequences x and y and "z," then the identification of peptide z would allow protein C to be differentiated from both proteins A and B. Thus protein C would be placed in protein group 2. Following peptide clustering a protein that contains all the peptides in the protein group is used in the PRO-FDR calculation. For example, protein B would represent protein group 1 with a peptide coverage of c = 2, and the ion scores (S) of both peptides x and y would be used in the PRO-FDR calculation. Although proteins A and B may both be expressed, for statistical reasons we considered the protein group as a single protein identification. If the representative protein (B) is determined to be identified above the defined PRO-FDR, then the final report contains all of the homologous proteins (A and B), not just the representative protein. Our method assumes that if a protein is identified above a given error rate, the constituent peptides are also identified with the same confidence. Therefore, this method makes no statistical distinction between proteins in the same group that are identified by differing degrees of peptide coverage. Although it may be assumed that a protein is more likely to be expressed if it contains a larger number of matched peptides, it is indeterminate whether a subset of those peptides resulted from the presence of a homologous protein. As a result, the total number of protein groups identified represents the minimal number of expressed proteins.

Single Peptide Score Method—
The most common approach for separating random from real protein identifications is the SPM, in which a protein is identified if it contains at least a single peptide at or above the Mascot-derived ion score threshold. Mascot defines a peptide match as significant if the probability of that event occurs by chance with a frequency of <5%. Thus a peptide with an ion score corresponding to at least an absolute probability 0.05 is considered a real match and used in the protein identification. To model this method, the ion score thresholds that corresponded to each absolute probability needed to be determined over the entire dataset. This was accomplished by searching both the normal and random databases, forming a histogram of the frequency of peptide matches as a function of ion score, and then applying the equation below to calculate the distribution of peptide false-discovery rates (PEP-FDR).

(Eq. 6)

Fig. 3 shows the distribution of peptide matches that occurred at or above a range of ion scores for the normal and random database search results as well as the calculated PEP-FDRs. At an ion score of 27 or above, 5% of the peptides matching in the normal database would be random identifications. Typically, when the SPM is applied, only peptide matches with ion scores equal to or exceeding 27 would be used to identify proteins.

View larger version (24K): [in this window] [in a new window]   FIG. 3. Real discovery rates as a function of single ion score thresholds. The frequency of peptide matches at or above Mascot ion score thresholds were plotted for both the random and normal database searches. Equation 6 was applied to calculate the real discovery rate (100% –PEP-FDR) for each threshold. To achieve a 1% PEP-FDR, only peptides exceeding ion scores of the 34 would be selected to identify proteins. However, real matches may occur at lower scores.

View larger version (22K): [in this window] [in a new window]   FIG. 4. PRO-FDRs and PEP-FDRs as a function of mascot ion score threshold. Using the single peptide score threshold method, peptides were selected for protein identification if they exceeded specific Mascot ion scores. This method was applied at each Mascot ion score threshold, and the resulting PEP-FDRs and PRO-FDRs were plotted. From the figure above it is evident that a FDR associated with peptide matches does not correspond to a FDR for protein identifications.

View larger version (22K): [in this window] [in a new window]   FIG. 5. Ratio of peptides to proteins identified at a distribution of ion score thresholds. Proteins were identified by peptides at a distribution of peptide ion score thresholds in both the normal and random database searches. The ratio of peptides to proteins was calculated and plotted as a function of ion score. The ratio of peptides to proteins is as much as three times higher in the normal database over the random database, indicating that random peptide matches do not cluster to proteins as do real peptide matches.

View larger version (15K): [in this window] [in a new window]   FIG. 6. Number of peptides (A) and proteins (B) identified by PROVALT, SPM, and CPM. Using the PROVALT method of protein identification (1% PRO-FDR), the number of proteins identified was increased an average of 25% above the number identified by SPM and CPM. The PROVALT method also allowed the 83% more peptides than was identified by the SPM at a 1% PRO-FDR.

Comparison of PROVALT and CPM—
We evaluated the effectiveness of the CPM by identifying proteins in both the normal and random databases and calculating FDRs based on cumulative protein score thresholds. We first extracted all top ranking unique peptides with an ion score at or above 1 from both databases, clustered them to the corresponding proteins, and calculated a total protein score by summing the individual ion scores. Fig. 7 is the frequency distribution of proteins that match at a range of protein score thresholds (from 20 to 120) for the normal and random databases as well as the calculated PRO-FDRs. As stated above, determining a protein score threshold has previously been largely empirical, and for small datasets Mascot reports as "significant" hits all proteins with a cumulative score above the ion score threshold. However, Fig. 7 indicates that employing this method for this dataset would result in a PRO-FDR of 97% at a score of 27 (ion score determined from Fig. 3). To achieve a PRO-FDR of 1%, a minimum protein score of 110 would be required. Fig. 6, A and B, compares the number of proteins and peptides identified at a 1% PRO-FDR when using the CPM and PROVALT methods. Although a high confidence PRO-FDR for protein identifications is achieved using both, a vast number of nonrandom protein matches is discarded by the CPM (Fig. 7). On the other hand, if a lower protein score threshold is used, the PRO-FDR would be unacceptably high.

View larger version (26K): [in this window] [in a new window]   FIG. 7. Cumulative score method and resulting real discovery rates. Proteins were identified using the CPM at a distribution of cumulative protein score thresholds for both the normal and random database searches. The number of protein identifications for each database was plotted along with the resulting real discovery rate at each cumulative protein score threshold. When the CPM is applied using the cumulative protein scores above, a cumulative score of 84 would be required to achieve a 5% FDR, and a large number of potentially real protein matches would be missed.

Comparison of PROVALT and Modified CPM—
Because the frequency of random peptide matches increases as the Mascot ion score decreases, the application of a low score cutoff to filter potentially random peptide matches has been proposed (26, 27). However, this method ignores higher scoring potentially random matches that will positively contribute to the final PRO-FDR calculation. For comparison of PROVALT with this version of the CPM, we applied several peptide ion score thresholds (1, 10, 20, 30, and 40) to filter peptides prior to protein identification with the CPM at a 1% PRO-FDR. Fig. 8 compares the distribution of peptides at various peptide ion score ranges using our method and the cumulative score methods to identify proteins. From Fig. 8 it is evident that our method is more discriminating in the inclusion of lower scoring peptides than the cumulative score method, which utilizes a higher number of lower scoring peptides for protein identification. In fact, when the cumulative score method is employed with peptide score thresholds <30, the predominant number of peptides is within the lowest ion score range. Thus, protein identification with this method is mostly facilitated by inclusion of the peptides that are most likely random matches. Using PROVALT, the largest number of peptides used for protein identification falls within the highest ion score range. The cumulative score method does not attribute any significance to the lower scoring peptides but rather assumes that their presence with higher scoring peptides on the same proteins makes them significant. However, because so few high scoring peptides exist in the random database, lower scoring peptides that match to the normal database have an unfair statistical advantage. PROVALT utilizes only a stochastic peptide ion score threshold that is based on the assumption that random matches will occur with an even distribution across all proteins in the database, whereas real matches will cluster to the proteins that are actually expressed. Thus, score thresholds are calculated according to the probability that some number of peptides match above a given ion score to the same protein regardless of total protein score. This allows us to have high confidence in all of the peptides that are selected by our method and thus the proteins that they identify. Without high confidence in the lower scoring peptides, the cumulative score method provides essentially the same amount of quality information as the SPM.

View larger version (59K): [in this window] [in a new window]   FIG. 8. Comparison of peptides used in protein identifications from cumulative score methods and the PROVALT method. The CPM (1% PRO-FDR) was employed after filtering peptides below given ion scores. The peptides used to identify proteins from each ion score filter were grouped into bins according to ion score. The distribution of peptide scores was then compared between each cumulative score method and the PROVALT (1% PRO-FDR) method. From the graph, it is evident that the PROVALT method is more selective in the inclusion of lower scoring peptides.

	DISCUSSION

TOP ABSTRACT EXPERIMENTAL PROCEDURES RESULTS DISCUSSION REFERENCES

The main goal in using external tools such as PROVALT is to separate the valid from incorrect protein identifications, the difficulty of which increases with the size of the dataset. As has been shown, the problem with using either the SPM or CPM is balancing the tradeoffs between choosing high stringency that discards potentially useful information and low stringency that does not discard the useless information. Thus it would be advantageous to determine a measure of confidence in protein identifications using all possible peptide contributions to that protein and select proteins based on a user-defined minimum confidence. Previous work has demonstrated the utility of the random database approach for calculating FDRs for peptide identifications (38). Such methods facilitate the calculation of the proportion of random peptide matches among all peptide matches deemed significant. However, the calculation of a PEP-FDR provides no information about the error rate of a specific protein identification. With the ultimate goal being the identification of proteins rather than peptides, a more thorough approach would be to define a PRO-FDR. With this in mind, we have demonstrated the use of PROVALT, a computational tool designed to work in conjunction with Mascot results files to provide a method for organizing large proteomic datasets and calculating FDRs for protein identifications.

PROVALT assigns a PRO-FDR based on the distribution of proteins with minimum peptide coverage (c) identified in peptide ion score bins (B_s), information that can be determined for any proteomics project. However, special consideration should be given to the dataset size when applying this analysis because the statistical significance of the calculated error rates is diminished for small datasets. As has been the case with the use of PEP-FDRs (38), the calculation of PRO-FDRs as performed by PROVALT is best suited for large datasets exceeding several hundred protein identifications. Otherwise, Mascot is best used in its native form due to the fact that manual verification of questionable identifications is feasible. The utility of PROVALT is as a high throughput tool for large datasets because it assigns a measure of statistical significance to all identified proteins, thus diminishing the need for manual verification.

To use PROVALT, the user must determine the maximum degree of peptide coverage (C) to use in the calculation of FDRs. The concept of a FDR is based on the assumption that the null hypotheses (proteins identified in the random database) follow a quasi-normal distribution of minimal ion scores (S). Indeed, score distributions at increasing c may not necessarily look normal themselves, especially when the sample size is small. Thus, as the sample size (number of protein identifications) increases, for each c the score distribution will approach normality, and the FDR calculations will become more accurate. In practice, determining C prior to PROVALT analysis will be dependent on the distribution of minimal ion scores S in the normal and random protein assignments. Upon assignment of beginning c = C by the user, a version of Fig. 2 can be generated and visual inspection of the distribution will suggest whether the assumption of normality is reasonable. The user can increase or decrease the value of C until a statistically significant level of peptide coverage is determined for the normal database. It is sensible to assume that as the size of the dataset decreases, C will also decrease until the method degenerates into a SPM at C = 1. Accordingly, the native form of Mascot may be best suited for these situations.

Another approach, used in tools like Protein Prophet or PROT_PROBE, is to combine the conditional probabilities of individual peptide matches to calculate an overall probability for the protein identification (35–37). A protein identification is then considered significant if its individual probability exceeds a chosen threshold. The major difference between this method and that of PROVALT is that PROVALT calculates a PRO-FDR based on minimum ion score thresholds and peptide coverage but does not calculate individual probabilities for each protein identification. It is important to note that the FDR does not represent the probability that a feature is significant but rather the expected proportion of random matches resulting when proteins are identified using peptide score and coverage criteria. On the other hand, an individual protein probability does not represent the proportion of false discoveries that may result when a probability threshold is employed. Thus probability thresholds alone may not provide an assessment of overall significance for datasets that include a large number of protein identifications. Sadygov et al. (37) observed that protein probabilities always approach 1 when calculated through a binomial distribution of peptide probabilities that do not account for database or dataset size. This is crucial because as the size of the dataset increases the frequency of peptide matches to a given protein will increase regardless of the quality of the peptide match, resulting in a number of proteins being identified by a subset of random peptides. In contrast, the exploitation of a random database allows the calculation of FDRs regardless of database search parameters, database size, or dataset size. Another apparent benefit of this approach is that the entire distribution of random peptide matches is observed and factored into the final PRO-FDR calculation. As the number of random matches increases with dataset size, the significance thresholds necessary to achieve a specified PRO-FDR will also increase, thus compensating for the frequency of higher-scoring random matches. Ideally, however, one would prefer to have both protein probabilities and PRO-FDR estimations. This would allow researchers to select a set of proteins based on a minimum FDR but remove those that have an unacceptable individual protein probability, which is especially applicable for low peptide coverage members of high confidence protein groups.

In this work, the traditional methods of protein identification when using Mascot have been examined. Both the SPM and CPM strategies mentioned above propose the filtering of peptides in some manner prior to protein identification. We feel this is incorrect because such strategies ignore the phenomenon of peptide clustering and fail to assign any statistical significance to the thresholds used for the protein identifications. In addition, an accurate treatment of redundant protein and peptide identifications is crucial if a legitimate FDR is to be determined, and careful consideration must also be given to how peptides are partitioned among homologous proteins. The PROVALT algorithm, although designed to function in conjunction with Mascot, is independent of the scoring algorithm used for the peptide identification. This method reduces a complex list of peptide matches to a nonredundant list of proteins that have been identified at a user-specified FDR. Using peptide and protein identifications resulting from the proteomic analysis of T. cruzi epimastigotes, we compared the PROVALT method with the traditional methods of protein identification. The PROVALT method was shown to be superior to both methods by the number of proteins identified and in the inclusion of lower scoring nonrandom peptide matches.

All software used in this study to parse Mascot results files, cluster peptides to proteins, and select proteins for identification at a specified FDR is integrated into the software package PROVALT. This software will be made publicly available at kiwi.rcr.uga.edu/tcprot.

	ACKNOWLEDGMENTS

 
We thank Kumar Kolli for technical assistance with the MS analysis. We also thank Lance Wells and Carl Bergmann for critical reading of this manuscript.

	FOOTNOTES

 
Received, December 22, 2004, and in revised form, February 5, 2005.

Published, MCP Papers in Press, February 9, 2005, DOI 10.1074/mcp.M400215-MCP200

¹ The abbreviations used are: SPM, single peptide score method; PROVALT, protein validation technology; ion score, Mascot assigned peptide score, which represents the statistical significance of the peptide match; CPM, cumulative protein score method; FDR, false-discovery rate; PRO-FDR, protein false-discovery rate; PEP-FDR, peptide false-discovery rate; SCX, strong cation exchange chromatography; RP, reverse phase chromatography.

^* This work was funded in part with federal funds from National Institutes of Health Grants RO1 AI-033106 and PO1 AI-044979 (to R. L. T.) and P41 RR-018502 (to R. O.). The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked "advertisement" in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.

Both authors contributed equally to this work.

^|| To whom correspondence should be addressed: Complex Carbohydrate Research Center, University of Georgia, 315 Riverbend Rd., Athens, GA 30602-4712. Tel.: 706-542-4429; Fax: 706-542-4412; E-mail: orlando{at}ccrc.uga.edu

	REFERENCES

TOP ABSTRACT EXPERIMENTAL PROCEDURES RESULTS DISCUSSION REFERENCES

 

1. Rep, M., Dekker, H. L., Vossen, J. H., de Boer, A. D., Houterman, P. M., Speijer, D., Back, J. W., de Koster, C. G., and Cornelissen, B. J. (2002) Mass spectrometric identification of isoforms of PR proteins in xylem sap of fungus-infected tomato. Plant Physiol. 130, 904 –917[Abstract/Free Full Text]

2. Bendt, A. K., Burkovski, A., Schaffer, S., Bott, M., Farwick, M., and Hermann, T. (2003) Towards a phosphoproteome map of Corynebacterium glutamicum. Proteomics 3, 1637 –1646[CrossRef][Medline]

3. Zhang, H., Li, X. J., Martin, D. B., and Aeboersold, R. (2003) Identification and quantification of N-linked glycoproteins using hydrazide chemistry, stable isotope labeling and mass spectrometry. Nat. Biotechnol. 21, 660 –666[CrossRef][Medline]

4. Winters, M. S., and Day, R. A. (2003) Detecting protein-protein interactions in the intact cell of Bacillus subtilis. J. Bacteriol. 185, 4268 –4275[Abstract/Free Full Text]

5. Aebersold, R., and Mann, M. (2003) Mass spectrometry-based proteomics. Nature 422, 198 –207[CrossRef][Medline]

6. Taylor, J. A., and Johnson, R. S. (1997) Sequence database searches via de novo peptide sequencing by tandem mass spectrometry. Rapid. Commun. Mass Spectrom. 11, 1067 –1075[CrossRef][Medline]

7. Taylor, J. A., and Johnson, R. S. (2001) Implementation and uses of automated de novo peptide sequencing by tandem mass spectrometry. Anal. Chem. 73, 2594 –2604[Medline]

8. Johnson, R. S., and Taylor, J. A. (2002) Searching sequence databases via de novo peptide sequencing by tandem mass spectrometry. Mol. Biotechnol. 22, 301 –315[Medline]

9. Dancik, V., Addona, T. A., Clauser, K. R., Vath, J. E., and Pevzner, P. A. (1999) De novo peptide sequencing via tandem mass spectrometry. J. Comput. Biol. 6, 327 –342[CrossRef][Medline]

10. Chen, T., Kao, M. Y., Tepel, M., Rush, J., and Church, G. M. (2001) A dynamic programming approach to de novo peptide sequencing via tandem mass spectrometry. J. Comput. Biol. 8, 325 –337[CrossRef][Medline]

11. Sunyaev, S., Liska, A. J., Golod, A., Shevchenko, A., and Shevchenko, A. (2003) MultiTag: Multiple error-tolerant sequence tag search for the sequence-similarity identification of proteins by mass spectrometry. Anal. Chem. 75, 1307 –1315[Medline]

12. Mann, M., and Wilm, M. (1994) Error-tolerant identification of peptides in sequence databases by peptide sequence tags. Anal. Chem. 66, 4390 –4399[Medline]

13. Tabb, D. L., Saraf, A., and Yates, J. R., III (2003) GutenTag: High-throughput sequence tagging via an empirically derived fragmentation model. Anal. Chem. 75, 6415 –6421[Medline]

14. Perkins, D. N., Pappin, D. J., Creasy, D. M., and Cottrell, J. S. (1999) Probability-based protein identification by searching sequence databases using mass spectrometry. Electrophoresis 20, 3551 –3567[CrossRef][Medline]

15. Eng, J. K., McCormack, A. L., and Yates, J. R., III (1994) An approach to correlate tandem mass spectral data of peptides with amino acid sequences in a protein database. J. Am. Soc. Mass Spectrom 5, 976 –989[CrossRef]

16. Zhang, N., Aebersold, R., and Schqikowski, B. (2002) ProbID: A probabilistic algorithm to identify peptides through sequence database searching using tandem mass spectral data. Proteomics 2, 1406 –1412[CrossRef][Medline]

17. Colinge, J., Masselot, A., Giron, M., Dessingy, T., and Magnin, J. (2003) OLAV: Towards high-throughput tandem mass spectrometry data identification. Proteomics 3, 1454 –1463[CrossRef][Medline]

18. Bafna, V., and Edwards, N. (2001) SCOPE: A probabilistic model for scoring tandem mass spectra against a peptide database. Bioinformatics 17, S13 –S21[Abstract]

19. Sadygov, R. G., and Yates, J. R., III (2003) A hypergeometric probability model for protein identification and validation using tandem mass spectral data and protein sequence databases. Anal. Chem. 75, 3792 –3798[Medline]

20. Field, H. I., Fenyo, D., and Beavis, R. C. (2002) RADARS, a bioinformatics solution that automates proteome mass spectral analysis, optimizes protein identification, and archives data in a relational database. Proteomics 2, 36 –47[CrossRef][Medline]

21. Hernandez, P., Gras, R., Frey, J., and Appel, R. D. (2003) Popitam: Towards new heuristic strategies to improve protein identification from tandem mass spectrometry data. Proteomics 6, 870 –880

22. Pappin, D. J. C., Hojrup, P., and Bleasby, A. J. (1993) Rapid identification of proteins by peptide-mass fingerprinting. Curr. Biol. 3, 327 –332[CrossRef][Medline]

23. Mawuenyega, K. G., Kaji, H., Yamuchi, Y., Shinkawa, T., Saito, H., Taoka, M., Takahashi, N., and Isobe, T. (2003) Large scale identification of Caenorhabditis elegans proteins by multidimensional liquid chromatography-tandem mass spectrometry. J. Proteome. Res. 2, 23 –35[CrossRef][Medline]

24. Laukens, K., Deckers, P., Esmans, E., Van Onckelan, H., and Witters, E. (2004) Construction of a two-dimensional gel electrophoresis protein database for the Nicotiana tabacum cv. Bright Yellow-2 cell suspension culture. Proteomics 4, 720 –727[Medline]

25. O’Neil, K. A., Miller, F. R., Barder, T. J., and Lubman, D. M. (2003) Profiling the progression of cancer: Separation of microsomal proteins in MCF10 breast epithelial cell lines using nonporous chromatophoresis. Proteomics 3, 1256 –1269[CrossRef][Medline]

26. Lee, C. L., Hsiao, H. H., Lin, C. W., Wu, S. P., Huang, S. Y., Wu, C. Y., Wang, A. H., and Khoo, K. H. (2003) Strategic shotgun proteomics approach for efficient construction of an expression map of targeted protein families in hepatoma cell lines. Proteomics 3, 2472 –2486[CrossRef][Medline]

27. Lasonder, E., Ishihama, Y., Andersen, J. S., Vermunt, A. M., Pain, A., Sauerwein, R. W., Eling, W. H., Hall, N., Waters, A. P., Stunnenberg, H. G., and Mann, M. (2002) Analysis of the Plasmodium falciparum proteome by high-accuracy mass spectrometry. Nature 419, 537 –542[CrossRef][Medline]

28. Chamrad, D. C., Korting, G., Stuhler, K., Meyer, H. E., Klose, J., and Bluggel, M. (2004) Evaluation of algorithms for protein identification from sequence databases using mass spectrometry data. Proteomics 4, 619 –628[CrossRef][Medline]

29. Keller, A., Nesvizhskii, A. I., Kolker, E., and Aebersold, R. (2002) Empirical statistical model to estimate the accuracy of peptide identifications made by MS/MS and database search. Anal. Chem. 74, 5383 –5392[Medline]

30. Fenyo, D., and Beavis, R. C. (2003) A method for assessing the statistical significance of mass spectrometry-based protein identifications using general scoring schemes. Anal. Chem. 75, 768 –774[Medline]

31. Petritis, K., Kangas, L. J., Ferguson, P. L., Anderson, G. A., Pasa-Tolic, L., Lipton, M. S., Auberry, K. J., Strittmatter, E. F., Shen, Y., Zhao, R., and Smith, R. D. (2003) Use of artificial neural networks for the accurate prediction of peptide liquid chromatography elution times in proteome analysis. Anal. Chem. 75, 1039 –1048[Medline]

32. Cargile, B. J., Bundy, J. L., Freeman, T. W., and Stephenson, J. L., Jr. (2004) Gel based isoelectric focusing of peptides and the utility of isoelectric point in protein identification. J. Proteome. Res. 3, 112 –119[CrossRef][Medline]

33. Resing, K. A., Meyer-Arendt, K., Mendoza, A. M., Aveline-Wolf, L. D., Jonscher, K. R., Pierce, K. G., Old, W. M., Cheung, H. T., Russell, S., Wattawa, J. L., Goehle, G. R., Knight, R. D., and Ahn, N. G. (2004) Improving reproducibility and sensitivity in identifying human proteins by shotgun proteomics. Anal. Chem. 76, 3556 –3568[Medline]

34. Lin, D., Tabb, D. L., and Yates, J. R., III (2003) Large-scale protein identification using mass spectrometry. Biochim. Biophys. Acta 1646, 1 –10[Medline]

35. Nesvizhskii, A. I., Keller, A., Kolker, E., and Aebersold, R. (2003) A statistical model for identifying proteins by tandem mass spectrometry. Anal. Chem. 75, 4646 –4658[Medline]

36. Nesvizhskii, A. I., and Aebersold, R. (2004) Analysis, statistical validation and dissemination of large-scale proteomics datasets generated by tandem MS. Drug. Discov. Today 4, 173 –181

37. Sadygov, R. G., Liu, H., and Yates, J. R. (2004) Statistical models for protein validation using tandem mass spectral data and protein amino acid sequence databases. Anal. Chem. 76, 1664 –1671[Medline]

38. Peng, J., Elias, J. E., Thoreen, C. C., Licklider, L. J., and Gygi, S. P. (2003) Evaluation of multidimensional chromatography coupled with tandem mass spectrometry (LC/LC-MS/MS) for large-scale protein analysis: The yeast proteome. J. Proteome. Res. 2, 43 –50[CrossRef][Medline]

39. Kislinger, T., Rahman, K., Radulovic, D., Cox, B., Rossant, J., and Emili, A. (2003) PRISM, a generic large scale proteomic investigation strategy for mammals. Mol. Cell. Proteomics 2, 96 –106[Abstract/Free Full Text]

40. Benjamini, Y., and Hochberg, Y. (1995) Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. R. Stat. Soc. B. 64, 479 –498

41. Storey, J. D. (2003) The positive false discovery rate: A Bayesian interpretation and the q-value. Ann. Statist. 31, 2013 –2035[CrossRef]

42. Storey, J. D., and Tibshirani, R. (2003) Statistical significance for genomewide studies. Proc. Natl. Acad. Sci. U. S. A. 16, 9440 –9445

43. Rondinelli, E., Silva, R., Carvalho, J. F., de Almeida Soares, C. M., de Carvalho, E. F., and de Castro, F. T. (1988) Trypanosoma cruzi: An in vitro cycle of cell differentiation in axenic culture. Exp. Parasitol. 66, 197 –204[CrossRef][Medline]

44. Sonsmann, G., Romer, A., and Schomburg, D. J. (2002) Investigation of influence of charge derivatization on the fragmentation of multiply protonated peptides. J. Am. Soc. Mass Spectrom. 12, 47 –58

45. Rappsilber, J., and Mann, M. (2002) What does it mean to identify a protein in proteomics? Trends. Biochem. Sci. 27, 74 –78[CrossRef][Medline]

This article has been cited by other articles:

				R. Raaijmakers, W. Pluk, C. H. Schroder, J. Gloerich, E. A.M. Cornelissen, H. J.C.T. Wessels, J. L. Willems, L. A.H. Monnens, and L. P.W.J. van den Heuvel Proteomic profiling and identification in peritoneal fluid of children treated by peritoneal dialysis Nephrol. Dial. Transplant., April 19, 2008; (2008) gfn212v1. [Abstract] [Full Text] [PDF]

				P. D. Curtis, J. Atwood III, R. Orlando, and L. J. Shimkets Proteins Associated with the Myxococcus xanthus Extracellular Matrix J. Bacteriol., November 1, 2007; 189(21): 7634 - 7642. [Abstract] [Full Text] [PDF]

				W. Yang, R. L. Woltjer, I. Sokal, C. Pan, Y. Wang, M. Brodey, E. R. Peskind, J. B. Leverenz, J. Zhang, D. P. Perl, et al. Quantitative Proteomics Identifies Surfactant-Resistant {alpha}-Synuclein in Cerebral Cortex of Parkinsonism-Dementia Complex of Guam but Not Alzheimer's Disease or Progressive Supranuclear Palsy Am. J. Pathol., September 1, 2007; 171(3): 993 - 1002. [Abstract] [Full Text] [PDF]

				S. Sun, K. Meyer-Arendt, B. Eichelberger, R. Brown, C.-Y. Yen, W. M. Old, K. Pierce, K. J. Cios, N. G. Ahn, and K. A. Resing Improved Validation of Peptide MS/MS Assignments Using Spectral Intensity Prediction Mol. Cell. Proteomics, January 1, 2007; 6(1): 1 - 17. [Abstract] [Full Text] [PDF]