Advances in mass spectrometry-based proteomics have yielded a substantial mapping of the tyrosine phosphoproteome and thus provided an important step towards a systematic analysis of intracellular signaling networks in higher eukaryotes. In this study we decompose an uncharacterized proteomic data set of 481 unique phosphotyrosine (pY)peptides by sequence similarity to known ligands of the Src Homology 2 (SH2) and the phosphotyrosine binding (PTB) domains. From 20 clusters we extract 16 known and four new interaction motifs. Using quantitative mass spectrometry we pull down pY-specific binding partners for peptides corresponding to the extracted motifs. We confirm numerous of previously known interaction motifs and find 15 new interactions mediated by phosphosites not previously known to bind SH2 or PTB domains. Remarkably, a novel hydrophobic N-terminal motif ([LVI]-[LVI]-pY) is identified and validated as a binding motif for the SH2 domain containing inositol phosphatase SHIP2. Our decomposition of the in vivo pY proteome furthermore suggests that two thirds of the pY sites mediate interaction while the remaining third governs processes such as enzyme activation and nucleic acid binding.