Methodologies to infer differential kinase activities from global phosphoproteomics datasets have provided important insight into kinase functions in health and disease, and have catalysed the discovery of new kinase-substrate associations, the description of drug mechanisms of action and many other discoveries. However, these methods rely on previously annotated kinase-substrate associations. Given that the vast majority of identified phosphosites have no known upstream kinase and that approximately a third of human kinases have none or less than 5 known substrates, to date, these methods have been blind to large parts of the human signalling networks. With the recent availability of unbiased kinase substrate specificity maps for all human kinases, we are presented with the exciting opportunity to study kinase activities in a purely data-driven way. However, robust and appropriately benchmarked methods to take advantage of these kinase specificity maps for kinase activity inference are still lacking. Here we present such a method, called PhosX. Our method performs better than the best method that was based on known kinase-substrate associations with the added benefit that it can infer activities for all kinases. PhosX also performs better than the only other method to date that can use these kinase specificity maps. The addition of machine-learned kinase-substrate relationships and the integration of graph-based algorithms is currently ongoing to further improve PhosX's differential kinase activity inference. We expect PhosX to become an important tool in the data-driven studies of human cell signalling, allowing discoveries relating to context-specific functions of well-studied kinases but also shedding light on the vast understudied signalling space.