microservices call graph

We’ve had a microservice architecture in place and roughly knew the service inter-dependencies. This was not a problem, as long as the number of services was kept small, and the calls rather limited and well-defined. However, the way calls were made was almost fully transparent to the invoker, looked like any other call. This is good news when developing, but it turned out that people found this very handy and started calling services from all over the place, and before long we’d lost track of who called whom, from where, and who depended on whom. We could keep some record statically somewhere with this information whenever someone added a new inter-service dependency, but that would surely soon go out of sync. We could also use dynamic call tracing, but that would leave paths that almost never occurred; unit test coverage would also help, but then again coverage has always been lacking.

We wanted a solution whereby no matter what a developer did or did not, all cases of calls could be automatically be identified. On top of that we wanted to be able to visualize this information. So we turned to static compiled code analysis using Apache Commons BCEL. The idea was to use a Visitor to visit all generated classes of our services, and for each method found collect all the calls made to another microservice, with all necessary information about the call, including the caller’s call site, the method signature, the target service and the target method’s signature. This task was helper by the fact that each service’s server interface implementation, as well as the client api library for making calls to it, were code generated, and hence they all used uniform packaging and was easy to tell apart calls through the api to other services from other calls.

Luckily there’s some work already done on this which we’ve taken advantage of, and in particular by Matthieu Vergne on a dynamic call manager to retrieve information about dynamic calls statically and by George Gousios on extracting call graph information. So we’ve worked on top of that, fine-tuned and also provided custom exporters, such as graphviz or tsv, as well adding facilities for slicing and dicing our code as needed, for example generating graphs answering questions such as:

• who’s calling this api?
• what is this service calling?
• what are all the calls made amongst this subset of services?

Here’s an example of how the calls between the travel and user service look like when using a graphviz exporter using an X11 randomized colour palette (the calls are simplified to make the point here):

java -jar services-callgraph.jar --services travel,user --exporter=dot --colours=x11

digraph services {
  graph [fontname = "Handlee"];
  node [fontname = "Handlee"];
  edge [fontname = "Handlee"];

  rankdir=LR;
  bgcolor=white;  // common for both svg and x11 schemes
  colorscheme=svg

  subgraph cluster_travelservice {
    style=rounded
    color=lightgrey;
    node [style=filled];
    fontsize = 20;

    // Callers
    "travelservice.module.travel.service.TravelServiceImpl" [label="TravelServiceImpl", color=aquamarine2]
    "travelservice.cache.CacheLoaders" [label="CacheLoaders", color=aquamarine2]

    // Apis
    label = "travelservice";
  }
  subgraph cluster_userservice {
    style=rounded
    color=lightgrey;
    node [style=filled];
    fontsize = 20;

    // Callers

    // Apis
    "apiclient.userservice.api.UserApi" [label="UserApi", color=gold]

    label = "userservice";
  }


  "travelservice.cache.CacheLoaders" -> "apiclient.userservice.api.UserApi" [label = "isActiveUser()", color="dimgrey"]
  "travelservice.module.travel.service.TravelServiceImpl" -> "apiclient.userservice.api.UserApi" [label = "hasRight()", color="palegreen"]
  "travelservice.module.travel.service.TravelServiceImpl" -> "apiclient.userservice.api.UserApi" [label = "getUserJob()", color="blueviolet"]
  "travelservice.cache.CacheLoaders" -> "apiclient.userservice.api.UserApi" [label = "searchByUserLogin()", color="steelblue"]
  "travelservice.module.travel.service.TravelServiceImpl" -> "apiclient.userservice.api.UserApi" [label = "getUserEntity()", color="lightcyan"]
  "travelservice.module.travel.service.TravelServiceImpl" -> "apiclient.userservice.api.UserApi" [label = "searchByUserId()", color="springgreen"]
}

If we add more services in the mix the graph reveals all the hidden calls that might need to re-structured:

We’ve opted to show a subset of essential attributes of the calls in the graphs to be more legible. One can always get the complete information by choosing an alternative exporter.