An overview about the API ecosystem and potential future trends

With this post I like to describe the current API ecosystem and show also potential future trends. My name is Christoph, I`am working for a long time in the API space (founder of Fusio an open source API management tool). This post should also summarize the history and current state of the API space so that new developers can build up on those ideas.

History

To start lets take a short look at the history of web services and how we are landed here. The web started with single HTML pages, then there was a need that also machines can talk to another, because of this XML was invented to exchange data between services. As we know now, this was not the best idea, since it is a markup language and not a data format. XML has its strength to describe data i.e. like for Word or ePUB documents but it is not great for exchanging data. I don`t want to go too deep into this topic but the main problem was/is, that XML is basically a huge tree and to access data inside this tree a developer must use the DOM which is not easy to work with. Because of this we have switched from XML to JSON. JSON was born as notation for Javascript but today almost every language can parse JSON structures into its own internal representation, so it is a more natural fit for exchanging data.

In the time there has evolved a great ecosystem around XML, so there was SOAP, which is basically an RPC system based on XML to invoke methods on a remote server. Then there was WSDL, which is like the current OpenAPI spec but to describe a SOAP service and there was XSD which is a schema to describe an XML structure, you can think of it like JsonSchema but for XML. All those XML specifications still exist and are in use but not so much for the API space anymore.

Present

After the switch from XML to JSON there was a need to create all those tools which are already available for XML. There was a time period where multiple standards competed like RAML, Blueprint and Swagger to describe an REST API. That means they described what endpoints and HTTP methods are available and also all headers, query parameters, request and response payload. In the end the OpenAPI initiative was founded to build a standard to describe a REST API. They selected the Swagger spec and developed based on this the OpenAPI specification which we know today.

Currently the most popular way to create a web service is by implementing a REST API which serves JSON. There are also other implementations like GraphQL or gRPC but in comparison they are less popular. Today almost every service on the web provides some sort of REST API to expose data or provide functionalities so that it can be called by other machines. Some of them have also an OpenAPI specification which describes the available endpoints.

Besides the OpenAPI specification there has also evolved the AsyncAPI specification which allows to describe event driven system i.e. if a service has a webhook mechanism, then you can use the AsyncAPI specification to describe the available events and payloads. The AsyncAPI spec is very similar to the OpenAPI spec and some parts are identical.

Issues

We have now looked at the past and described the current situation. Now lets take a look at the current problems in our API space:

Integration

One of the biggest problem of the current API space, and where many companies invest a lot of money in, is the topic of API integration. API integration means the process to integrate an external API into your app. Currently you can either integrate an API directly by working with the REST API manually, that means that you basically need to build a client in your target language which makes the actual HTTP calls. Sometimes the service provides already an SDK in specific languages which you can use, but this depends always on the service and mostly not all languages are supported. This process takes a lot of time and is error-prone so there is much room for improvement.

Specification

It is great that we have a standard specification to describe an API but unfortunately the OpenAPI spec has currently some flaws when it comes to generating code based on an OpenAPI specification. This is mainly because it uses at its core JsonSchema which is designed to validate JSON structures but not to model data which can be used for code generation. I have already wrote another post explaining the problems in detail, please feel free to check it out.

Versioning

It is inevitable that an API will change, since in most cases the service provider wants to extend or redesign the API functionality. Currently there are two popular solutions to versioning, first the provider adds the version to the route i.e. “/v1/my/service” or it uses the “Accept” header field including an MIME type with a specific version. Mostly a provider deprecates a specific endpoint, then all consumers have time to migrate to the new version and after this the initial implementation will stop working. So there is no standard and every provider has its own logic howto handle the deprecation process.

Authentication

For Authentication there is the OAuth2 specification which works great. Basically it describes different ways how you can obtain an Access-Token. The Access-Token can then be used to access protected parts of the API. The problems are currently the handling of such tokens, i.e. how to obtain and extend an Access-Token before it expires.

Future

To change the future we must formulate a vision how the API world could be. Regarding the problems listed above I would formulate the following vision:

In the future, if I want to integrate an API from an external provider, I only want to get the OpenAPI specification from this provider and generate based on this specification a client in my target language. A provider should not have the task to provide a Client-SDK since we can generate this based on the specification. The client automatically works with a specific version of the API and the client should never brake since the provider adds new functionality through a standardized process. If the API has changed and I regenerate my code, I will get information which endpoints are deprecated and how long I have time to migrate to a different method. The generated code is well integrated with the ecosystem of my target language. I will only have to provide my client key/secret and the client is able to automatically obtain an Access-Token and it also handles the extension of an Access-Token before it expires.

Solutions

Now lets take a look at possible solutions to build this future:

Integration/Specification

There are already man apps available which try to solve this problem by providing a single service which handles the integration. The problem with this approach is, that we don`t want to have a single service which handles this communication. From my view we need a decentralized approach to solve this problem at the specification level so that there is no need for such an app.

To enable this the OpenAPI specification needs to have solid schema language which can be used to generate high quality code (we have internally already developed a schema language called TypeSchema). Then it would be possible to generate actual useful code.

Every language community should have its own standard OpenAPI generator, the idea to have a single repository containing all languages is not optimal. Code generation for a specific language mostly requires language specific tools and packages so you dont want i.e. generate PHP code in Java. You want to have a Java generator which generates Java, and a PHP generator which generates PHP. The gRPC community has also realized this and moved away from its monolithic repository to an approach where every community maintains its own generator.

Versioning

There is already a standard on the way which describes HTTP deprecation headers how to communicate the deprecation of an endpoint. This is a great step but it wold be great if we could obtain this information directly from the specification so that we don`t need to call the actual endpoint to obtain this information. The OpenAPI specification contains also a deprecated property to indicate that an operation is deprecated, this is also great, but there are no more information how long it is possible to use this operation. So an idea would be to add additional meta information to the specification where the time-lines are described, i.e. by adding a deprecationDate and sunsetDate property.

Authentication

Regarding authentication there are complete apps build around the process to manage the client credentials of remote APIs. I think in an perfect world we should not have a need for such apps. This means my client knows automatically how to obtain and extend an Access-Token. Currently the OAuth2 specification describes the following flows:

• Authorization code
• Implicit
• Client credentials
• Resource owner password credentials.

The authorization code flow can only be used, if a user controls you app through a web browser, then the app can redirect the user to the service provider and the user can allow access for the app. For clients which want to work programmatically with the API this is not always possible, since not every app is a web app. For this case most service providers provide a way to generate an Access-Token which you can then use directly.

I think we should use the “Client credentials” flow for this case, that means the user provides the client key/secret in a config file and the client is able to automatically obtain an access token if its not available. Currently this is not a standard but I think it would be a good direction. Then we could also implement this logic in the client which we can automatically generate based on the OpenAPI specification, then a user has not to handle this authentication process, he only needs to provide the fitting credentials.

Summary

So I hope I could provide you an overview about the history and the current state of the API space. I have also tried to cover the most important problems and areas where we can improve.

The topic how to build an actual API was not covered, since I think there we have a great landscape of general frameworks, API gateway/management systems and no/low code platforms, so that any developer is able to setup an great REST API. We are also working in this space, if you like you can checkout Fusio which is an open source API management system.

Also please let me know if I have missed a specific problem or technology which could be relevant in this scope.