How the jsii open source framework meets developers where they are

A central part of the value proposition that the AWS Cloud Development Kit (AWS CDK) is set to deliver is the ability for developers to express their infrastructure requirements in the programming languages they are most comfortable with. The DevOps movement has blurred the line between application code and infrastructure definition, and it is only natural that DevOps engineers would want to write both in the same language.

In this article, I will explain how this proposition directed our approach to meeting developers where they are and led to the creation of jsii. I will also discuss how we interact with the community at large and recognize their contributions to the projects.

From our experience using the AWS SDKs, it was immediately clear that having to manually write the AWS CDK in each desired language was not going to scale. Writing high-level abstractions requires carefully designing APIs, listening for user feedback, and improving the APIs over time, while striving to maintain backward compatibility as much as possible. Starting from scratch, we had thousands of AWS service features to support, and bridging that gap for just one programming language would be a titanic task.

To solve this problem, we designed a strategy that would allow us to write our AWS CDK Construct libraries in a single language (TypeScript), and generate bindings for other languages automatically. This idea became jsii, an open source framework that allows JavaScript classes (authored in TypeScript) to be re-used from a variety of other programming languages (e.g., .NET, Java, Python, etc.).

The jsii compiler is a modified TypeScript compiler, which extracts class library APIs from the compiled modules and produces a language-independent representation of the API. That representation is then used by jsii-pacmak (the package maker) to generate library bindings in the supported languages. Those libraries are backed by the original JavaScript library, which is loaded into a helper node process, ensuring consistent behavior across all supported languages.

Celebrating the community

The AWS CDK would not be where it is right now without the help of the community at large. The core maintainer team can simply not work through the available AWS features fast enough to completely close the gap, and community-issued pull requests have been extremely useful in moving the project forward.

On the other hand, jsii has received fewer contributions in the form of code. Although we are trying to make the project more approachable through improved documentation and better abstractions, at the core, effective contributions to jsii will always require enough proficiency in multiple different programming languages to represent class library APIs in the most natural way possible.

The few languages currently supported by jsii have been developed with the help from people external to the AWS CDK core maintainers team. The teams owning the AWS SDKs for .NET, Java, and Python have helped us ensure that our APIs in those languages are at the right bar, and they have helped us evaluate the trade-offs that we had to make in order for this approach to work. More recently, JetBrains engineers have been working toward bringing “native” Kotlin support to jsii, and the Amazon Elastic Container Service (Amazon ECS) team is helping us deliver support for Go.

Code, however, is not the only way people can contribute to an open source project, and jsii has adopted the All Contributors standard as the way to recognize code contributions as well as bug reports, feature requests, talks and articles, project maintenance, documentation, tooling, tutorials, and more.

Beyond the AWS CDK

Although we could have developed jsii as an implementation detail of the CDK, we decided to treat it as a separate product right from the start. Our vision is that jsii should be usable by any project that can benefit from authoring a class library once and then allowing it to be re-used from many different programming languages. We believe that no language is better (or worse) than another, as their purpose is to allow developers to represent their ideas accurately.

For example, the Apache Spark project has interfaces for Scala, Java, Python, etc.; however, they are all maintained separately, and as a result there historically have been feature discrepancies between the various languages. A tool like jsii would enable such projects to reduce the effort needed to maintain interfaces for any supported programming language. And, adding more programming languages would not require any significant effort from the maintainers, as this task would be largely offloaded to jsii itself.

Of course, this goal is part of a longer-term vision, because several limitations in the jsii project that would need to be addressed before it could be used as a general-purpose tool within any context. For example, the reliance on an external node runtime adds friction in leveraging jsii-generated libraries in certain contexts, such as AWS Lambda, as users must bring in a layer with node when attempting to use a non-JavaScript runtime. This also introduces a performance overhead, related to data marshaling for inter-process communication, that can be deal-breaking for high-throughput applications.

Another caveat is that jsii runtimes currently leak memory. Two distinct processes (each with their own garbage collector) may hold references to the same object, and the mechanism to support coordinating garbage collection across the two processes does not exist yet.

Looking forward

The obvious future development for jsii is the addition of more programming languages. AWS customers should be able to expect the CDK to be available in at least those languages for which AWS publishes an SDK. In fact, there are several feature requests open on the jsii repository tagged with language-request that help us track customer interest, and GitHub ? reactions are accounted for in our prioritization framework.

We also want to lower the barrier to entry for bringing new programming languages to the jsii ecosystem. By creating the right abstractions, enthusiasts would be able to write plug-ins for the jsii toolchain that bring support for their favorite programming language without having to wait for AWS to support those languages.

To ensure APIs can be represented correctly in other languages, jsii restricts usage of some TypeScript features (type aliases, function parameters, etc.). We want to enable use of as much of TypeScript as possible and, thus, only restrict usage of features that cannot possibly be represented in other languages.

If you are interested in the project and want to know more, please refer to the GitHub repository and documentation. All contributions are welcome, so don’t hesitate to submit feature requests, up-vote or comment on existing issues, or get into the codebase and submit a pull request—even if only to fix a typo in the documentation.