Open Distro for Elasticsearch: What it Means and Why it's Important

Recently Amazon launched Open Distro for Elasticsearch, a distribution of Elasticsearch with a number of additional features. The project was created out of concern that Elasticsearch was starting to include proprietary features, and that Elastic was straying from its open source roots. Open Distro for Elasticsearch contains a significant set of additions including:

• Node-to-node encryption
• User authentication, including single sign on (SSO)
• Role-based access control (RBAC)
• Audit logging
• Support for SQL searches
• Cluster performance analysis tools

Why is Amazon Releasing an Elasticsearch Distribution?

Amazon's main motivation for releasing Open Distro for Elasticsearch is a matter of license. Elasticsearch is an open source product that contains proprietary components. These features are entirely optional and are only made available after activating a license through the Kibana web UI.

The problem, according to Amazon, is that there is a blurred line between these proprietary components and the open source components. This makes it difficult for developers to know whether the code they're contributing to Elasticsearch improves the open source product or a proprietary component. Amazon also believes that Elastic is focusing their development efforts on their proprietary solutions and spending less effort on open source.

Adrian Cockcroft, VP of Cloud Architecture Strategy at AWS, says that "the innovation focus has shifted from furthering the open source [Elasticsearch] distribution to making the proprietary distribution popular. This means the majority of new Elasticsearch users are now, in fact, running proprietary software." The post continues, "[Elastic's direction] has created uncertainty about the longevity of the open source project as it is getting less innovation focus."

Elastic responded by stating their commitment to open source and how they've openly communicated the use of proprietary software in Elasticsearch. They specifically called out Amazon for "[asking] for special working relationship[s] in order to collaborate on code, demanding preferential treatment that would place them above our users." Considering Amazon hosts a widely used managed Elasticsearch service, it's easy to see why they would want to play a bigger role in Elasticsearch's development.

What Does this Mean for Elasticsearch?

With Amazon pushing its own open source Elasticsearch toolkit, the question is: how will this impact the development of Elasticsearch? Right now, it's difficult to say. Many of the features Amazon provides in their distribution implement features found in Elastic's commercial offerings. If Open Distro for Elasticsearch draws customers away from Elasticsearch's proprietary offerings, it could jeopardize not only Elastic's control over Elasticsearch's development, but also their ability to monetize it.

This also risks fragmenting the Elasticsearch community, especially if Amazon decides to modify Elasticsearch itself. Amazon stated their commitment to open sourcing their tools and pushing changes upstream, but if those changes aren't accepted by Elastic, it could lead to a fork. Amazon has stated that Open Distro for Elasticsearch is not a fork, but they're effectively putting pressure on Elastic to accept these additions despite conflicting interests.
Here at LogDNA, we're very familiar with Elasticsearch and its role in the log management industry. We're interested in seeing what Amazon has planned for their Elasticsearch distribution, and how it might help the Elasticsearch community grow. We'll keep an eye out for any new developments.