The untold story of Elastic Metal

The untold story of Elastic Metal Build • Erwan Paillard • 19/07/23 • 8 min read

The untold story of Elastic Metal began in January 2018 at Scaleway, when our team of six devoted individuals embarked on an ambitious mission: position Scaleway at the forefront of the industry of the Cloud.

The Dedibox stack, which had served faithfully for 13 years, was starting to show its limits compared to the advanced solutions available in the market. We realized that it was isolated from Scaleway's innovative cloud products that we were actively developing. Driven by the desire of innovating, we worked to create Elastic Metal—a groundbreaking solution that would redefine the boundaries of Bare Metal hosting. We knew that our success relied on quick action and innovation.

Thus, Elastic Metal emerged as the answer, enabling us to overcome obstacles and seize new opportunities in the ever-evolving Bare Metal landscape. We knew our competitors were already on their way, so we had to move quickly.

January 2018: Picking the battle plan

We dedicated several days to exploring various solutions, ultimately leading us to two promising options.

The first solution revolved around prioritizing speedy delivery. To achieve this, we proposed the creation of a new service within Scaleway's infrastructure, responsible for seamlessly managing all the dependencies of our information system (IS). This encompassed trust management, quotas, CLI integration, and Terraform implementation. Additionally, we planned to establish a connection with the Dedibox backend to oversee server-related tasks such as ordering, maintenance, and installation. However, we were mindful of certain drawbacks associated with this approach, including the absence of worldwide services, limited isolation by availability zones (AZ), lengthy installation times, a distinct network infrastructure compared to Scaleway's, the lack of cloudinit support, fixed features, and no future planned enhancements.

The second option was more comprehensive and ambitious. It involved integrating the service from the first solution into Scaleway's ecosystem and developing an entire backend system for server management. While this approach offered greater control and customization possibilities, it entailed a significant investment in development resources to accommodate the new network topology, implementing a fresh installation stack, and handling other associated challenges.

In view of the workload and the delays required by the second solution, we started with the first solution, with the aim of gradually migrating to the second.

Now we had to choose the most suitable language. We had no particular technical constraints except having to connect to the Scaleway gateway which transforms all external sources: CLI, API and Terraform into an internal gRPC call. After a first study to see if it was feasible in PHP / Symfony, we realized that as it stands, it would be very complicated to implement the gRPC server. We left for Go in pairs, while the second PHP / Symfony pair would take care of Dedibox and help us by creating the few new API routes essential to the new product.

June 2018: From beta to product launch

Within a span of just six months, we developed a groundbreaking service known as BMaaS (Bare Metal As A Service). This offering leverages Scaleway's robust authentication system, quota management, and Trust infrastructure. To establish a connection with the Dedibox component, we adopted a simple approach: each account within the new service was linked to a corresponding "phantom" account in the Dedibox IS. To facilitate this integration, we introduced a new admin endpoint in Dedibox. This endpoint allowed us to retrieve existing Dedibox accounts associated with Scaleway accounts, or create new ones when necessary. For the operational aspects, BMaaS interacts with the Dedibox API through conventional means. This includes executing server commands, managing installations, and enabling remote access. By utilizing the established Dedibox API, we ensured compatibility and seamless functionality while delivering an enhanced experience through BMaaS.

Bare Metal As A Service.webp

One particularity of these new accounts was that they should not be billed by Dedibox, as Scaleway assumed responsibility for them. Additionally, their quotas were intended to be unlimited, as they were managed within Scaleway's new service. To avoid any potential vulnerabilities, safeguards were implemented. These included restrictions such as accounts being unable to directly access the Dedibox console, with only API access being permitted. Consumption of BMaaS within Dedibox was monitored to ensure alignment between the two systems.

The initial version of the service progressed to a private beta phase, catering to a select group of clients. This initial release, referred to as v1, allowed users to consume Bare Metal on an hourly basis. It supported the installation of three operating systems (Ubuntu, Debian, and CentOS) with predefined partitioning, as well as booting from a rescue image. Continual iterations followed, introducing features such as remote access management, including KVM for custom installations, reverse management for both IPv4 and IPv6, and server event listing. This progress led to the opening of the product for public beta testing from July to October 2018.

Further enhancements were made to support IP Failover (IPFO), enabling the ordering of additional IPv4 addresses that could be assigned and moved between machines.

February 2019: integrating the services

Efforts were directed towards preparing for the introduction of new offerings to create a more comprehensive catalog. Within these new offerings, two distinct configurations were specifically designed to deliver the best possible performance available in the market. These configurations can now be found in the Titanium range.

The tooling team took charge of integrating our service into Scaleway's CLI and Terraform, thereby completing the cloud integration of BMaaS.

January 2020: making the installation autonomous

It was time to address the next phase of our initial objective: introducing a new backend stack to manage Bare Metal servers and modernize the installation process. We began exploring various server installation solutions as proof of concepts to replace the Dedibox backend and bring additional features and speed to our installation system.

We…