This post was authored by Scott Konersmann, Partner Engineering Manager, SQL Server, Slava Oks, Partner Group Engineering Manager, SQL Server, and Tobias Ternstrom, Principal Program Manager, SQL Server.
Introduction
We first announced SQL Server on Linux in March, and recently released the first public preview of SQL Server on Linux (SQL Server v.Next CTP1) at the Microsoft Connect(); conference. We’ve been pleased to see the positive reaction from our customers and the community; in the two weeks following the release, there were more than 21,000 downloads of the preview. A lot of you are curious to hear more about how we made SQL Server run on Linux (and some of you have already figured out and posted interesting articles about part of the story with “Drawbridge”). We decided to kick off a blog series to share technical details about this very topic starting with an introduction to the journey of offering SQL Server on Linux. Hopefully you will find it as interesting as we do! J
Summary
Making SQL Server run on Linux involves introducing what is known as a Platform Abstraction Layer (“PAL”) into SQL Server. This layer is used to align all operating system or platform specific code in one place and allow the rest of the codebase to stay operating system agnostic. Because of SQL Server’s long history on a single operating system, Windows, it never needed a PAL. In fact, the SQL Server database engine codebase has many references to libraries that are popular on Windows to provide various functionality. In bringing SQL Server to Linux, we set strict requirements for ourselves to bring the full functional, performance, and scale value of the SQL Server RDBMS to Linux. This includes the ability for an application that works great on SQL Server on Windows to work equally great against SQL Server on Linux. Given these requirements and the fact that the existing SQL Server OS dependencies would make it very hard to provide a highly capable version of SQL Server outside of Windows in reasonable time it was decided to marry parts of the Microsoft Research (MSR) project Drawbridge with SQL Server’s existing platform layer SQL Server Operating System (SOS) to create what we call the SQLPAL. The Drawbridge project provided an abstraction between the underlying operating system and the application for the purposes of secure containers and SOS provided robust memory management, thread scheduling, and IO services. Creating SQLPAL enabled the existing Windows dependencies to be used on Linux with the help of parts of the Drawbridge design focused on OS abstraction while leaving the key OS services to SOS. We are also changing the SQL Server database engine code to by-pass the Windows libraries and call directly into SQLPAL for resource intensive functionality.
Requirements for supporting Linux
SQL Server is Microsoft’s flagship database product which with close to 30 years of development behind it. At a high level, the list below represents our requirements as we designed the solution to make the SQL Server RDBMS available on multiple platforms:
- Quality and security must meet the same high bar we set for SQL Server on Windows
- Provide the same value, both in terms of functionality, performance, and scale
- Application compatibility between SQL Server on Windows and Linux
- Enable a continued fast pace of innovation in the SQL Server code base and make sure new features and fixes appear immediately across platforms
- Put in place a foundation for future SQL Server suite services (such as Integration Services) to come to Linux
To make SQL Server support multiple platforms, the engineering task is essentially to remove or abstract away its dependencies on Windows. As you can imagine, after decades of development against a single operating system, there are plenty of OS-specific dependencies across the code base. In addition, the code base is huge. There are tens of millions of lines of code in SQL Server.
SQL Server depends on various libraries and their functions and semantics commonly used in Windows development that fall into three categories:
- “Win32” (ex. user32.dll)
- NT Kernel (ntdll.dll)
- Windows application libraries (such as MSXML)
You can think of these as core library functions, most of them have nothing to do with the operating system kernel and only execute in user mode.
While SQL Server has dependencies on both Win32 and the Windows kernel, the most complex dependency is that of Windows application libraries that have been added over the years in order to provide new functionality. Here are some examples:
- SQL Server’s XML support uses MSXML which is used to parse and process XML documents within SQL Server.
- SQLCLR hosts the Common Language Runtime (CLR) for both system types as well as user defined types and CLR stored procedures.
- SQL Server has some components written in COM like the VDI interface for backups.
- Heterogeneous distributed transactions are controlled through Microsoft Distributed Transaction Coordinator (MS DTC)
- SQL Server Agent integrates with many Windows subsystems (shell execution, Windows Event Log, SMTP Mail, etc.).
These dependencies are the biggest challenge for us to overcome to meet our goals of bringing the same value and having a very high level compatibility between SQL Server on Windows and Linux. As an example, to re-implement something like SQLXML would take a significant amount of time and would run a high risk of not providing the same semantics as before, and could potentially break applications. The option of completely removing these dependencies would mean we must also remove the functionality they provide from SQL Server on Linux. If the dependencies were edge cases and only impacting very few customer visible features, we could have considered it. As it turns out, removing them would cause us to have to remove tons of features from SQL Server on Linux which would go against our goals around compatibility and value across operating systems.
We could take the approach of doing this re-implementation piecemeal, bringing value little by little. While this would be possible, it would also go against the requirements because it would mean that there would be a significant gap between SQL Server on Linux and Windows for years. The resolution lies in the right platform abstraction layer.
Building a PAL
Software that is supported across multiple operating systems always has an implementation of some sort of Platform Abstraction Layer (PAL). The PAL layer is responsible for abstraction of the calls and semantics of the underlying operating system and its libraries from the software itself. The next couple of sections consider some of the technology that we investigated as solutions to building a PAL for SQL Server.
SQL Operating System (SOS or SQLOS)
In the SQL Server 2005 release, a platform layer was created between the SQL Server engine and Windows called the SQL Operating System (SOS). This layer was responsible for user mode thread scheduling, memory management, and synchronization (see SQLOS for reference). A key reason for the creation of SOS was that it allowed for a centralized set of low level management and diagnostics functionality to be provided to customers and support (subset of Dynamic Management Views/DMVs and Extended Events/XEvents). This layer allowed us to minimize the number of system calls involved in scheduling execution by running non-preemptively and letting SQL Server do its own resource management. While SOS improved performance and greatly helped supportability and debugging, it did not provide a proper abstraction layer from the OS dependencies described above, i.e. Windows semantics were carried through SOS and exposed to the database engine.
In the scenario where we would completely remove the dependencies on the underlying operating system from the database engine, the best option was to grow SOS into a proper Platform Abstraction Layer (PAL). All the calls to Windows APIs would be routed through a new set of equivalent APIs in SOS and a new host extension layer would be added on the bottom of SOS that would interact with the operating system. While this would resolve the system call dependencies, it would not help with the dependencies on the higher-level libraries.
Drawbridge
Drawbridge was an Microsoft Research project (see Drawbridge for reference) that focused on drastically reducing the virtualization resource overhead incurred when hosting many Virtual Machines on the same hardware. The research involved two ideas. The first idea was a “picoprocess” which consists of an empty address space, a monitor process that interacts with the host operating system on behalf of the picoprocess, and a kernel driver that allows a driver to populate the address space at startup and implements a host Application Binary Interface (ABI) that allows the picoprocess to interact with the host. The second idea was a user mode Library OS, sometimes referred to as LibOS. Drawbridge provided a working Windows Library OS that could be used to run Windows programs on a Windows host. This Library OS implements a subset of the 1500+ Win32 and NT ABIs and stubs the rest to either succeed or fail depending on the type of call.
Our needs didn’t align with the original goals of the Drawbridge research. For instance, the picoprocess idea isn’t something needed for moving SQL Server to other platforms. However, there were a couple of synergies that stood out:
- Library OS implemented most of the 1500+ Windows ABIs in user mode and only 45-50 ABIs were needed to interact with the host. These ABIs were for address space and memory management, host synchronization, and IO (network and disk). This made for a very small surface area that needs to be implemented to interact with a host. That is extremely attractive from a platform abstraction perspective.
- Library OS was capable of hosting other Windows components. Enough of the Win32 and NT layers were implemented to host CLR, MSXML, and other APIs that the SQL suite depends on. This meant that we could get more functionality to work without rewriting whole features.
There were also some risk and reward tradeoffs:
- The Microsoft Research project was complete and there was no support for Drawbridge. Therefore, we needed to take a source snapshot and modify the code for our purposes. The risks were around the costs to ramp up a team on the Library OS, modify it to be suitable for SQL Server, and make it perform comparably with Windows. On the positive side, this would mean everything is in user mode and we would own all the code within the stack. Performance critical code can be optimized because we can modify all layers of the stack including SQL Server, the Library OS, and the host interface as needed to make SQL Server perform. Since there are no real boundaries in the process, it is possible for SQL Server to call Linux.
- The original Drawbridge project was built on Windows and used a kernel driver and monitor process. This would need to be dropped in favor of a user mode only architecture. In the new architecture, the host extension (referred to as PAL in the Drawbridge design) on Windows would move from a kernel driver to just a user mode program. Interestingly enough, one of the researchers had developed a rough prototype for Linux that proved it could be done.
- Because the technologies were created independently there was a large amount of overlapping functionality. SOS had subsystems for object management, memory management, threading/scheduling, synchronization, and IO (disk and network). The Library OS and Host Extension also had similar functionality. These systems would need to be rationalized down to a single implementation.
| Technologies |
SOS |
Library OS |
Host Extension |
|
Object Management |
|
|
|
|
Memory Management |
|
|
|
|
Threading/Scheduling |
|
|
|
|
Synchronization |
|
|
|
|
I/O (Disk, Network) |
|
|
|
Meet SQLPAL
As a result of the investigation, we decided on a hybrid strategy. We would merge SOS and Library OS from Drawbridge to create the SQL PAL (SQL Platform Abstraction Layer). For areas of Library OS that SQL Server does not need, we would remove them. To merge these architectures, changes were needed in all layers of the stack.
The new architecture consists of a set of SOS direct APIs which don’t go through any Win32 or NT syscalls. For code without SOS direct APIs they will either go through a hosted Windows API (like MSXML) or NTUM (NT User Mode API – this is the 1500+ Win32 and NT syscalls). All the subsystems like storage, network, or resource management will be based on SOS and will be shared between SOS direct and NTUM APIs.
This architecture provides some interesting characteristics:
- Everything running in process boils down to the same platform assembly code. The CPU can’t tell the difference between the code that is providing Win32 functionality to SQL Server or native Linux code.
- Even though the architecture shows layering, there are no real boundaries within the process (There is no spoon!). If code running in SQL Server which is performance critical needs to call Linux it can do that directly with a very small amount of assembler via the SOS direct APIs to setup the stack correctly and process the result. An example where this has been done is the disk IO path. There is a small amount of conversion code left to convert from Windows scatter/gather input structure to Linux vectored IO structure. Other disk IO types don’t require any conversions or allocations.
- All resources in the process can be managed by SQLPAL. In SQL Server, before SQLPAL, most resources such as memory and threads were controlled, but there were some things outside it’s control. Some libraries and Win32/NT APIs would create threads on their own and do memory allocations without using the SOS APIs. With this new architecture, even the Win32 and NT APIs would be based on SQLPAL so every memory allocation and thread would be controlled by SQL PAL. As you can see this also benefits SQL Server on Windows.
- For SQL Server on Linux we are using about 81 MB of uncompressed Windows libraries, so it’s a tiny fraction (less than 1%) of a typical Windows installation. SQLPAL itself is currently around 8 MB.
Process Model
The following diagram shows what the address space looks like when running. The host extension is simply a native Linux application. When host extension starts it loads and initializes SQLPAL, SQLPAL then brings up SQL Server. SQLPAL can launch software isolated processes that are simply a collection of threads and allocations running within the same address space. We use that for things like SQLDumper which is an application that is run when SQL Server encounters a problem to collect an enlightened crash dump.
One point to reiterate is that even though this might look like a lot of layers there aren’t any hard boundaries between SQL Server and the host.
Evolution of SQLPAL
At the start of the project, SQL Server was built on SOS and Library OS was independent. The eventual goal is to have a merged SOS and Library OS as the core of SQL PAL. For public preview, this merge wasn’t fully completed, but the heart of SQLPAL had been replaced with SOS. For example, threads and memory already use SOS functionality instead of the original Drawbridge implementations.
The result is that there are two instances of SOS running inside the CTP1 release: one in SQL Server and one in SQLPAL . This works fine because the SOS instance in SQL Server is still using Win32 APIs which call down into the SQLPAL. The SQLPAL instance of the SOS code has been changed to call the host extension ABIs (i.e. the native Linux code) instead of Win32.
Now we are working on removing the SOS instance from SQL Server. We are exposing the SOS APIs from the SQLPAL. Once this is completed everything will flow through the single SQLPAL SOS instance.
More posts
We are planning more of these posts to share to tell you about our journey, which we believe has been amazing and a ton of fun worth sharing. Please provide comments if there are specific areas you are interested in us covering!
Thanks!