0%
Database Internals: Build the Engine
Storage Structures
Durability
Concurrency Control
Query Processing
Access Paths
SQLite and Embedded Engines
The two engines in the previous lessons are servers. This one is a library, and almost every difference between them follows from that single structural fact rather than from any difference in ambition.
What an embedded engine is
A library linked into the application, so a query is a function call inside the application's own process rather than a request sent to another program.
Defining the model first is right, because every consequence in this lesson derives from it. There is no server to connect to because there is no server.
Where the database lives
In an ordinary file, or a small number of files, which the application opens directly with the same permissions it has for anything else.
Stating the storage arrangement is necessary. The database is a file rather than a service with a file behind it.
What disappears when the server does
The network round trip, the connection handshake, the authentication exchange, the connection pool and the separate process to supervise and restart.
Listing what goes away is valuable, because it is a long list and it explains the appeal. Several entire categories of operational concern are simply absent.
Why the round trip matters more than it sounds
A query that takes microseconds of work pays tens or hundreds of microseconds in a server architecture just to be asked, so for small queries the communication dominates the computation entirely.
Quantifying the saving is important, since it is the strongest argument for the model. The library is not faster at the hard work, it is faster at the easy work.
Why that makes the model attractive for small queries
An application doing many tiny reads gets most of its time back, which is why this arrangement is preferred for local data access even by applications that could run a server.
Drawing the practical conclusion is worthwhile. The choice is a performance decision and not only a packaging one.
What appears in exchange
The application is now the database process, so a crash of the application is a crash of the database, and a memory error anywhere in the application can corrupt database state.
Being honest about the cost is necessary. The isolation a separate process provides is genuinely lost.
Why that shared fate is the central trade
A server protects the database from the application's bugs and the application from the database's, and an embedded engine removes that boundary in both directions.
Naming the trade explicitly is the section's most important content. The boundary was doing real work.
How concurrency works without a server
Several processes may open the same file, and they coordinate through locks the filesystem provides rather than through a lock manager inside a shared process.
Explaining the coordination is essential. The arbiter is the filesystem rather than the engine.
Why that coordination is weaker
Filesystem locking is coarser than a lock manager, it varies between platforms, and it has no way to make a process that has died release what it held promptly.
Giving the limitation is important, because it is the root of the characteristic failures later in the lesson. The mechanism is less capable than its server equivalent.
Why network filesystems break this badly
Their locking is frequently incomplete or advisory, and the engine depends on it being correct, so a database file on a network share risks corruption rather than merely poor performance.
Giving the hard prohibition is necessary, and it is the single most important operational rule in the lesson. The failure is silent and it destroys data.
Why that rule is broken so often
A shared drive looks like the obvious way to let two machines use one database, and it fails in a way that appears intermittent rather than structural.
Explaining why the mistake recurs is valuable. The wrong answer is the intuitive one.
What the access control is
File permissions, since there is no server to authenticate against and no notion of database users with their own credentials.
Covering the trust model is necessary. The security boundary moves to the filesystem.
Why that is sometimes exactly right
For data belonging to one user on one device, the operating system's own permissions are the correct boundary, and a separate authentication system would add nothing.
Being fair to the model is right. The simplification is a genuine fit for a large class of uses.
Why it is sometimes disqualifying
An application that must give different people different access to the same data cannot express that here, because the engine has nowhere to put the distinction.
Giving the counter-case is necessary. The limitation is structural rather than a missing feature.
Why the file format matters more than usual
The file is the interface between versions of the application and between different applications, so its stability is a compatibility promise rather than an implementation detail.
Introducing the format's role is worthwhile. The storage layer is public here in a way a server's never is.
Why long-term format stability is a feature
A format readable by every version for decades makes the file suitable as an application's own document format and as an archival container, which is a use a server cannot serve at all.
Giving the consequence is valuable. The stability enables a category of use.
Why a single file is operationally pleasant
Backing up, copying, shipping and deleting a database are ordinary file operations, so much of what requires tooling elsewhere requires nothing here.
Covering the practical benefit is right. The simplicity is real and it is often decisive.
Why copying it carelessly is still wrong
A file copied while a writer is active captures a partial state, so the copy may be inconsistent or unreadable even though the operation appeared to succeed.
Giving the caveat immediately is important. The pleasant property has one sharp edge.
What the correct copy methods are
An interface the engine provides for taking a consistent copy while the database is in use, or a statement that writes a fresh compacted database to a new file.
Giving the remedy alongside the caveat is practical. Both options are built in.
Why this engine is the most widely deployed
It is inside operating systems, browsers, phones, aircraft and countless applications, because almost any program that needs structured local storage needs exactly this and nothing more.
Giving the scale is worthwhile. The ubiquity is a consequence of the fit.
Where it fits well
Per-user or per-device data, application configuration and state, caches, test fixtures, analytics staging, and anything that wants a queryable document format.
Listing the good fits is practical. The list is long and specific.
Where it does not fit
Many writers on different machines, data that must be reachable over a network, and access control that distinguishes between people.
Giving the boundary is necessary. The exclusions follow from the structure.
Why the usual objection is weaker than it sounds
The common criticism is that it cannot handle concurrency, when in fact it handles many concurrent readers well and is limited to one writer at a time, which many workloads never exceed.
Correcting the received opinion is valuable, because the objection is usually applied too broadly. The real limit is narrower than the reputation.
Why measuring before rejecting it is worth the effort
A single writer can commit a large number of transactions per second on local storage, which is more than a great many applications ever need.
Giving the practical advice is right. The decision deserves a measurement.
What to take from the embedded model
Three things. A query is a function call inside the application's process rather than a request to a server, which removes the round trip, the pool, the authentication and the supervision, and makes small queries dramatically cheaper. In exchange the application and the database share a failure domain and a memory space, coordination between processes happens through filesystem locks that are coarser and less reliable than a lock manager, and a database file on a network filesystem risks corruption rather than slowness. And the file itself is the interface, which makes it an excellent document and archive format, makes backup an ordinary file operation, and makes copying it under an active writer a way to produce a broken copy.
Never put an embedded database file on a network filesystem shared between machines. The engine relies on the filesystem's locking being correct to coordinate processes, and network filesystem locking is frequently incomplete, advisory or quietly broken, so two machines can both believe they hold a write lock. The result is not slow access or a clear error, it is corruption, and it appears intermittently under load rather than immediately, which means it typically reaches production. If two machines must share the data, the honest answer is that this is the wrong engine for that requirement rather than something to configure around.