The Ten Guidelines of Schema Progress

25 December 2016

Knowledge outlives code, and a invaluable database helps many functions over time. These ten guidelines will assist develop your database schema with out breaking your functions.

1.  Prod is just not like dev.

Manufacturing is just not growth. In manufacturing, a number of codebases rely in your knowledge, and these ten guidelines beneath must be adopted exactingly.

A dev setting could be rather more relaxed.  Alone in your growth machine experimenting with a brand new function, you don’t have any customers to interrupt.  You may soften the principles, as long as you harden them when transitioning to manufacturing.

2.  Develop your schema, and by no means break it.

The shortage of widespread vocabulary makes all of it too straightforward to automate the mistaken practices. I’ll use the phrases progress and breakage as outlined in Wealthy Hickey’s Spec-ulation discuss.  In schema phrases:

• progress is offering extra schema
• breakage is eradicating schema, or altering the that means of current schema.

In distinction to those phrases, many individuals use “migrations“, “refactoring“, or “evolution“. These usages are inclined to concentrate on repeatability, comfort, and the wants of recent applications, ignoring the excellence between progress and breakage. The issue right here is clear: Breakage is unhealthy, so we do not need it to be extra handy!

Utilizing exact language underscores the prices of of breakage. Most migrations are simply categorized as progress or breakage by contemplating the principles beneath.  Progress migrations are appropriate for manufacturing, and breakage migrations are, at finest, a dev-only comfort. Preserve them broadly separate.

3. The database is the supply of reality.

Schema progress must be reproducible from one setting to a different.  Reproducibility helps the event and testing of recent schema earlier than placing it into manufacturing and in addition the reuse of schema in numerous databases. Schema progress additionally must be evident within the database itself, in an effort to decide what the database has, what it wants, and when progress occurred.

For each of those causes, the database is the right supply of reality for schema progress. When the database is the supply of reality, reproducability and auditability occur without cost through the unusual
question and transaction capabilities of the database.  (In case your database is less than the duties of queries and transactions you will have greater issues past the scope of this text).

Storing schema in a database is strictly extra highly effective than storing schema as textual content recordsdata in supply management. The database is the precise residence for schema, plus it supplies validation, construction, question, transactions, and historical past. A supply management system supplies solely historical past and is separate from the information itself.

Observe that this does not imply “by no means put schema info in supply management”. Supply management could also be handy for different causes, e.g. it might be extra readily accessible. You might redundantly retailer schema in supply management, however keep in mind that the database is definitive.

4.  Rising is including.

As you purchase extra details about your area, develop your schema to match. You may develop a schema by including new issues, and solely by including new issues, for instance:

• including new attributes to an current ‘kind’
• including new varieties
• including relationships between varieties

5.  By no means take away a reputation.

Eradicating a named schema element at any stage is a breaking change for applications that rely upon that identify. By no means take away a reputation.

6.  By no means reuse a reputation.

The that means of a reputation is established when the identify is first launched. Reusing that identify to imply one thing considerably totally different breaks applications that rely upon that that means. This may be even
worse than eradicating the identify, because the breakage will not be as instantly apparent.

7.  Use aliases.

In case you are accustomed to database refactoring patterns, the recommendation in Guidelines 5 and Six could seem stark. In any case, one function of refactoring is to undertake higher names as we uncover them. How can we
do this if names can by no means be eliminated or modified in that means?

The straightforward resolution is to make use of more than one alias to consult with the identical schema entity. Contemplate the next instance:

• In iteration 1, customers of your system are recognized by their e-mail with an attribute named :consumer/id. 
• In iteration 2, you uncover that customers generally have non-email identifiers for customers and that you simply wish to retailer a consumer’s e-mail even when not utilizing the e-mail as an identifier. Briefly, you would like that :consumer/id was named :consumer/primary-email.

No drawback! Simply create :consumer/primary-email as an alias for :consumer/id. Older applications can proceed to make use of :consumer/id, and newer applications can use the now-preferred :consumer/primary-email.

8.  Namespace all names.

Namespaces vastly cut back the price of getting a reputation mistaken, as the identical native identify can safely have totally different meanings in numerous namespaces.  Persevering with the earlier instance, think about that the native
identify id is used to consult with a UUID in a number of namespaces, e.g. :stock/id, :order/id, and so forth. The truth that :consumer/id is not a UUID is inconsistent, and newer applications shouldn’t must put up with this.

Namespaces allow you to enhance the state of affairs with out breaking current applications. You may introduce :user-v2/id, and new applications can ignore names within the consumer namespace. If you happen to do not like v2, it’s also possible to decide a extra semantic identify for the brand new namespace.

9.  Annotate your schema.

Databases are good at storing knowledge about your schema. Including annotations to your schema will help each human readers and make sense of how the schema grew over time. For instance:

• you would annotate names that aren’t really helpful for brand spanking new applications with a :schema/deprecated flag, or you would get fancier nonetheless with :schema/deprecated-at or :schema/deprecated-because. Observe that such deprecated names are nonetheless by no means eliminated (Rule 5).
• you would present :schema/see-also or :schema/see-instead tips to extra present conventions. 

In actual fact, all of the database refactoring patterns which might be usually carried out as breaking modifications could possibly be carried out non-destructively, with the refactoring particulars recorded as an annotation. For instance, the breaking “split column” refactoring may as an alternative be carried out as schema progress:

• add N new columns
• (non-obligatory) add a :schema/split-into attribute on the unique column whose worth is the brand new columns, and presumably even the recipe for the break up

10. Plan for accretion.

If a system goes to develop in any respect, then applications should not bake in limiting presumptions.  For instance: If a schema states that :consumer/id is a string, then applications can depend on :consumer/id being a string and never often an integer or a boolean.  However a program can not assume {that a} consumer entity can be restricted to a the set of attributes beforehand seen, or that it understands the semantics of attributes that it has not seen earlier than.

Are these guidelines particular to a selected database?

No. These guidelines apply to virtually any SQL or NoSQL database.  The foundations even apply to the so-called “schemaless” databases.  A greater phrase for schemaless is “schema-implicit”, i.e. the schema is implicit in your knowledge and the database has no reified consciousness of it.  With an implicit schema, all the principles nonetheless apply, besides that the database is impotent that can assist you (no Rule 3).

In Context

Lots of the assets on migrations, refactoring, and database evolution emphasize repeatability and the wants of recent applications, with out making the top-level distinctions of progress vs. breakage and prod vs. dev. Consequently, these assets encourage breaking the principles on this article.

Fortunately, these assets can simply be recast in growth-only phrases.  You can develop your schema with out breaking your app. You can repeatedly deploy with out repeatedly propagating breakage.  This is what it appears to be like like in Datomic.