What is already done?

• PERIOD data type, which can represent anchored intervals of time; that is, a chunk of time with a definite beginning and a definite end (in contrast to a SQL INTERVAL, which is not anchored to any specific beginning or end time).
  • Critical for usability because it acts as a set of time, so you can easily test for containment and other operations without using awkward constructs like BETWEEN or lots of comparisons (and keeping track of inclusivity/exclusivity of boundary points).
  • Critical for performance because you can index the values for efficient “contains” and “overlaps” queries (among others).

• Temporal Keys (called Exclusion Constraints, and will be available in the next release of PostgreSQL, 9.0), which can enforce the constraint that no two periods of time (usually for a given resource, like a person) overlap. See the documentation (look for the word “EXCLUDE”), and see my previous articles (part 1 and part 2) on the subject.
  • Critical for usability to avoid procedural, error-prone hacks to enforce the constraint with triggers or by splitting time into big chunks.
  • Critical for performance because it performs comparably to a UNIQUE index, unlike the other procedural hacks which are generally too slow to use for most real systems.

What needs to be done?

• Range Types — Aside from PERIOD, which is based on TIMESTAMPTZ, it would also be useful to have very similar types based on, for example, DATE. It doesn’t stop there, so the natural conclusion is to generalize PERIOD into “range types” which could be based on almost any subtype.
• Range Keys, Foreign Range Keys — If Range Types are known to the Postgres engine, that means that we can have syntactic sugar for range keys (like temporal keys, except for any range type), etc., that would internally use Exclusion Constraints.
• Range Join — If Range Types are known to the Postgres engine, there could be syntactic sugar for a “range join,” that is, a join based on “overlaps” rather than “equals”. More importantly, there could be a new join type, a Range Merge Join, that could perform this join efficiently (without a Range Merge Join, a range join would always be a nested loop join).
• Simple table logs — The ability to easily create an effective “audit log” or similar trigger-based table log, that can record changes and be efficiently queried for historical state or state changes.

I’ll be speaking on this subject (specifically, the new Exclusion Constraints feature) in the upcoming PostgreSQL Conference EAST 2010 (my talk description) in Philadelphia later this month and PGCon 2010 (my talk description) in Ottawa this May. In the past, these conferences and others have been a great place to get ideas and help me move the temporal features forward.

The existing features have been picking up a little steam lately. The temporal-general mailing list has some traffic now — fairly low, but enough that others contribute to the discussions, which is a great start. I’ve also received some great feedback from a number of people, including the folks at PGX. There’s still a ways to go before we have all the features we want, but progress is being made.

So, right away, it’s meaningless to talk about the scalability of the relational model. Given a particular question, it might be difficult to break it down into bite-sized pieces and distribute it to N worker nodes. But going with MapReduce doesn’t solve that scalability problem — it just means that you will have a hard time defining a useful map or reduce operation, or you will have to change the question into something easier to answer.

There may exist scalability problems in:

• SQL, which defines requirements outside the scope of the relational model, such as ACID properties and transactional semantics.
• Traditional architectures and implementations of SQL, such as the “table is a file” equivalence, lack of sophisticated types, etc.
• Particular implementations of SQL — e.g. “MySQL can’t do it, so the relational model doesn’t scale”.

Why are these distinctions important? As with many debates, terminology confusion is at the core, and prevents us from dealing with the problems directly. If SQL is defined in a way that causes scalability problems, we need to identify precisely those requirements that cause a problem, so that we can proceed forward without losing all that has been gained. If the traditional architectures are not suitable for some important use-cases, they need to be adapted. If some particular implementations are not suitable, developers need to switch or demand that it be made competitive.

The NoSQL movement (or at least the hype surrounding it) is far too disorganized to make real progress. Usually, incremental progress is best; and sometimes a fresh start is best, after drawing on years of lessons learned. But it’s never a good idea to start over with complete disregard for the past. For instance, an article from Digg starts off great:

The fundamental problem is endemic to the relational database mindset, which places the burden of computation on reads rather than writes.

That’s good because he blames it on the mindset not the model, and then identifies a specific problem. But then the article completely falls flat:

Computing the intersection with a JOIN is much too slow in MySQL, so we have to do it in PHP.

A join is faster in PHP than MySQL? Why bother even discussing SQL versus NoSQL if your particular implementation of SQL — MySQL — can’t even do a hash join, the exact operation that you need? Particularly when almost every other implementation can (including PostgreSQL)? That kind of reasoning won’t lead to solutions.

So, where do we go from here?

1. Separate the SQL model from the other requirements (some of which may limit scalability) when discussing improvements.
2. Improve the SQL model (my readers know that I’ve criticized SQL’s logical problems many times in the past).
3. Improve the implementations of SQL, particularly how tables are physically stored.
4. If you’re particularly ambitious, come up with a relational alternative to SQL that takes into account what’s been learned after decades of SQL and can become the next general-purpose DBMS language.

EDIT 2010-03-09: I should have cited Josh Berkus’s talk on Relational vs. Non-Relational (complete list of PGX talks), which was part of the inspiration for this post.

PostgreSQL is a great example: it gets error recovery right. I didn’t say it’s perfect, but graceful degradation is designed into postgres from the start, and if it doesn’t degrade gracefully, that’s probably a bug. I am saying this as a person who experienced a not-so-graceful PostgreSQL problem first-hand back in 2006 that was due to an OOM condition on FreeBSD (and yes, malloc() returned NULL). But:

• The bug occurred under fairly strange circumstances where very small allocations ate the memory very completely, leaving no room (not even a few dozen bytes), during a transaction that included extensive DDL (note that most database systems can’t even do transactional DDL at all).
• The bug was fixed!

A database system getting error recovery right is no accident, fluke, or statistical outlier. Database systems are meant to get this right so that applications do not have to do it. In other words, write applications against PostgreSQL (or another robust relational database system), and you get error recovery for free.

The application may be performing a complex series of updates or state changes, and trying to ensure graceful error recovery without a database system to help may be nearly impossible. But by simply wrapping those operations in a single transaction, the application only needs to know how to handle a few classes of errors, and will always see consistent data afterward.

Moral of the story: if you’re not building on top of a robust DBMS, and you have any significant amount of state to manage, then you are not getting error recovery right (unless you put in the serious analysis required). Similar problems apply to those using a DBMS like a filesystem, with no constraints (to prevent corruption) and related operations split across transactional boundaries. So: use PostgreSQL, use transactions, and use constraints.

1. Run out of memory.
2. Kill PostgreSQL.
3. Look for processes that might be using too much memory, and kill them, hopefully freeing memory.

Notice that step #2 is independent of factors like:

• Is PostgreSQL consuming a significant share of the memory resources?
• Will killing PostgreSQL alleviate any memory pressure at all?

The reason for this is because linux, when under memory pressure, invokes the badness() function to determine which process to kill. One of the things that the function counts against a process is the total VM size of the process (no surprise), plus half of the total VM size of all of the children. That sounds reasonable — wait a minute, what about shared memory?

For every connection, PostgreSQL makes a child process, which shares memory with the parent; and if it is a big postgres instance, that can be a significant amount of memory. But the badness function counts the same shared memory against the parent 1+N/2 times, where N is the number of connections open! For example, if you have shared_buffers set to 1GB on an 8GB machine, and have 20 connections open, then the linux badness function thinks that PostgreSQL is using about 11GB of memory, when it is actually using 1GB!

It gets worse: killing a process does not free shared memory. And there are already administrator-controlled limits that start off fairly low (32MB, I think), so the administrator would have to make a mistake in order for shared memory to be the problem.

And it gets even worse: linux makes very little attempt to avoid getting into a bad situation in the first place. On most operating systems, if you ask for too much memory, malloc() will eventually fail, which will probably cause the application to terminate; or if the application is written to handle such conditions (like PostgreSQL), it will gracefully degrade. On linux, by default, brk() (the system call underlying malloc()) will happily return success in almost every situation, and the first hint that your application gets of a problem is a SIGKILL.

I have heard many excuses for this appalling set of behaviors, but none of them are satisfactory to me. Here is one explanation. Notice the implicit assumption is that the most important thing that a user may be running is a text editor, and it should just autosave every once in a while to avoid problems. Keeping processes actually running is apparently not important to Linux. The next general philosophy present in the email is that applications are stupid, and they will never get the error paths or rollback right, but PostgreSQL seems to do that just fine (it needs very little memory to roll back, and tries to free memory before that to make it less likely to run into a problem). And the author seems to think that the first hint of a system problem should be a SIGKILL based on some heuristic (which, in the case of linux, is seriously flawed, as I pointed out above).

Among other justifications are:

• “You can configure linux to prevent this problem.” Sounds like MySQL, where you have to declare your tables to be transaction-safe. Why not safe by default, and configure for performance?
• “Modern systems overcommit, and there are always edge cases where you get into problems.” OK, so keep them as edge cases, and at least attempt to avoid problem situations. If some bizarre set of circumstances, or maybe a malicious process, is able to cause memory problems on your system, then so be it. But well-behaved processes should get some indication of trouble prior to SIGKILL. And the real problem processes should be identified with some accuracy and reasonable intelligence (i.e. recognize that shared memory can’t be freed by the OOM Killer, and therefore shouldn’t be considered).

I have been complaining about this insanity for years:

• http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg120596.html (2007)
• http://lkml.org/lkml/2007/2/9/275 (2007)
• http://lkml.org/lkml/2008/2/5/495 (2008)

It makes me happy that other free software operating systems are still under active development, as illustrated by the recent release of FreeBSD 8.0. I am not saying FreeBSD is better in every way than Linux, but I do believe that competition is important even in the free software world.

This post is obviously a rant, and an ill-informed one, at that. I am no expert on memory management, so let me know if I am way off-base. However, I have never heard reasonable justifications for these things in response to my emails to lkml, in personal discussions with kernel hackers or other knowledgeable Linux folk, or anywhere else. If you do have a reasonable explanation, I am more than willing to listen.

• A schedule conflict is a typical business problem.
• The later you try to resolve a schedule conflict, the more costly it is to resolve.
• In particular, there is a big jump in the cost the moment after conflicting data is recorded.
• Therefore, it’s best for the DBMS itself to enforce the constraint, because only the DBMS can avoid the conflict effectively before the conflict is recorded.

Then, I opened up a discussion to see how people are dealing with these schedule conflicts. In the comments I received at the end of the article, as well as other anecdotes from conferences, user groups, mailing lists, and my own experience, the solutions fall into a few categories:

• The rate of conflicts is so low that the costs are not important. For instance, you may make 0.1% of your customers unhappy, and need to refund them, but perhaps that’s a cost you’re willing to pay.
• The application receives so few requests that performance is not an object, and serialization of all requests is a viable option. The serialization is done using big locks and a read-check-write cycle. Even if performance is not an object, these applications sometimes run into maintenance problems or unexpected outages because of the big locks required.
• You can break the time slices into manageable chunks, e.g. one day chunks aligned at midnight. This kind of solution is highly specific to the business, reduces the flexibility of the business, and often requires a substantial amount of custom, error-prone procedural code.
• Complex procedural code: usually a mix of application code, functions in the DBMS, row-level locking, static data in tables that only exists for the purposes of row-level locks, etc. This kind of solution is generally very specific to the application and the business, requires lots of very error-prone custom procedural code, is difficult to adequately test, and it’s hard to understand what’s going on in the system at any given time. Hunting down sporadic performance problems would be a nightmare.

Those solutions just aren’t good enough. We use relational database systems because they are smart, declarative, generally useful for many problems, and maintainable (Note: these principles contrast with NoSQL, which is moving in the opposite direction — more on that in another article).

[UPDATE: The following project has been committed for the next release of PostgreSQL; the feature is now called "Exclusion Constraints"; and the new version of PostgreSQL will be called 9.0 (not 8.5). See the documentation under the heading "EXCLUDE".]

The project that I’ve been working on for PostgreSQL 8.5 is called “Operator Exclusion Constraints“. These are a new type of constraint that most closely resembles the UNIQUE constraint, because one tuple can preclude the existence of other tuples. With a UNIQUE constraint on attribute A of a table with attributes (A, B, C), the existence of the tuple (5, 6, 7) precludes the existence of any tuple (5, _, _) in that table at the same time. This is different from a foreign key, which requires the existence of a tuple in another table; and different from a CHECK constraint which rejects tuples independently from any other tuple in any table (and the same goes for NOT NULL).

The same semantics as a UNIQUE constraint can be easily specified as an Operator Exclusion Constraint, with a minor performance penalty at insert time (one additional index search, usually only touching pages that are already in cache). Exclusion constraints are more general than UNIQUE, however. For instance, with a complex type such as CIRCLE, you can specify that no two circles in a table overlap — which is a constraint that is impossible to specify otherwise (without resorting to the poor solutions mentioned above).

This applies to temporal keys very nicely. First, get the PERIOD data type, which allows you a better way to work with periods of time (sets of time, really), rather than points in time. Second, you need to install the btree_gist contrib module. Then, use an exclusion constraint like so:

[UPDATE 2010-03-09: Syntax updated to reflect the version of this project committed for PostgreSQL 9.0. ]

CREATE TABLE room_reservation
(
  name   TEXT,
  room   TEXT,
  during PERIOD,
  EXCLUDE USING gist (room WITH =, during WITH &&)
);

That will prevent two reservations on the same room from overlapping. There are a few pieces to this that require explanation:

• && is the “overlaps” operator for the PERIOD data type.
• USING gist tells PostgreSQL what kind of index to create to enforce this constraint. The operators must map to search strategies for this index method, and searching for overlapping periods requires a GiST index.
• Because we are using GiST, we need GiST support for equality searches for the TEXT data type, which is the reason we need the btree_gist contrib module.
• Conflicts will only occur if two tuples have equal room numbers, and overlapping periods of time for the reservation.

This solution:

• Performs well under light and heavy contention. Not quite as well as a UNIQUE constraint, but much better than the alternatives, and without the surprises you might get from using big locks. Note that the constraint will be enforced at some point, so ignoring the problem is not a high-performance alternative (interpersonal communication has higher latency than a computer).
• Is declarative. The implementation shows through a little bit — the user will know that an index is being used, for instance — but it’s a relatively simple declaration. As a consequence, it’s not very error-prone from the schema designer’s standpoint.
• Is not specific to the business. You don’t have to decide on an appropriate time slice (e.g. one hour, one day, etc.); you don’t have to try to partition locks in creative ways; you don’t have to write procedural code (in the database system or application); and you don’t have to come up with interesting ways to detect a conflict or notify the user.

Temporal keys are just one part of the support required for effective temporal data management inside the DBMS. However, it’s one of the most important pieces that requires support from the core engine, and cannot be implemented as a module.

Does your database system know anything about a schedule conflict? Should it?

Constraints are always enforced at some point, it’s just a matter of when, how, and the cost of correcting the situation.

Consider two professors who try to reserve the same lecture hall at overlapping times (let’s say 1-2pm, and 1:30-2:30pm). Now imagine two possible resolutions:

1. They are blissfully unaware of the schedule conflict until they have already promoted their lecture, attracted an audience, and arrived at the building. The professor scheduled for 1:30-2:30pm will be disappointed to find it already in use, and the students will be confused. The second lecture may have to be canceled due to the confusion, or lose a substantial number of attendees.
2. At the time of scheduling, one professor is given an error message to the effect of “this room is already reserved, please choose a different time”.

Observe that neither one really solves the problem: the second resolution still forces one professor to choose a less-desirable time. However, it is a much cheaper resolution. As with many problems, early detection is simplest and cheapest to resolve.

Of course, there are many resolutions that fall between the first and the second. For instance, you might run a program every hour that checks for conflicts and alerts the parties involved. That may work, but there are still problems:

• You’ve now introduced uncertainty into the system: do I have a reservation or not? If there is a conflict later, will I be kicked out or will they?
• You have to come up with rules for resolution: does the first one win? How do you define “first” if transactions are running for a while? What if someone makes a reservation “first” but then makes all kinds of changes later; were they really first or did they just game the system?
• If someone’s reservation gets bumped, you have to now have a new strange state for a reservation, in which it is disabled, the organizer has (hopefully) been notified, and it needs to change.
• Notice that everything is very procedural and specific to the business. You have to have a notification mechanism, and rules for how to resolve it.
• Let’s go back to the definition of “first”: say you have made a reservation, and you get bumped by the conflict detector. In between the time you made the reservation and the time you were notified of a conflict, someone else reserved your second choice. Are you now first in line for that time slot? If so, that has a cascading effect such that it’s almost impossible for the person that took the second-choice slot to know that they are at risk of being bumped.

These thought experiments might seem like edge cases, but reservation systems have two common traits that make these problems very real:

1. They tend to start allowing reservations of a specific resource at a specific time, published in advance.
2. There tend to be some resources and time slots that have a much higher value than the others.

These two traits lead to heavy contention.

Now, how would you go about enforcing such a constraint (non-overlapping time periods) in the database? While considering possible solutions, think about:

• Does the solution work for a wide variety of use cases, or only in special cases?
• How well would it perform, under low contention and high contention, and under varied workloads?
• Can it be implemented in a general purpose RDBMS, like PostgreSQL?
• Is it procedural in nature, or declarative?

I think it’s worthwhile to consider the problem, so I will end this article now, and provide some approaches, as well as my real answer, in the next article. In the meantime, feel free to post ideas as comments.

There were a lot of great talks. Scott Bailey’s temporal data talk was excellent, of course — but I’m not an impartial observer, I expressed enthusiasm before the conference began. Another talk that I found very interesting was Bill Karwin’s talk on practical object oriented models in SQL. The talk was well-researched and the points were made very clearly, although I would have liked to see a little more depth. There was some audience discussion, and I’d like to see the points hashed out in a little more detail.

I’m always excited to present the projects that I’m working on, and this conference was no exception. I thoroughly enjoyed discussing Operator Exclusion Constraints, the new feature that I’m writing for PostgreSQL 8.5 that enables “temporal keys”. Slides are here — I’ll write in more detail in another article. I was pleased with the audience participation and questions during the talk, and I hope that everyone enjoyed listening to me as much as I enjoyed speaking.

Slides for my talk on extensibility are available here.

In the past, I’ve worked on a temporal extension to PostgreSQL that implements the PERIOD data type. This is a data type that offers both a definite beginning and a definite end time, which is important for describing things that happen over a period of time, rather than instantaneously. Trying to use separate attributes for “start” and “end” is bad for a number of reasons, and will certainly be addressed in a subsequent blog entry. For now, I’ll just say that I believe the PERIOD data type is fundamentally important for handling all kinds of time data, which I believe is a common problem.

At WEST, I’ll be presenting my progress on temporal keys. Temporal keys are used to prevent overlapping periods of time — a schedule conflict — by using an index and following the same concurrent behavior as UNIQUE with minimal performance cost (one extra index search, to be precise).

Temporal keys cannot be expressed in PostgreSQL 8.4, unless you resort to triggers and a full table lock (ouch!). So, additional backend support is required. This is accomplished in my patch for operator exclusion constraints, which are a more general way of using arbitrary operators and index searches to enforce a constraint. I plan to do what’s required for the patch to be accepted in PostgreSQL 8.5.

Temporal modeling is a common problem. It seems like almost every PostgreSQL conference has had at least one talk on the matter, so we know there is some demand for improvement. If you’re interested, I hope you come to WEST and chat with Scott or I, and let’s see if we can come up with some real solutions.

The first observation I’d like to make is that, when considering a relational database, we care about sets of things. If you have an attribute that holds “notes”, there’s not much you can do with a set of such values. I’m not saying that notes are unimportant, or that such fields should be avoided; but you should recognize the limitations of such attributes. All you can do is store text that a user wrote, or present a user with a specifically-requested note. The user then must actually read the note to learn anything — there’s nothing automatic about it (a database doesn’t help much with manual processes).

In the above example, the data type doesn’t matter much at all. It really makes no logical difference if you store it as text or as a PNG image of the rendered text (there may be a difference in convenience and performace, as with all format choices, but that’s beside the point). If the noun you are working with is “note”, the database system can’t help you any more if the attribute is “text” than it can if the attribute is of type “bytea” holding an image of the rendered text.

Astute readers will now be thinking: “but what about full-text search?”. Exactly. PostgreSQL has an excellent full-text search system, and powerful data types like “tsvector” and “tsquery”. But these aren’t just better data types that magically work, these represent a change in nouns that the business understands. A search engine doesn’t index a bunch of text fields, it indexes documents. You don’t submit text to the search engine, you submit a query. A document isn’t just a string of characters, it is a set of word stems with positional and weighting information. The business knows these things, and these nouns are spoken in conference rooms by executives and to customers by sales people.

I’m drawing a direct connection between the business and the data types because I sometimes think that data types are treated as an implementation detail. Let’s say you store the notes as the “text” type and are treating it as such. When you go to implement search, the natural thing to do would be something like: ...WHERE note LIKE '%foo%' OR note like '%bar%' OR ..., which is absolutely the wrong approach. The right approach is using a real query against a real document type, not trying to match substrings.

Treating the data types as an implementation detail is also related to the idea of only using primitive types, as though SQL were java with no class library. Primitive data types lead to awkward application code trying to answer questions that the database system should be able to answer. The philosophy of primitive data types is reflected in the SQL standard, which defines few types and does not seem to encourage sophisticated types like other languages do (of course, the SQL standard breaks away from the primitive data types because some types are just too important). For some reason database people have a negative reaction to any complex type, despite how many times they have proven to be important (for another great example, check out the geometry type in PostGIS, a sophisticated GIS system built entirely as a PostgreSQL plugin).

I believe a lot of this negative reaction is because of 1NF, which essentially describes what a relation is. The concern is that any type more complex than a string might be called a violation of 1NF, which I think is ridiculous. There is no useful definition of an atomic value (except maybe a boolean or bit or something): a string is a list (or array) of characters, that can be spliced, cut, and concatenated in many ways; an integer has a sign portion and a magnitude; and a timestamp is quite obviously non-atomic (and often treated as separate components). Additionally, normalization doesn’t have anything to do with data type choices; it only has to do with attributes and certain specific types of constraints (join dependencies, of which functional dependencies are a special case). So choosing a different data type carries no danger of violating a normal form. That being said, normalization is a small part of database design, and there are very legitimate reasons to be skeptical of strange data type choices.

I’m very pleased that PostgreSQL offers such a powerful type system. In many ways this is more challenging for a database system than an application language — for instance, PostgreSQL offers a variety of index access methods, execution plans, and statistics that all depend on understanding your data types and how to operate on them. In PostgreSQL, a user-defined data type is truly first-class, meaning that you can make use of all of this infrastructure in creative ways. However, creating a good data type is not a trivial process, so it’s best to approach it as a real project by itself.

I started the Temporal PostgreSQL project quite a while ago to offer a “period” data type (a time interval with an absolute beginning and end), because I think that is an important noun that is underserved by database systems. More on this later.

First off, I’m strictly discussing SQL NULL here. SQL NULL is peculiar in a number of ways, and the general excuse for this is that there is a need to represent “missing information” — which may be true. But there are lots of ways to represent missing information, as I pointed out in a previous post, and SQL’s approach to missing information is, well, “unique”.

• “NULL is not a value” — If you hear this one, beware: it’s in direct contradiction to the SQL standard, which uses the phrase “null value” dozens of times. It’s hard to imagine that NULL is not any kind of value at all; because it’s routinely passed to functions and operators, predicates can evaluate to NULL, and SQL uses a kind of three-valued logic (3VL) in some contexts. The phrase “NULL is not a value” also raises the question: “what is it, then?”.
• NULL means “unknown” (i.e. the third truth value) — This doesn’t hold up either. SUM of no tuples returns NULL, but clearly the SUM of no tuples is not unknown! SQL will happily generate NULLs from aggregates or outer joins without any NULLs at all in the database. Do you not know something you did know before, or do you now know that you don’t know something that you didn’t know you didn’t know before? Also, if NULL means “unknown”, how do you differentiate a boolean field for which you do not know the value, and a boolean field for which you do know the value, and it happens to be “unknown” (perhaps this is why boolean columns are a PostgreSQL extension and not part of the core SQL standard)?
• “NULL is false-like” — Don’t think of NULL as false-like, or “more false than true”. It’s a tempting rule of thumb, but it’s misleading. For instance, in a WHERE clause, a NULL predicate is treated like FALSE. However, in a constraint (like a CHECK constraint), NULL is treated like TRUE. Perhaps most importantly, when in a 3VL context (like a boolean expression), this misconception leads to problems when you try to invert the logic, e.g., use NOT.
• “Oh, that makes sense” — When you see individual behaviors of NULL, they look systematic, and your brain quickly sees a pattern and extrapolates what might happen in other situations. Often, that extrapolation is wrong, because NULL semantics are a mix of behaviors. I think the best way to think about NULL is as a Frankenstein monster of several philosophies and systems stitched together by a series of special cases.
• p OR NOT p — Everyone should know that this is not always true in SQL. But most people tend to reason assuming that this is always true, so you have to be very careful, and work against your intuition very deliberately, in order to form a correct SQL query.
• SUM() versus + (addition) — SUM is not repeated addition. SUM of 1 and NULL is 1, but 1 + NULL is NULL.
• Aggregates ignore NULLs — According to the standard, aggregates are supposed to ignore NULLs, because the information is missing. But why is it OK to ignore the missing information in an aggregate, but not, say, with the + operator? Is it really OK to just ignore it?
• Aggregates return NULL — According to the standard, aggregates are supposed to return NULL when they have no non-NULL input. Just because you don’t have any input tuples, does that really mean that the result is undefined, missing, or unknown? It’s certainly not unknown! What about SUM over zero tuples, wouldn’t the most useful result be zero?
• SQL breaks its own rules — The aforementioned aggregate rules don’t work very well for COUNT(), the simplest of all aggregates. So, they have two versions of count: COUNT(*) breaks the “aggregates ignore nulls” rule and the “aggregates return null” rule, and COUNT(x) only breaks the latter. But wait! There’s more: ARRAY_AGG() breaks the former but not the latter. But no exception is made for SUM — it still returns NULL when there are no input tuples.
• NULLs appear even when you have complete information — Because of OUTER JOIN and aggregates, NULLs can appear even when you don’t have any NULLs in your database! As a thought experiment, try to reconcile this fact with the various “definitions” of NULL.
• WHERE NOT IN (SELECT ...) — This one gets everyone at one point or another. If the subselect produces any NULLs, then NOT IN can only evaluate to FALSE or NULL, meaning you get no tuples. Because it’s in a WHERE clause, it will return no results. You are less likely to have a bunch of NULLs in your data while testing, so chances are everything will work great until you get into production.
• x >= 10 or x <= 10 — Not a tautology in SQL.
• x IS NULL AND x IS DISTINCT FROM NULL — You probably don’t know this, but this expression can evaluate to TRUE! That is, if x = ROW(NULL).
• NOT x IS NULL is not the same as x IS NOT NULL — If x is ROW(1,NULL), then the former will evaluate to TRUE, and the latter will evaluate to FALSE. Enjoy.
• NOT x IS NULL AND NOT x IS NOT NULL — Want to know if you have a value like ROW(1, NULL)? To distinguish it from NULL and also from values like ROW(1,1) and ROW(NULL,NULL), this expression might help you.
• NULLs can exist inside some things, but not others — If you concatenate: firstname || mi || lastname, and “mi” happens to be null, the entire result will be null. So strings cannot contain a NULL, but as we see above, a record can.

I believe the above shows, beyond a reasonable doubt, that NULL semantics are unintuitive, and if viewed according to most of the “standard explanations,” highly inconsistent. This may seem minor; that is, if you’re writing SQL you can overcome these things with training. But it is not minor, because NULL semantics are designed to make you think you understand them, and think that the semantics are intuitive, and think that it’s part of some ingenious consistent system for managing missing information. But none of those things are true.

I have seen lots of discussions about NULL in various forums and mailing lists. Many of the participants are obviously intelligent and experienced, and yet make bold statements that are, quite simply, false. I’m writing this article to make two important points:

1. There is a good case to be made that NULL semantics are very counterproductive; as opposed to a simple “error early” system that forces you to write queries that explicitly account for missing information (e.g. with COALESCE). “Error early” is a more mainstream approach, similar to null pointers in java or None in python. If you want compile-time checking, you can use a construct like Maybe in haskell. SQL attempts to pass along the problem, hoping the next operator will turn ignorance into knowledge — but it does not appear that anyone thought through this idea, quite frankly.
2. You should not attempt to apply your intellect to NULL, it will lead you in the wrong direction. If you need to understand it, understand it, but always treat it with skepticism. Test the queries, read the standard, do what you need to do, but do not attempt to extrapolate.