The only thing you really need to understand about databases is that they store data for you, ie. they persist data. So once you get that then you start asking questions like "where" and "how".
The "where" is on hard drives. Don't let anyone tell you that a purely memory resident data store is a database, it's not. It's a cache. If you aren't writing to disk you are not a persistent database. Think memcached and redis. Although I would say the later is more of a database because you can configure it to write on every update.
The "how" is where all the devilish details are. How do you put data into and get data out of a database. How does the database store your data on disk. Well, there are two main schools of thought in the market place at the moment. Relational[0] and non-relational. Relational is the older more mature, more well supported, more well understood, more "standards" compliant of the two. Nowadays, non-relational is championed by the NoSQL[1] movement which eschews the relational model for a looser concept, less defined, less "compliant", less understood model. You will find many more and varied answers to "how" in the non-relational space precisely because the renaissance in non-relational is so fresh (the concept itself has been around a long time). Nevertheless, NoSQL has been attracting considerable attention over the last year or so in the open source community.
Every developer worth his salt will know or at least be familiar with the relational model and at least one or two of it's implementations. MySQL, PostgreSQL and SQLite are just three of the most popular open source RDBMS's (relational database management system)[2] around today. They all have varying degrees of support for the primary language used to speak to RDBMS's, SQL[3].
There are many open source, non-relational implementations in production today. HBase, Cassandra, Riak, Voldemort and MongoDB are all playing in the NoSQL field. Popular systems within NoSQL draw their lineage from either the Google approach (gfs[4], big table[5]) or the Amazon approach (dynamo[6]). Yet another is Neo4j, a graph database[7] which, at the moment, is lumped under the "NoSQL" moniker but is a separate beast entirely. Just like their relational cousins, the way NoSQL solutions do what they do and what they are best suited for vary.
As with most interesting discussions, which approach is best is left as an exercise for the reader. I'll just throw my two cents in for good measure. First I would say that you need to know SQL. Knowing SQL and not a specific system (like mysql or postgresql) will allow you to move in between relational systems and let you pick the right one for the job. After taking a look and familiarizing yourself with the SQL world, I would take a very serious look at what is going on with NoSQL. Best I can tell, there is a lot of mindshare and "cool" factor foisted upon this space right now because of the scale non-relational concepts can bring to enterprise users. Be sure, though, that rdbms is the predominant player in the database world. Every company large and small use some form of rdbms. Newer more forward looking companies that have large and very large scale needs to store and analyze vast quantities of unstructured or loosely structured data are looking more and more to non-relational NoSQL solutions.
I specifically avoid linking to particular implementations but rather link to broader concepts here. It is the concept that is more important than the implementation when learning about the big and confusing world of databases.
I want to be careful in how I phrase this, because I feel like this was a good effort, but I disagree with many parts of this description.
First, databases are not "just" about persistence, and I don't mean that in a pedantic way. The point of a database management system is to manage your data over time and allow you to make use of it. This includes things like stating constraints to ensure that the data in the database is always correct and providing methods for querying your data at varying levels of complexity.
Second, there are a number of in-memory databases. Oracle sells one. SQLite is often used as one. The fact that they are in-memory does not make them not databases.
Third, relational versus non-relational is not really an argument about "how the data is stored on disk" or about being "standards compliant." A database traditionally consists primarily of three things: a data model, a query language, and an implementation. A data model, like "everything is a table with column headers and rows" can be written to disk in many different ways and queried in many different ways. (It's an abstraction.) Indexing structures and data formats may be mostly the same even when the data model is different. The (current) "relational vs non-relational" debate, such as it is, is usually about whether it makes sense to change the data model and query language in order to ensure that the implementation is scalable on "cloud-like" platforms with specific data access needs. (That said, there are many different types of non-relational models and query languages designed for different purposes.)
Fourth, there isn't one SQL. There are in fact a number of standards, SQL-86, SQL-89, SQL-92, and so on. It makes sense to learn the standard first (at least up to 92 or so), rather than learning particular implementation specifics. Most databases implement SQL-92 or above, though there is somewhat wide variance in additional features like indexing structures and functions and in data modeling languages.
The only thing you really need to understand about databases is that they store data for you, ie. they persist data.
...is like saying "the only thing you really need to know about sex is that it is for procreation." :-)
Sorry - couldn't resist. Utilizing databases effectively is often an area that is overlooked or kind of an afterthought. In my experience, often issues of scale stem from poor database design.
I put this in a separate comment, but any of Tom Kyte's books are excellent for gaining an understanding of how to use databases effectively. Although they are Oracle focused, there is still a lot to be learned about practical database design and utilization from his books. He also has a great site called “Ask Tom” where he answers questions. I have learned a ton from simply reading questions and his responses. Again, all Oracle focused, but very valuable nonetheless.
The problem with the standards is that getting copies costs actual money. If one really wants to start learning about databases, it can be very difficult to decide what to pay for. Implementation documentation, at least, is generally free.
I completely agree. My intent was to suggest that one should learn to the standard first rather than any particular implementation, not that one should read any of the actual standards documents directly. (Yikes!) By analogy, if you want to learn C, read K&R to learn something approximating C89, rather than picking up a book on how to code to the specific dialect of C understood by GCC 4.5.
For what it is worth, I learned from Database Systems: The Complete Book.
DS:TCB is pretty explicit about which of the SQL it teaches is part of which standard. That said, I suspect that any general database book should do a reasonable job.
The "where" is on hard drives. Don't let anyone tell you that a purely memory resident data store is a database, it's not. It's a cache. If you aren't writing to disk you are not a persistent database. Think memcached and redis. Although I would say the later is more of a database because you can configure it to write on every update.
The "how" is where all the devilish details are. How do you put data into and get data out of a database. How does the database store your data on disk. Well, there are two main schools of thought in the market place at the moment. Relational[0] and non-relational. Relational is the older more mature, more well supported, more well understood, more "standards" compliant of the two. Nowadays, non-relational is championed by the NoSQL[1] movement which eschews the relational model for a looser concept, less defined, less "compliant", less understood model. You will find many more and varied answers to "how" in the non-relational space precisely because the renaissance in non-relational is so fresh (the concept itself has been around a long time). Nevertheless, NoSQL has been attracting considerable attention over the last year or so in the open source community.
Every developer worth his salt will know or at least be familiar with the relational model and at least one or two of it's implementations. MySQL, PostgreSQL and SQLite are just three of the most popular open source RDBMS's (relational database management system)[2] around today. They all have varying degrees of support for the primary language used to speak to RDBMS's, SQL[3].
There are many open source, non-relational implementations in production today. HBase, Cassandra, Riak, Voldemort and MongoDB are all playing in the NoSQL field. Popular systems within NoSQL draw their lineage from either the Google approach (gfs[4], big table[5]) or the Amazon approach (dynamo[6]). Yet another is Neo4j, a graph database[7] which, at the moment, is lumped under the "NoSQL" moniker but is a separate beast entirely. Just like their relational cousins, the way NoSQL solutions do what they do and what they are best suited for vary.
As with most interesting discussions, which approach is best is left as an exercise for the reader. I'll just throw my two cents in for good measure. First I would say that you need to know SQL. Knowing SQL and not a specific system (like mysql or postgresql) will allow you to move in between relational systems and let you pick the right one for the job. After taking a look and familiarizing yourself with the SQL world, I would take a very serious look at what is going on with NoSQL. Best I can tell, there is a lot of mindshare and "cool" factor foisted upon this space right now because of the scale non-relational concepts can bring to enterprise users. Be sure, though, that rdbms is the predominant player in the database world. Every company large and small use some form of rdbms. Newer more forward looking companies that have large and very large scale needs to store and analyze vast quantities of unstructured or loosely structured data are looking more and more to non-relational NoSQL solutions.
I specifically avoid linking to particular implementations but rather link to broader concepts here. It is the concept that is more important than the implementation when learning about the big and confusing world of databases.
[0] http://en.wikipedia.org/wiki/Relational_model [1] http://en.wikipedia.org/wiki/NoSQL [2] http://en.wikipedia.org/wiki/Relational_database_management_... [3] http://en.wikipedia.org/wiki/SQL [4] http://en.wikipedia.org/wiki/Google_File_System [5] http://en.wikipedia.org/wiki/Big_table [6] http://en.wikipedia.org/wiki/Dynamo_(storage_system) [7] http://en.wikipedia.org/wiki/Graph_database