Document Databases
NoSQL Databases
Lecture 6 of NoSQL Databases (PA195)
David Novak & Vlastislav Dohnal
Faculty of Informatics, Masaryk University, Brno
Agenda
● Text (Document) Data Types
○ JSON: JavaScript Object Notation
○ XML: usage and comparison with JSON
● Document Databases: MongoDB
○ Database schema: Design
○ Using MongoDB: Updates, Queries, Indexes
○ Behind the scene
■ BSON format, Distribution, Replication, Transactions, ...
2
NoSQL Databases and Data Types
1. Key-value stores:
○ Can store any (text or binary) data
■ often, if using JSON data, additional functionality is available
2. Document databases
○ Structured text data - Hierarchical tree data structures
■ typically JSON, XML
3. Column-family stores
○ Rows that have many columns associated with a row key
■ can be written as JSON
3
Part 1: Document Data Types
4
Data Formats
● Binary Data (previous lecture)
○ often, we want to store objects (class instances)
○ objects can be binary serialized (marshalled)
■ and kept in a key-value store
○ there are several popular serialization formats
■ Protocol Buffers, Apache Thrift
● Structured Text Data
○ JSON, BSON (Binary JSON)
■ JSON is currently number one data format used on the Web
○ XML: eXtensible Markup Language
○ RDF: Resource Description Framework
5
JSON: Basic Information
● Text-based open standard for data interchange
○ Serializing and transmitting structured data
○ ECMA-404 standard
● JSON = JavaScript Object Notation
○ Originally specified by Douglas Crockford in 2001
○ Derived from JavaScript scripting language
○ Uses conventions of the C-family of languages
● Filename: *.json
● Internet media (MIME) type: application/json
● Language independent https://www.json.org 6
JSON:Example
source: I. Holubová, J. Kosek, K. Minařík, D. Novák. Big Data a NoSQL databáze. Praha: Grada Publishing, 2015. 7
JSON: Data Types (1)
● object – an unordered set of name:value pairs
○ these pairs are called properties (members) of an object
○ syntax: { name: value, name: value, name: value, ...}
● array – an ordered collection of values (elements)
○ syntax: [ comma-separated values ]
8
JSON: Data Types (2)
● value – string in double quotes / number / true
or false (i.e., Boolean) / null / object / array
9
JSON: Data Types (3)
● string – sequence of zero or more Unicode
characters, wrapped in double quotes
○ Backslash escaping
10
JSON: Data Types (4)
● number – like a C or Java number
○ Integer or floating-point
○ Octal and hexadecimal formats are not used
11
JSON Properties
● There is no way to write comments in JSON
○ Originally, there was but it was removed for security
● No way to specify precision/size of numbers
○ It depends on the parser and the programming language
● There exists a standard “JSON Schema”
○ A way to specify the schema of the data
○ Field names, field types, required/optional fields, etc.
○ JSON Schema is written in JSON, of course
■ see example below
12
JSON Schema: Example
source: I. Holubová, J. Kosek, K. Minařík, D. Novák. Big Data a NoSQL databáze. Praha: Grada Publishing, 2015. 13
Document with JSON Schema
source: I. Holubová, J. Kosek, K. Minařík, D. Novák. Big Data a NoSQL databáze. Praha: Grada Publishing, 2015. 14
XML: Basic Information
● XML: eXtensible Markup Language
○ W3C standard (since 1996)
● both human and
machine readable
● example:
source: http://en.wikipedia.org/wiki/XML 15
XML: Features and Comparison
● Standard ways to specify XML document schema:
○ DTD, XML Schema (XSD), etc.
○ concept of Namespaces; XML editors (for given schema)
● Technologies for parsing: DOM, SAX
● Many associated technologies:
○ XPath, XQuery, XSLT (transformation)
● XML is great for configurations, meta-data, etc.
● XML databases are mature, not considered NoSQL
● Currently, JSON format rules:
○ compact, easier to write, has all features typically needed 16
Part 2: Document Databases
17
Document Databases: Fundamentals
● Basic concept of data: Document
● Documents are self-describing pieces of data
○ Hierarchical tree data structures
○ Nested associative arrays (maps), collections, scalars
○ XML, JSON (JavaScript Object Notation), BSON, …
● Documents in a collection should be “similar”
○ Their schema can differ
● Often: Documents stored as values of key-value
○ Key-value stores where the values are examinable
○ Building search indexes on various keys/fields of the value 18
Why Document Databases
● XML and JSON are popular for data exchange
○ Recently mainly JSON
● Data stored in document DB can be used directly
● Databases often store objects from memory
○ Using RDBMS, we must do Object Relational Mapping (ORM)
■ ORM is relatively demanding
○ JSON is much closer to the structure of memory objects
■ It was originally for JavaScript objects
■ Object Document Mapping (ODM) is faster
19
Document Databases: Representatives
Ranked list: http://db-engines.com/en/ranking/document+store
MS Azure
DocumentDB
20
Part 2.1: MongoDB - Basics & Querying
21
MongoDB
● Initial release: 2009
○ Written in C++
○ Open-source
○ Cross-platform
● JSON documents
● Basic features:
○ High performance – many indexes
○ High availability – replication + eventual consistency +
automatic failover
○ Automatic scaling – automatic sharding across the cluster
○ MapReduce support
https://www.mongodb.com/ 22
MongoDB: Terminology
● each JSON document:
○ belongs to a collection
○ has a field _id
■ unique within the collection
● each collection:
○ belongs to a “database”
RDBMS MongoDB
database instance database
schema ---
table collection
row document
rowid _id
https://www.mongodb.com/ 23
Documents
● Use JSON for API communication
● Internally: BSON
○ Binary representation of JSON
○ For storage and inter-server communication
● Document has a maximum size: 16MB (in BSON)
○ Not to use too much RAM, bandwidth
○ GridFS tool can divide larger files into fragments
24
Document Fields
● Every document must have the field _id
○ Used as a primary key
○ Unique within the collection
○ Immutable
○ Any type other than an array
○ Can be generated automatically
● Restrictions on field names:
○ The field names cannot start with the $ character
■ Reserved for operators
○ The field names cannot contain the . character
■ Reserved for accessing sub-fields
25
Database Schema
● Documents have flexible schema
○ Collections do not enforce specific data structure
○ In practice, documents in a collection are similar
● Key decision of data modeling:
○ References vs. embedded documents
○ In other words: Where to draw lines between aggregates
■ Structure of data
■ Relationships between data
26
Schema: Embedded Docs
● Related data in a single document structure
○ Documents can have subdocuments (in a field or array)
https://www.mongodb.com/ 27
Schema: Embedded Docs (2)
● Denormalized schema
● Main advantage:
Manipulate related data in a single operation
● Use this schema when:
○ One-to-one relationships: one doc “contains” the other
○ One-to-many: if children docs have one parent document
● Disadvantages:
○ Documents may grow significantly during the time
○ Impacts both read/write performance
■ Document must be relocated on disk if its size exceeds allocated space
■ May lead to data fragmentation on the disk 28
Schema: References
https://www.mongodb.com/
● Links/references from one document to another
● Normalization of the schema
29
Schema: References (2)
● More flexibility than embedding
● Use references:
○ When embedding would result in duplication of data
■ and only insignificant boost of read performance
○ To represent more complex many-to-many relationships
○ To model large hierarchical data sets
● Disadvantages:
○ Can require more roundtrips to the server
■ Documents are accessed one by one
30
Querying: Basics
● Mongo query language
● A MongoDB query:
○ Targets a specific collection of documents
○ Specifies criteria that identify the returned documents
○ May include a projection to specify returned fields
○ May impose limits, sort, orders, …
● Basic query - all documents in the collection:
db.users.find()
db.users.find( {} )
31
Querying: Example
https://www.mongodb.com/ 32
Querying: Selection
db.inventory.find({ type: "snacks" })
● All documents from collection inventory where the type field
has the value snacks
db.inventory.find({ type: { $in: [ 'food',
'snacks' ] } } )
● All inventory docs where the type field is either food or snacks
db.inventory.find( { type: 'food', price: {
$lt: 9.95 } } )
● All ... where the type field is food and the price is less than 9.95
33
Inserts
db.inventory.insertOne( { _id: 10, type:
"misc", item: "card", qty: 15 } )
● Inserts a document with three fields into collection inventory
○ User-specified _id field
db.inventory.insertOne( { type: "book", item:
"journal" } )
● The database generates _id field
$ db.inventory.find()
{ _id: ObjectId("58e209ecb3e168f1d3915300"),
type: "book", item: "journal" }
34
Updates
db.inventory.updateMany(
{ type: "book", item : "journal" },
{ $set: { qty: 10 } },
{ upsert: true } )
● Finds all docs matching query
{ type: "book", item : "journal" }
● and sets the field { qty: 10 }
● upsert: true
○ if no document in the inventory collection matches
○ creates a new document (generated _id)
■ it contains fields _id, type, item, qty 35
MapReduce
collection "accesses":
{
"user_id": <ObjectId>,
"login_time": <time_the_user_entered_the_system>,
"logout_time": <time_the_user_left_the_system>,
"access_type": <type_of_the_access>
}
● How much time did each user spend logged in
○ Counting just accesses of type “regular”
db.accesses.mapReduce(
function() { emit (this.user_id, this.logout_time - this.login_time); },
function(key, values) { return Array.sum( values ); },
{
query: { access_type: "regular" },
out: "access_times"
}
)
36
In JavaScript
Part 2.2: MongoDB - Indexes
Indexes
● Indexes are the key for MongoDB performance
○ Without indexes, MongoDB must scan every document in a
collection to select matching documents
● Indexes store some fields in easily accessible form
○ Stores values of a specific field(s) ordered by the value
● Defined per collection
● Purpose:
○ To speed up common queries
○ To optimize performance of other specific operations
38
Indexes: Example of Use
https://www.mongodb.com/ 39
Indexes: Example of Use (2)
● The index can be traversed in order to return
sorted results (without sorting)
https://www.mongodb.com/ 40
Indexes: Example of Use (3)
● MongoDB does not need to inspect data outside
of the index to fulfill the query
http://www.mongodb.org/ 41
Index Types
● Default: _id
○ Exists by default
■ If applications do not specify _id when inserting a doc, it is created.
○ Unique
● Single Field
○ User-defined indexes on a single field of a document
● Compound
○ User-defined indexes on multiple fields
● Multikey index
○ To index the content stored in arrays
○ Creates separate index entry for each array element
42
Index Types (2)
● Index on score
field (ascending)
https://www.mongodb.com/
● Compound Index
on userid
(ascending) AND
score field
(descending)
● Multikey index on
the addr.zip field
43
Index Types (3)
● Ordered Index
○ B+-Tree (see above)
● Hashed Indexes
○ Fast O(1) indexes the hash of the value of a field
■ Only equality matches
● Geospatial Index (operators docs)
○ 2d indexes = use planar geometry when returning results
■ For data representing points on a two-dimensional plane
○ 2dsphere indexes = spherical (Earth-like WGS84) geometry
■ For data representing latitude, longitude
● Text Indexes
○ Searching for string content in a collection 44
Part 2.3: MongoDB - Behind the Scene
MongoDB: Behind the Scene
● BSON format
● Distribution models
○ Replication
○ Sharding
○ Balancing
● MapReduce
● Transactions
● Journaling
46
BSON (Binary JSON) Format
● Binary-encoded serialization of JSON documents
○ Representation of documents, arrays, JSON simple data
types + other types (e.g., Date and BinData)
https://www.bsonspec.org/ 47
Value length in bytes
Data type
(string, here)
64b floating
point type 32b int type
BSON: Basic Types
● byte – 1 byte (8-bits)
● int32 – 4 bytes (32-bit signed integer)
● int64 – 8 bytes (64-bit signed integer)
● double – 8 bytes (64-bit IEEE 754 floating point)
http://www.bsonspec.org/ 48
BSON Grammar
document ::= int32 e_list "\x00"
● BSON document
● int32 = total number of bytes in document
e_list ::= element e_list | ""
● Sequence of elements
http://www.bsonspec.org/ 49
BSON Grammar (2)
element ::= "\x01" e_name double
| "\x02" e_name string
| "\x03" e_name document
| "\x04" e_name document
| "\x05" e_name binary
| …
Floating point
UTF-8 string
Embedded document
Array
Binary data
…
e_name ::= cstring
● Field key
cstring ::= (byte*) "\x00"
string ::= int32 (byte*) "\x00"
50
Data Replication
● Master/slave replication
● Replica set = a group of
instances that host the
same data set
○ primary (master) – handles
all write operations
○ secondaries (slaves) –
apply operations from the
primary so that they have
the same data set
51
Replication: Read & Write
● Write operation:
1. Write operation is applied on the primary
2. Operation is recorded to primary’s oplog (operation log)
3. Secondaries replicate the oplog + apply the operations to
their data sets
● Read: All replica set members can accept reads
○ By default, application directs its reads to the primary
■ Guaranties the latest version of a document
■ Decreases read throughput
○ Read preference mode can be set
■ See below
52
Replication: Read Modes
Read Preference
Mode
Description
primary operations read from the primary of the replica set
primaryPreferred operations read from the primary, but if unavailable,
operations read from secondary members
secondary operations read from the secondary members
secondaryPreferred operations read from secondary members, but if
none is available, operations read from the primary
nearest operations read from the nearest member (= shortest
ping time) of the replica set
53
Replica Set Elections
● If the primary
becomes unavailable,
an election
determines a new
primary
○ Elections need some
time
○ No primary =>
no writes
54
Replica Set: CAP
● Let us have three nodes in the replica set
○ Let’s say that the master is disconnected from the other two
■ The distributed system is partitioned
○ The two slaves “think” that the master failed
■ Because they form a partition with more than half of the nodes
■ And elect a new master
○ The master finds out, that it is alone
■ Specifically, that can communicate with less than half of the nodes
■ And it steps down from being master (handles just reads)
● In case of just two nodes in RS
○ Both partitions will become read-only
■ Similar case can occur with any even number of nodes in RS
○ Therefore, we can always add an arbiter node to even-sized
RS
55
Sharding
● MongoDB enables
collection partitioning
(sharding)
56
Collection Partitioning
● Mongo partitions collection’s data by the shard key
○ shard key = field(s) that exist in each document in the
collection, it is indexed
■ Since Mongo 4.2, the value is mutable
■ Since Mongo 4.4, the shard key can be missing -> NULL is then used
○ Divided into chunks, distributed across shards
■ Range-based partitioning
■ Hash-based partitioning
○ When a chunk grows beyond
the size limit, it is split
■ Metadata change, no data migration
● Data balancing:
○ Background chunk migration 57
Sharding: Components
● MongoDB runs in cluster of different node types:
● Shards – store the data
○ Each shard is a replica set
■ Can be a single node
● Mongos (Query routers) – interface btw. client
applications and sharded cluster
○ Direct operations to the relevant shard(s)
■ + return the result to the client
○ More than one => to divide the client request load
● Config servers – store the cluster’s metadata
○ Mapping of the cluster’s data set to the shards
○ Recommended number: 3 58
Sharding: Diagram
59
Journaling
● Write operations are applied in memory and into
a journal before done in the data files (on disk)
○ To restore a consistent state after a hard shutdown
○ Can be switched on/off
● Journal directory – holds journal files
● Journal file = write-ahead redo logs
○ Append only file
○ Deleted when all the writes are durable
○ When size > 1GB of data, MongoDB creates a new file
■ The size can be modified
● Clean shutdown removes all journal files 60
$isolated operator
is deprecated
Transactions
● Write ops: atomic at the level of single document
○ Including nested documents
○ Sufficient for many cases, but not all
○ When a write operation modifies multiple documents,
other operations may interleave
● Transactions: (docs)
○ Isolation of a write operation that affects multiple docs
db.foo.update( { field1 : 1 , $isolated : 1 }, { $inc
: { field2 : 1 } } , { multi: true } )
○ Two-phase commit
■ Multi-document updates
■ In a session (.start/endSession), do .start/abort/commitTransaction 61
Summary
● Concepts of text documents
○ JSON vs. XML
○ binary format of JSON
● MongoDB principles
○ data manipulation
○ replication
○ distribution
○ transactions
62
Questions?
Please, any questions? Good question is a gift...
Spotted a bug/typo? Report it please.
63
References
● I. Holubová, J. Kosek, K. Minařík, D. Novák. Big Data a
NoSQL databáze. Praha: Grada Publishing, 2015. 288 p.
● Sadalage, P. J., & Fowler, M. (2012). NoSQL Distilled: A
Brief Guide to the Emerging World of Polyglot
Persistence. Addison-Wesley Professional, 192 p.
● RNDr. Irena Holubova, Ph.D. MMF UK course NDBI040:
Big Data Management and NoSQL Databases
● MongoDB Manual: http://docs.mongodb.org/manual/
64