Notes from MongoDB Training

Wednesday morning

1. MongoDB Introduction

To see options to configure:

$ bin/mongod --help

Journaling (100ms default), disk flush (60s default), data file size, logfiles, etc. MongoDB writes to a memory-mapped journal file, then begins asynchronously to write to the database. In the event that the node goes down, the journal is replayed.

$ mongod -v
$ mongod -vv (up to 5 vs)

Can store these changes/settings in the configuration file, mongodb.conf. Typically, this file is on the path /etc/mongodb.conf, but in formal, data-center installations, it often ends up on a different path, e.g.: /data/mongodb/mongodb.conf.

MongoDB shell

$ mongo --port 30000

In shell, issue the following to see how MongoDB was launched:

> db.adminCommand( { getCmdLineOpts: 1 } )
{
    "argv" : [
        "/usr/bin/mongod",
        "--config",
        "/etc/mongodb.conf"
    ],
    "parsed" : {
        "config" : "/etc/mongodb.conf",
        "dbpath" : "/var/lib/mongodb",
        "logappend" : "true",
        "logpath" : "/var/log/mongodb/mongodb.log"
    },
    "ok" : 1
}

> db.adminCommand( { getCmdLineOpts: 1 } )
{
    "argv" : [
        "bin/mongod",
        "--port",
        "30000"
    ],
    "parsed" : {
        "port" : 30000
    },
    "ok" : 1
}

Everything in the shell is a command:

> db.adminCommand( { listDatabases : 1 } )
{
    "databases" : [
        {
            "name" : "training",
            "sizeOnDisk" : 218103808,
            "empty" : false
        },
        {
            "name" : "digg",
            "sizeOnDisk" : 218103808,
            "empty" : false
        },
        {
            "name" : "twitter",
            "sizeOnDisk" : 486539264,
            "empty" : false
        },
        {
            "name" : "local",
            "sizeOnDisk" : 1,
            "empty" : true
        },
        {
            "name" : "test",
            "sizeOnDisk" : 1,
            "empty" : true
        }
    ],
    "totalSize" : 922746880,
    "ok" : 1
}

...which is the long-hand for:

> show dbs
digg    0.203125GB
local    (empty)
test    (empty)
training    0.203125GB
twitter    0.453125GB

...which is a wrapper for the longer command (that organizes stuff differently).

Here's how to see what the size of a document would be:

> Object.bsonsize( { "hello" : "world" } )
22

2. CRUD and the MongoDB shell

Exercise: inserting some documents.

> db.people.insert( { "name" : "Smith", "age" : 30 } )
> for( i = 0; i < 1000; i++ ) { db.people.insert( { a : 20 } ); }
> db.people.find()
{ "_id" : ObjectId("50634c529cf47e02347c2ee2"), "a" : 20 }
{ "_id" : ObjectId("50634c529cf47e02347c2ee3"), "a" : 20 }
{ "_id" : ObjectId("50634c529cf47e02347c2ee4"), "a" : 20 }
{ "_id" : ObjectId("50634c529cf47e02347c2ee5"), "a" : 20 }
{ "_id" : ObjectId("50634c529cf47e02347c2ee6"), "a" : 20 }
{ "_id" : ObjectId("50634c529cf47e02347c2ee7"), "a" : 20 }
{ "_id" : ObjectId("50634c529cf47e02347c2ee8"), "a" : 20 }
{ "_id" : ObjectId("50634c529cf47e02347c2ee9"), "a" : 20 }
{ "_id" : ObjectId("50634c529cf47e02347c2eea"), "a" : 20 }
{ "_id" : ObjectId("50634c529cf47e02347c2eeb"), "a" : 20 }
{ "_id" : ObjectId("50634c529cf47e02347c2eec"), "a" : 20 }
{ "_id" : ObjectId("50634c529cf47e02347c2eed"), "a" : 20 }
{ "_id" : ObjectId("50634c529cf47e02347c2eee"), "a" : 20 }
{ "_id" : ObjectId("50634c529cf47e02347c2eef"), "a" : 20 }
{ "_id" : ObjectId("50634c529cf47e02347c2ef0"), "a" : 20 }
{ "_id" : ObjectId("50634c529cf47e02347c2ef1"), "a" : 20 }
{ "_id" : ObjectId("50634c529cf47e02347c2ef2"), "a" : 20 }
{ "_id" : ObjectId("50634c529cf47e02347c2ef3"), "a" : 20 }
{ "_id" : ObjectId("50634c529cf47e02347c2ef4"), "a" : 20 }
{ "_id" : ObjectId("50634c529cf47e02347c2ef5"), "a" : 20 }
Type "it" for more

Exercise, page 9

> db.exercise.insert( { "_id":"Jill", x : 1 } )
> db.exercise.findOne()
{ "_id" : "Jill", "x" : 1 }
> db.exercise.insert( { "_id":"Jill", x : 2 } )
E11000 duplicate key error index: test.exercise.$_id_  dup key: { : "Jill" }

> db.exercise.insert( { "_id" : 1.78 } )
> db.exercise.find()
{ "_id" : "Jill", "x" : 1 }
{ "_id" : 1.78 }

Auto-generated indices:

> db.system.indexes.find()
{ "v" : 1, "key" : { "_id" : 1 }, "ns" : "test.people", "name" : "_id_" }
{ "v" : 1, "key" : { "_id" : 1 }, "ns" : "test.exercise", "name" : "_id_" }

Embedded documents

> db.foo.insert( { a : { b : 2 } } )
> db.foo.find()
{ "_id" : ObjectId("506350809cf47e02347c32cb"), "a" : { "b" : 2 } }

(How to discover how 'find' works...)

> db.foo.find
function (query, fields, limit, skip, batchSize, options) {
    return new DBQuery(this._mongo, this._db, this, this._fullName, this._massageObject(query), fields, limit, skip, batchSize, options || this.getQueryOptions());
}



> db.foo.insert( { a : 100, b : 200 } )
> db.foo.find( { a : 100 } )
{ "_id" : ObjectId("506350f89cf47e02347c32cc"), "a" : 100, "b" : 200 }
> db.foo.find( { a : 100, b : 200 }, { a : true } )
{ "_id" : ObjectId("506350f89cf47e02347c32cc"), "a" : 100 }

Getting server status

> db.serverStatus()
{
    "host" : "russ-elite-book:30000",
    "version" : "2.2.0",
    "process" : "mongod",
    "pid" : 4305,
    "uptime" : 6099,
    "uptimeMillis" : NumberLong(6098936),
    "uptimeEstimate" : 6030,
    "localTime" : ISODate("2012-09-26T19:09:23.219Z"),
    "locks" : {
        "." : {
            "timeLockedMicros" : {
                "R" : NumberLong(152395),
                "W" : NumberLong(798928)
            },
            "timeAcquiringMicros" : {
                "R" : NumberLong(5791312),
                "W" : NumberLong(13902)
            }
        },
        "admin" : {
            "timeLockedMicros" : {

            },
            "timeAcquiringMicros" : {

            }
        },
        "local" : {
            "timeLockedMicros" : {
                "r" : NumberLong(5088),
                "w" : NumberLong(0)
            },
            "timeAcquiringMicros" : {
                "r" : NumberLong(829),
                "w" : NumberLong(0)
            }
        },
        "digg" : {
            "timeLockedMicros" : {
                "r" : NumberLong(12318),
                "w" : NumberLong(2280581)
            },
            "timeAcquiringMicros" : {
                "r" : NumberLong(339),
                "w" : NumberLong(29432)
            }
        },
        "test" : {
            "timeLockedMicros" : {
                "r" : NumberLong(2532),
                "w" : NumberLong(1361803)
            },
            "timeAcquiringMicros" : {
                "r" : NumberLong(144),
                "w" : NumberLong(1140)
            }
        },
        "training" : {
            "timeLockedMicros" : {
                "r" : NumberLong(3918),
                "w" : NumberLong(1443396)
            },
            "timeAcquiringMicros" : {
                "r" : NumberLong(235),
                "w" : NumberLong(2509)
            }
        },
        "twitter" : {
            "timeLockedMicros" : {
                "r" : NumberLong(3650),
                "w" : NumberLong(2556026)
            },
            "timeAcquiringMicros" : {
                "r" : NumberLong(206),
                "w" : NumberLong(837704)
            }
        }
    },
    "globalLock" : {
        "totalTime" : NumberLong("6098936000"),
        "lockTime" : NumberLong(798928),
        "currentQueue" : {
            "total" : 0,
            "readers" : 0,
            "writers" : 0
        },
        "activeClients" : {
            "total" : 0,
            "readers" : 0,
            "writers" : 0
        }
    },
    "mem" : {
        "bits" : 64,
        "resident" : 142,
        "virtual" : 1068,
        "supported" : true,
        "mapped" : 448,
        "mappedWithJournal" : 896
    },
    "connections" : {
        "current" : 2,
        "available" : 817
    },
    "extra_info" : {
        "note" : "fields vary by platform",
        "heap_usage_bytes" : 65816224,
        "page_faults" : 0
    },
    "indexCounters" : {
        "btree" : {
            "accesses" : 149948,
            "hits" : 149948,
            "misses" : 0,
            "resets" : 0,
            "missRatio" : 0
        }
    },
    "backgroundFlushing" : {
        "flushes" : 101,
        "total_ms" : 171,
        "average_ms" : 1.693069306930693,
        "last_ms" : 0,
        "last_finished" : ISODate("2012-09-26T19:08:44.294Z")
    },
    "cursors" : {
        "totalOpen" : 0,
        "clientCursors_size" : 0,
        "timedOut" : 2
    },
    "network" : {
        "bytesIn" : 108016512,
        "bytesOut" : 19095,
        "numRequests" : 75547
    },
    "opcounters" : {
        "insert" : 65438,
        "query" : 450,
        "update" : 0,
        ...
        "local" : {
            "accessesNotInMemory" : 0,
            "pageFaultExceptionsThrown" : 0
        },
        "test" : {
            "accessesNotInMemory" : 0,
            "pageFaultExceptionsThrown" : 0
        },
        "training" : {
            "accessesNotInMemory" : 0,
            "pageFaultExceptionsThrown" : 0
        },
        "twitter" : {
            "accessesNotInMemory" : 0,
            "pageFaultExceptionsThrown" : 0
        }
    },
    "ok" : 1
}

How findOne() works:

> db.foo.findOne
function (query, fields, options) {
    var cursor = this._mongo.find(this._fullName, this._massageObject(query) || {}, fields, -1, 0, 0, options || this.getQueryOptions());
    if (!cursor.hasNext()) {
        return null;
    }
    var ret = cursor.next();
    if (cursor.hasNext()) {
        throw "findOne has more than 1 result!";
    }
    if (ret.$err) {
        throw "error " + tojson(ret);
    }
    return ret;
}

Links

Wednesday afternoon

Embedded documents

> db.foo.find( { "a.b" : 2 } )
{ "_id" : ObjectId("506350809cf47e02347c32cb"), "a" : { "b" : 2 } }
> db.foo.insert( { a : { b : 2 } } )
> db.foo.find( { "a.b" : 2 } )
{ "_id" : ObjectId("506363a59cf47e02347c32ce"), "a" : { "b" : 2 } }
> db.foo.find( { a : { b : 2 } } )
{ "_id" : ObjectId("506363a59cf47e02347c32ce"), "a" : { "b" : 2 } }
> db.foo.insert( { a : { b : 2, c : 1 } } )
> db.foo.find( { a : { b : 2 } } )
{ "_id" : ObjectId("506363a59cf47e02347c32ce"), "a" : { "b" : 2 } }
> db.foo.find( { "a.b" : 2 } )
{ "_id" : ObjectId("506363a59cf47e02347c32ce"), "a" : { "b" : 2 } }
{ "_id" : ObjectId("506363d79cf47e02347c32cf"), "a" : { "b" : 2, "c" : 1 } }

This is to illustrate that a document,

{
  "a" :
  {
    "b":2,
    "c":1
  }
}

matched by { "a.b":2 } because this query is only concerned with matching that a.b is 2 and not whether the rest of the document matches. { "a":{"b":2} } says that the subdocument (underneath a) is closed to include only { "b":2 }.

Matching operators:

> db.foo.drop()
true
> db.foo.insert( { a:100, b:200 })
> db.foo.insert( { a:50, b:200 })
> db.foo.find()
{ "_id" : ObjectId("506365b29cf47e02347c32d0"), "a" : 100, "b" : 200 }
{ "_id" : ObjectId("506365b79cf47e02347c32d1"), "a" : 50, "b" : 200 }
> db.foo.find( { a:{ $gte:60 } } )
{ "_id" : ObjectId("506365b29cf47e02347c32d0"), "a" : 100, "b" : 200 }

> db.foo.find( { a:{ $in: [50, 60] } } )
{ "_id" : ObjectId("506365b79cf47e02347c32d1"), "a" : 50, "b" : 200 }

Exercises, page 11

1.

> use training
switched to db training
> show collections
scores
system.indexes
> db.scores.findOne()
{
    "_id" : ObjectId("4c90f2543d937c033f424701"),
    "kind" : "quiz",
    "score" : 50,
    "student" : 0
}
> db.scores.find( { "score" : { $lt : 65 } } )
{ "_id" : ObjectId("4c90f2543d937c033f424701"), "kind" : "quiz", "score" : 50, "student" : 0 }
{ "_id" : ObjectId("4c90f2543d937c033f424703"), "kind" : "exam", "score" : 56, "student" : 0 }
{ "_id" : ObjectId("4c90f2543d937c033f424706"), "kind" : "exam", "score" : 58, "student" : 1 }
{ "_id" : ObjectId("4c90f2543d937c033f424709"), "kind" : "exam", "score" : 53, "student" : 2 }
{ "_id" : ObjectId("4c90f2543d937c033f42470a"), "kind" : "quiz", "score" : 58, "student" : 3 }
{ "_id" : ObjectId("4c90f2543d937c033f424710"), "kind" : "quiz", "score" : 54, "student" : 5 }
{ "_id" : ObjectId("4c90f2543d937c033f424711"), "kind" : "essay", "score" : 50, "student" : 5 }
{ "_id" : ObjectId("4c90f2543d937c033f424712"), "kind" : "exam", "score" : 50, "student" : 5 }
{ "_id" : ObjectId("4c90f2543d937c033f424714"), "kind" : "essay", "score" : 53, "student" : 6 }
{ "_id" : ObjectId("4c90f2543d937c033f424715"), "kind" : "exam", "score" : 51, "student" : 6 }
{ "_id" : ObjectId("4c90f2543d937c033f424718"), "kind" : "exam", "score" : 63, "student" : 7 }
{ "_id" : ObjectId("4c90f2543d937c033f424719"), "kind" : "quiz", "score" : 57, "student" : 8 }
{ "_id" : ObjectId("4c90f2543d937c033f42471e"), "kind" : "exam", "score" : 60, "student" : 9 }
{ "_id" : ObjectId("4c90f2543d937c033f42471f"), "kind" : "quiz", "score" : 50, "student" : 10 }
{ "_id" : ObjectId("4c90f2543d937c033f424722"), "kind" : "quiz", "score" : 64, "student" : 11 }
{ "_id" : ObjectId("4c90f2543d937c033f424725"), "kind" : "quiz", "score" : 59, "student" : 12 }
{ "_id" : ObjectId("4c90f2543d937c033f424729"), "kind" : "essay", "score" : 63, "student" : 13 }
{ "_id" : ObjectId("4c90f2543d937c033f42472e"), "kind" : "quiz", "score" : 54, "student" : 15 }
{ "_id" : ObjectId("4c90f2543d937c033f424731"), "kind" : "quiz", "score" : 54, "student" : 16 }
{ "_id" : ObjectId("4c90f2543d937c033f424734"), "kind" : "quiz", "score" : 61, "student" : 17 }

2.

> db.scores.find( { } ).sort( { "score" : 1 } ).limit( 1 )
{ "_id" : ObjectId("4c90f2543d937c033f424701"), "kind" : "quiz", "score" : 50, "student" : 0 }
> db.scores.find( { } ).sort( { "score" : -1 } ).limit( 1 )
{ "_id" : ObjectId("4c90f2543d937c033f42471c"), "kind" : "quiz", "score" : 99, "student" : 9 }

3.

> db.stories.find( { "shorturl" : { $gt : { "view_count" : 1000 } } } ).count()
10000

4.

> db.stories.find( { $or : [ { "media" : "news" }, { "media" : "images" } ] }
).count()
8986
> db.stories.find( { "topic.name" : "Comedy" } ).count()
422
> db.stories.find( { $or : [ { "media" : "news" }, { "media" : "images" } ], "topic.name" :
"Comedy" } ).count()
308

5.

> db.stories.find( { $or : [ { "topic.name" : "Television" }, { "media" : "videos" } ] }
).count()
1218

Updating documents...

> db.stuff.insert( { _id:123, "foo" : "bar" } )
> db.stuff.find()
{ "_id" : 123, "foo" : "bar" }
> db.stuff.update( { _id:123 }, { "hello" : "world" } )
> db.stuff.find()
{ "_id" : 123, "hello" : "world" }

This is "replace"--uncommon. This works thus:

> db.func.update
function (query, obj, upsert, multi) {
    assert(query, "need a query");
    assert(obj, "need an object");
    var firstKey = null;
    for (var k in obj) {
        firstKey = k;
        break;
    }
    if (firstKey != null && firstKey[0] == "$") {
        this._validateObject(obj);
    } else {
        this._validateForStorage(obj);
    }
    if (typeof upsert === "object") {
        assert(multi === undefined, "Fourth argument must be empty when specifying upsert and multi with an object.");
        opts = upsert;
        multi = opts.multi;
        upsert = opts.upsert;
    }
    this._db._initExtraInfo();
    this._mongo.update(this._fullName, query, obj, upsert ? true : false, multi ? true : false);
    this._db._getExtraInfo("Updated");
}

This is upsert...

> db.stuff.update( { _id:123 }, { $set : { "foo" : "bar" } } )
> db.stuff.find()
{ "_id" : 123, "foo" : "bar", "hello" : "world" }

Pushing...

> db.stuff.insert( { a:1, b:[] } )
> db.stuff.find()
{ "_id" : 123, "foo" : "bar", "hello" : "world" }
{ "_id" : ObjectId("5063737c9cf47e02347c32d3"), "a" : 1, "b" : [] }
> db.stuff.update( { a : 1 }, { $push : { b : 2 } } )
> db.stuff.find()
{ "_id" : 123, "foo" : "bar", "hello" : "world" }
{ "_id" : ObjectId("5063737c9cf47e02347c32d3"), "a" : 1, "b" : [ 2 ] }

Updating, incrementing...

> db.stuff.insert( { _id:1, a : 10 } )
> db.stuff.find()
{ "_id" : 123, "foo" : "bar", "hello" : "world" }
{ "_id" : ObjectId("5063737c9cf47e02347c32d3"), "a" : 1, "b" : [ 2 ] }
{ "_id" : ObjectId("506373ca9cf47e02347c32d4"), "a" : 1, "c" : [ ] }
{ "_id" : 1, "a" : 10 }
> db.stuff.update( { _id:1 }, { $inc : { "a" : 5 } } )
> db.stuff.find()
{ "_id" : 123, "foo" : "bar", "hello" : "world" }
{ "_id" : ObjectId("5063737c9cf47e02347c32d3"), "a" : 1, "b" : [ 2 ] }
{ "_id" : ObjectId("506373ca9cf47e02347c32d4"), "a" : 1, "c" : [ ] }
{ "_id" : 1, "a" : 15 }

Exercises, page 12

1.

> db.scores.update( { "score" : { $gt : 90 } }, { $set : { "grade" : "A" } }, false, true
)
> db.scores.find( { "score" : { $gt : 90 } } )
{ "_id" : ObjectId("4c90f2543d937c033f42470d"), "grade" : "A", "kind" : "quiz", "score" : 98, "student" : 4 }
{ "_id" : ObjectId("4c90f2543d937c033f424713"), "grade" : "A", "kind" : "quiz", "score" : 98, "student" : 6 }
{ "_id" : ObjectId("4c90f2543d937c033f42471c"), "grade" : "A", "kind" : "quiz", "score" : 99, "student" : 9 }
{ "_id" : ObjectId("4c90f2543d937c033f424724"), "grade" : "A", "kind" : "exam", "score" : 96, "student" : 11 }
{ "_id" : ObjectId("4c90f2543d937c033f42472c"), "grade" : "A", "kind" : "essay", "score" : 92, "student" : 14 }
{ "_id" : ObjectId("4c90f2543d937c033f42472f"), "grade" : "A", "kind" : "essay", "score" : 95, "student" : 15 }
{ "_id" : ObjectId("4c90f2543d937c033f424732"), "grade" : "A", "kind" : "essay", "score" : 94, "student" : 16 }
{ "_id" : ObjectId("4c90f2543d937c033f424748"), "grade" : "A", "kind" : "exam", "score" : 92, "student" : 23 }
{ "_id" : ObjectId("4c90f2543d937c033f42474d"), "grade" : "A", "kind" : "essay", "score" : 93, "student" : 25 }
{ "_id" : ObjectId("4c90f2543d937c033f424755"), "grade" : "A", "kind" : "quiz", "score" : 98, "student" : 28 }
{ "_id" : ObjectId("4c90f2543d937c033f424758"), "grade" : "A", "kind" : "quiz", "score" : 94, "student" : 29 }
{ "_id" : ObjectId("4c90f2543d937c033f424761"), "grade" : "A", "kind" : "quiz", "score" : 98, "student" : 32 }
{ "_id" : ObjectId("4c90f2543d937c033f424764"), "grade" : "A", "kind" : "quiz", "score" : 95, "student" : 33 }
{ "_id" : ObjectId("4c90f2543d937c033f424766"), "grade" : "A", "kind" : "exam", "score" : 91, "student" : 33 }
{ "_id" : ObjectId("4c90f2543d937c033f424768"), "grade" : "A", "kind" : "essay", "score" : 98, "student" : 34 }
{ "_id" : ObjectId("4c90f2543d937c033f424774"), "grade" : "A", "kind" : "essay", "score" : 93, "student" : 38 }
{ "_id" : ObjectId("4c90f2543d937c033f424776"), "grade" : "A", "kind" : "quiz", "score" : 91, "student" : 39 }
{ "_id" : ObjectId("4c90f2543d937c033f42477a"), "grade" : "A", "kind" : "essay", "score" : 96, "student" : 40 }
{ "_id" : ObjectId("4c90f2543d937c033f42477b"), "grade" : "A", "kind" : "exam", "score" : 98, "student" : 40 }
{ "_id" : ObjectId("4c90f2543d937c033f424788"), "grade" : "A", "kind" : "quiz", "score" : 98, "student" : 45 }
Type "it" for more

Uh... this didn't really work. In fact, I goofed and got "grade" inserted, then fixed it. Trying the second half of the exercise, I never could get it to work. The query works; the update doesn't.

> db.scores.find( { $and : [ { "score" : { $lte : 90 } }, { "score" : { $gt : 80 } } ] }, {
$set : { "grade" : "B" } }, false, true )
error: { "$err" : "Unsupported projection option: grade", "code" : 13097 }

Here, I kept doing "find" instead of update (as I was copying and pasting in order not to have to re-type).

> db.scores.find( { $and : [ { "score" : { $lte : 90 } }, { "score" : { $gt : 80 } } ] }
)
{ "_id" : ObjectId("4c90f2543d937c033f424707"), "grade" : "B", "kind" : "quiz", "score" : 90, "student" : 2 }
{ "_id" : ObjectId("4c90f2543d937c033f42470f"), "grade" : "B", "kind" : "exam", "score" : 86, "student" : 4 }
{ "_id" : ObjectId("4c90f2543d937c033f4247b0"), "grade" : "B", "kind" : "essay", "score" : 85, "student" : 58 }
{ "_id" : ObjectId("4c90f2543d937c033f4247bf"), "grade" : "B", "kind" : "essay", "score" : 83, "student" : 63 }
{ "_id" : ObjectId("4c90f2543d937c033f4247d1"), "grade" : "B", "kind" : "essay", "score" : 85, "student" : 69 }
{ "_id" : ObjectId("4c90f2543d937c033f4247e6"), "grade" : "B", "kind" : "essay", "score" : 87, "student" : 76 }
{ "_id" : ObjectId("4c90f2543d937c033f4247f2"), "grade" : "B", "kind" : "essay", "score" : 84, "student" : 80 }
{ "_id" : ObjectId("4c90f2543d937c033f4247fb"), "grade" : "B", "kind" : "essay", "score" : 90, "student" : 83 }
{ "_id" : ObjectId("4c90f2543d937c033f4247fc"), "grade" : "B", "kind" : "exam", "score" : 88, "student" : 83 }
{ "_id" : ObjectId("4c90f2543d937c033f424800"), "grade" : "B", "kind" : "quiz", "score" : 90, "student" : 85 }
{ "_id" : ObjectId("4c90f2543d937c033f424803"), "grade" : "B", "kind" : "quiz", "score" : 84, "student" : 86 }
{ "_id" : ObjectId("4c90f2543d937c033f42480a"), "grade" : "B", "kind" : "essay", "score" : 90, "student" : 88 }
{ "_id" : ObjectId("4c90f2543d937c033f424817"), "grade" : "B", "kind" : "exam", "score" : 87, "student" : 92 }
{ "_id" : ObjectId("4c90f2543d937c033f424819"), "grade" : "B", "kind" : "essay", "score" : 84, "student" : 93 }
{ "_id" : ObjectId("4c90f2543d937c033f42481b"), "grade" : "B", "kind" : "quiz", "score" : 89, "student" : 94 }
{ "_id" : ObjectId("4c90f2543d937c033f42481d"), "grade" : "B", "kind" : "exam", "score" : 87, "student" : 94 }
{ "_id" : ObjectId("4c90f2543d937c033f424826"), "grade" : "B", "kind" : "exam", "score" : 87, "student" : 97 }
{ "_id" : ObjectId("4c90f2543d937c033f424829"), "grade" : "B", "kind" : "exam", "score" : 82, "student" : 98 }
{ "_id" : ObjectId("4c90f2543d937c033f42482e"), "grade" : "B", "kind" : "essay", "score" : 83, "student" : 100 }
{ "_id" : ObjectId("4c90f2543d937c033f42483b"), "grade" : "B", "kind" : "exam", "score" : 86, "student" : 104 }
Type "it" for more

2.

> db.scores.update( { "score" : { $lt : 60 } }, { $inc : { "score" : 10 } }, false, true
)
> db.scores.find( { "score" : { $lt : 60 } } )
> db.scores.find( { "score" : { $lt : 70 } } )
{ "_id" : ObjectId("4c90f2543d937c033f424701"), "kind" : "quiz", "score" : 60, "student" : 0 }
{ "_id" : ObjectId("4c90f2543d937c033f424703"), "kind" : "exam", "score" : 66, "student" : 0 }
{ "_id" : ObjectId("4c90f2543d937c033f424706"), "kind" : "exam", "score" : 68, "student" : 1 }
{ "_id" : ObjectId("4c90f2543d937c033f424709"), "kind" : "exam", "score" : 63, "student" : 2 }
{ "_id" : ObjectId("4c90f2543d937c033f42470a"), "kind" : "quiz", "score" : 68, "student" : 3 }
{ "_id" : ObjectId("4c90f2543d937c033f424710"), "kind" : "quiz", "score" : 64, "student" : 5 }
{ "_id" : ObjectId("4c90f2543d937c033f424711"), "kind" : "essay", "score" : 60, "student" : 5 }
{ "_id" : ObjectId("4c90f2543d937c033f424712"), "kind" : "exam", "score" : 60, "student" : 5 }
{ "_id" : ObjectId("4c90f2543d937c033f424714"), "kind" : "essay", "score" : 63, "student" : 6 }
{ "_id" : ObjectId("4c90f2543d937c033f424715"), "kind" : "exam", "score" : 61, "student" : 6 }
{ "_id" : ObjectId("4c90f2543d937c033f424718"), "kind" : "exam", "score" : 63, "student" : 7 }
{ "_id" : ObjectId("4c90f2543d937c033f424719"), "kind" : "quiz", "score" : 67, "student" : 8 }
{ "_id" : ObjectId("4c90f2543d937c033f42471e"), "kind" : "exam", "score" : 60, "student" : 9 }
{ "_id" : ObjectId("4c90f2543d937c033f42471f"), "kind" : "quiz", "score" : 60, "student" : 10 }
{ "_id" : ObjectId("4c90f2543d937c033f424722"), "kind" : "quiz", "score" : 64, "student" : 11 }
{ "_id" : ObjectId("4c90f2543d937c033f424725"), "kind" : "quiz", "score" : 60, "student" : 12 }
{ "_id" : ObjectId("4c90f2543d937c033f424729"), "kind" : "essay", "score" : 63, "student" : 13 }
{ "_id" : ObjectId("4c90f2543d937c033f42472e"), "kind" : "quiz", "score" : 64, "student" : 15 }
{ "_id" : ObjectId("4c90f2543d937c033f424731"), "kind" : "quiz", "score" : 64, "student" : 16 }
{ "_id" : ObjectId("4c90f2543d937c033f424734"), "kind" : "quiz", "score" : 61, "student" : 17 }
Type "it" for more

Write locking, favored over read locks, exists since 2.2 on a per database level. This means that if there's contention with lots of writes going on, it may be useful to move to a one collection per database layout for Account, Address, Payment, Partner.

How does finding a document work...

> db.tweets.find( { "user.followers_count" : 1000 } ).explain()
{
    "cursor" : "BasicCursor",
    "isMultiKey" : false,
    "n" : 8,
    "nscannedObjects" : 51428,
    "nscanned" : 51428,
    "nscannedObjectsAllPlans" : 51428,
    "nscannedAllPlans" : 51428,
    "scanAndOrder" : false,
    "indexOnly" : false,
    "nYields" : 0,
    "nChunkSkips" : 0,
    "millis" : 134,
    "indexBounds" : {

    },
    "server" : "russ-elite-book:30000"
}

Shows that we looked through 51428 documents and it took 134 milliseconds. This isn't good. Create an index:

> db.tweets.ensureIndex( { "user.followers_count" : 1 } )

> show collections
system.indexes
tweets
> db.system.indexes.find()
{ "v" : 1, "key" : { "_id" : 1 }, "ns" : "twitter.tweets", "name" : "_id_" }
{ "v" : 1, "key" : { "user.followers_count" : 1 }, "ns" : "twitter.tweets", "name" : "user.followers_count_1" }

Then, we re-run the query:

> db.tweets.find( { "user.followers_count" : 1000 } ).explain()
{
    "cursor" : "BtreeCursor user.followers_count_1",
    "isMultiKey" : false,
    "n" : 8,
    "nscannedObjects" : 8,
    "nscanned" : 8,
    "nscannedObjectsAllPlans" : 8,
    "nscannedAllPlans" : 8,
    "scanAndOrder" : false,
    "indexOnly" : false,
    "nYields" : 0,
    "nChunkSkips" : 0,
    "millis" : 0,
    "indexBounds" : {
        "user.followers_count" : [
            [
                1000,
                1000
            ]
        ]
    },
    "server" : "russ-elite-book:30000"
}

...and notice the difference. We only needed to look through 8 documents and it took "no time" at all.

3. Indexing

Multikey indices; tags is an array...

> db.foo.insert( { name:"Raleigh", "tags": [ "north", "carolina", "unc" ] } )
> db.foo.ensureIndex( { tags : 1 } )
> db.foo.find( { tags : "north" } ).explain()
{
    "cursor" : "BtreeCursor tags_1",
    "isMultiKey" : true,
    "n" : 1,
    "nscannedObjects" : 1,
    "nscanned" : 1,
    "nscannedObjectsAllPlans" : 1,
    "nscannedAllPlans" : 1,
    "scanAndOrder" : false,
    "indexOnly" : false,
    "nYields" : 0,
    "nChunkSkips" : 0,
    "millis" : 0,
    "indexBounds" : {
        "tags" : [
            [
                "north",
                "north"
            ]
        ]
    },
    "server" : "russ-elite-book:30000"
}

Only 64 indices per collection.

Polish off indices...

> db.foo.getIndexes()
[
    {
        "v" : 1,
        "key" : {
            "_id" : 1
        },
        "ns" : "twitter.foo",
        "name" : "_id_"
    },
    {
        "v" : 1,
        "key" : {
            "tags" : 1
        },
        "ns" : "twitter.foo",
        "name" : "tags_1"
    }
]
> db.system.indexes.find()
{ "v" : 1, "key" : { "_id" : 1 }, "ns" : "twitter.tweets", "name" : "_id_" }
{ "v" : 1, "key" : { "user.followers_count" : 1 }, "ns" : "twitter.tweets", "name" : "user.followers_count_1" }
{ "v" : 1, "key" : { "_id" : 1 }, "ns" : "twitter.foo", "name" : "_id_" }
{ "v" : 1, "key" : { "tags" : 1 }, "ns" : "twitter.foo", "name" : "tags_1" }
> db.foo.reIndex()...

Sparse indexing

A sparse index is when documents missing particular fields, that are explicitly referenced in an index, are not included in that index which narrows the search space. For instance, imagine the following document:

{
    title : "Don't Stop Me Now",
    artist : "Queen",
    metadata :
    {
        genre : "rock",
        length : 120,
        bps : 120,
        key : "A"
    }
}

If the index is built with the fields in the subdocument, metadata, thus:

    > db.foo.ensureIndex( { "metatdata.genre" : 1,
    > ..."metadata.length" : 1,
    > ..."metadata.bps" : 1,
    > ..."metadata.key" : 1 },
    > ...{ sparse : true } } )

...those titles for which there's no record of what key they are in will not encumber the index.

The problem with doing this is, however, that as new fields are added conveniently to the schema, all the collection must be traversed for each new index added. The advantages to sparse indices are:

• normal schema
• simplicity in update operations

...while the disadvantages are:

• need many indices (with impact on memory)
• indices are built at runtime (with impact on processing)

Multikeying

One solution to this problem, to continue the example, is multikeying, which groups elements in an array (instead of a subdocument as here) making the metadata items here just part of the same, single field:

{
    title : "Don't Stop Me Now",
    artist : "Queen",
    metadata :
    [
        { genre : "rock" },
        { length : 120 },
        { bps : 120 },
        { key : "A" }
    ]
}

The index is built and finds conducted:

    > db.foo.ensureIndex( { "metatdata" : 1 } )
    > db.foo.find( { metatdata : { duration : 120 } } )

There is a different problem with this approach. It is that anything, but a direct match on an element will lead to reverting to the basic cursor or simple traversal of the entire collection again. The advantages of this approach were:

• single index for multikey query
• index works for exact queries

...but the disadvantages are:

• updating is more difficult, though it can be done using the $pos operator
• this approach doesn't work with an arbitrary query using $elemMatch

Structured multikeying

To have but a single index, but one that's powerful enough to match on fields no matter how the fields are represented, use this approach. In essence it's a an array of key/value pairs.

{
    title : "Don't Stop Me Now",
    artist : "Queen",
    metadata :
    [
        { key : "genre", value : "rock" },
        { key : "length", value : 120 },
        { key : "bps", value : 120 },
        { key : "key", value : "A" }
    ]
}

The index is built thus:

    > db.foo.ensureIndex( { "metatdata.key" : 1, "metadata.value" : 1 } )
    > db.foo.find( { metatdata : { duration : 120 } } )

The advantages to this approach are:

• queries can be arbitrary using $elemMatch
• only a single index is needed

...while the (inevitable) disadvantages are:

• more, and more complex, data are needed for the same representation
• updating is more difficult

The new document is uglier and less straighforward; storage is amplified as "key" and "value" are everywhere repeated over and over.

4. Schema design

Thursday morning

5. Introduction to the drivers

(I took few notes because I'm really a Java guy.)

5. Durability, Availability and Replica Sets

(Yup, the duplicate section number was in fact in the otherwise very nice 10gen manual too. Sorry.)

There's a heartbeat that goes around between mongod processes.

Replication happens in a few seconds or a few minutes.

The primary replica is "elected" by all the active nodes.

You can only read and write off the primary.

If there's no majority and the primary isn't up, then the system goes into a WAIT state.

How to elect a new primary...

1. Latest oplog entry.
2. Priority.

After the former primary comes back up, it's only a secondary..

An arbiter is a node that isn't a replica (performs no database function), but can vote for a new primary.

By default, all replicas start out with equal priority (1).

A capped collection is a circular buffer.

You can copy the database by freezing the replica for a sufficient number of seconds during the copy (cp -R).

Thursday afternoon

> db.runCommand( { getLastError:1, w:3, wtimeout:1000 } )

--must not return until back from committing to at least 3 replicas.

The oplog...

What's in the database?

db
|— digg.0
|— digg.1
|— digg.ns
|— journal
|   |— j._0
|   |— prealloc.1
|   `— prealloc.2
|— mongod.lock
|— test.0
|— test.1
|— test.ns
|— training.0
|— training.1
|— training.ns
|— twitter.0
|— twitter.1
|— twitter.2

`— twitter.ns

Exercises, page 23

1.

Three.

2.

3.

Think about the scenarios above and the answers below.

4.

Three and two.

5.

Nothing can happen as there aren't enough voting members.

6.

?

Exercise, page 26

erect-replicas.sh:

#!/bin/sh
# subdirectories data/rs1, rs2 and rs3 must already exist.
../bin/mongod --port 30000 --replSet foo --logpath "1.log" --dbpath /data/rs1 --fork
../bin/mongod --port 30001 --replSet foo --logpath "2.log" --dbpath /data/rs2 --fork
../bin/mongod --port 30002 --replSet foo --logpath "3.log" --dbpath /data/rs3 --fork

replica-config.js:

It doesn't do any good to run this against the mongo shell from the command line as then launching the shell, config will still not be defined. Instead, simply copy and paste it into the shell.

config =
    { _id:"foo", members:
        [
            { _id:0, host:"localhost:30000" },
            { _id:1, host:"localhost:30001" },
            { _id:2, host:"localhost:30002" }
        ]
    }

Replica instructions

1. Create the new subdirectories.
2. Run erect-replicas.sh to launch the mongod processes for each (this takes a long time).
3. Launch the mongo shell, copy and paste in the replica config

   $ mongo --port 30000
   > (paste replica-config.js here so that config is defined)
   

4. Initiate the replicas (this takes a long time):

   > rs.initiate( config )
   

5. See the status:

   > rs.status()
   

Here's the illustration...

bash shell work...

~/mongo-training/data $ ./erect-replicas.sh
forked process: 5669
all output going to: /home/russ/mongo-training/data/1.log
log file [/home/russ/mongo-training/data/1.log] exists; copied to temporary file [/home/russ/mongo-training/data/1.log.2012-09-27T21-04-17]
child process started successfully, parent exiting
forked process: 5717
all output going to: /home/russ/mongo-training/data/2.log
log file [/home/russ/mongo-training/data/2.log] exists; copied to temporary file [/home/russ/mongo-training/data/2.log.2012-09-27T21-04-17]
forked process: 5724
all output going to: /home/russ/mongo-training/data/3.log
log file [/home/russ/mongo-training/data/3.log] exists; copied to temporary file [/home/russ/mongo-training/data/3.log.2012-09-27T21-04-17]

Mongo shell work...

~/mongo-training/data $ ../bin/mongo --port 30000
MongoDB shell version: 2.2.0
connecting to: 127.0.0.1:30000/test
> config =
... { _id:"foo", members:
... [
... { _id:0, host:"localhost:30000" },
... { _id:1, host:"localhost:30001" },
... { _id:2, host:"localhost:30002" }
... ]
... }
{
    "_id" : "foo",
    "members" : [
        {
            "_id" : 0,
            "host" : "localhost:30000"
        },
        {
            "_id" : 1,
            "host" : "localhost:30001"
        },
        {
            "_id" : 2,
            "host" : "localhost:30002"
        }
    ]
}
> rs.initiate( config )
{
    "info" : "Config now saved locally.  Should come online in about a minute.",
    "ok" : 1
}
> rs.status()
{
    "set" : "foo",
    "date" : ISODate("2012-09-27T21:12:22Z"),
    "myState" : 1,
    "members" : [
        {
            "_id" : 0,
            "name" : "localhost:30000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 485,
            "optime" : Timestamp(1348780280000, 1),
            "optimeDate" : ISODate("2012-09-27T21:11:20Z"),
            "self" : true
        },
        {
            "_id" : 1,
            "name" : "localhost:30001",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 62,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeat" : ISODate("2012-09-27T21:12:21Z"),
            "pingMs" : 62
        },
        {
            "_id" : 2,
            "name" : "localhost:30002",
            "health" : 1,
            "state" : 3,
            "stateStr" : "RECOVERING",
            "uptime" : 62,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeat" : ISODate("2012-09-27T21:12:22Z"),
            "pingMs" : 0
        }
    ],
    "ok" : 1
}
foo:PRIMARY>

Starting a replica...

$ mongod --port 30000

Looking at replica status...

foo:PRIMARY> rs.conf()
{
    "_id" : "foo",
    "version" : 1,
    "members" : [
        {
            "_id" : 0,
            "host" : "localhost:30000"
        },
        {
            "_id" : 1,
            "host" : "localhost:30001"
        },
        {
            "_id" : 2,
            "host" : "localhost:30002"
        }
    ]
}

How to modify a configuration element. See here:

foo:PRIMARY> rs.conf()
{
    "_id" : "foo",
    "version" : 1,
    "members" : [
        {
            "_id" : 0,
            "host" : "localhost:30000"
        },
        {
            "_id" : 1,
            "host" : "localhost:30001"
        },
        {
            "_id" : 2,
            "host" : "localhost:30002"
        }
    ]
}
foo:PRIMARY> var config=rs.conf()
foo:PRIMARY> config.members[ 2 ].priority = 0
0
foo:PRIMARY> rs.reconfig( config )
Thu Sep 27 16:00:47 DBClientCursor::init call() failed
Thu Sep 27 16:00:47 query failed : admin.$cmd { replSetReconfig: { _id: "foo", version: 2, members: [ { _id: 0, host: "localhost:30000" }, { _id: 1, host: "localhost:30001" }, { _id: 2, host: "localhost:30002", priority: 0.0 } ] } } to: 127.0.0.1:30000
Thu Sep 27 16:00:47 trying reconnect to 127.0.0.1:30000
Thu Sep 27 16:00:47 reconnect 127.0.0.1:30000 ok
reconnected to server after rs command (which is normal)

foo:PRIMARY> rs.conf()
{
    "_id" : "foo",
    "version" : 2,
    "members" : [
        {
            "_id" : 0,
            "host" : "localhost:30000"
        },
        {
            "_id" : 1,
            "host" : "localhost:30001"
        },
        {
            "_id" : 2,
            "host" : "localhost:30002",
            "priority" : 0
        }
    ]
}
foo:PRIMARY>

As soon as you reconfigure, the shell loses connection with the primary until it reconnects.

foo:PRIMARY> rs.status()
{
    "set" : "foo",
    "date" : ISODate("2012-09-27T22:02:39Z"),
    "myState" : 1,
    "members" : [
        {
            "_id" : 0,
            "name" : "localhost:30000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 3502,
            "optime" : Timestamp(1348783247000, 1),
            "optimeDate" : ISODate("2012-09-27T22:00:47Z"),
            "self" : true
        },
        {
            "_id" : 1,
            "name" : "localhost:30001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 110,
            "optime" : Timestamp(1348783247000, 1),
            "optimeDate" : ISODate("2012-09-27T22:00:47Z"),
            "lastHeartbeat" : ISODate("2012-09-27T22:02:39Z"),
            "pingMs" : 0,
            "errmsg" : "syncing to: localhost:30000"
        },
        {
            "_id" : 2,
            "name" : "localhost:30002",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 112,
            "optime" : Timestamp(1348783247000, 1),
            "optimeDate" : ISODate("2012-09-27T22:00:47Z"),
            "lastHeartbeat" : ISODate("2012-09-27T22:02:39Z"),
            "pingMs" : 0,
            "errmsg" : "syncing to: localhost:30000"
        }
    ],
    "ok" : 1
}

The errmsg isn't really an error at all.

Hiding a node...

foo:PRIMARY> var config=rs.conf()
foo:PRIMARY> config.members[ 2 ].hidden = true
true
foo:PRIMARY> rs.reconfig( config )
Thu Sep 27 16:08:06 DBClientCursor::init call() failed
Thu Sep 27 16:08:06 query failed : admin.$cmd { replSetReconfig: { _id: "foo", version: 3, members: [ { _id: 0, host: "localhost:30000" }, { _id: 1, host: "localhost:30001" }, { _id: 2, host: "localhost:30002", priority: 0.0, hidden: true } ] } } to: 127.0.0.1:30000
Thu Sep 27 16:08:06 trying reconnect to 127.0.0.1:30000
Thu Sep 27 16:08:06 reconnect 127.0.0.1:30000 ok
reconnected to server after rs command (which is normal)

foo:PRIMARY> rs.conf()
{
    "_id" : "foo",
    "version" : 3,
    "members" : [
        {
            "_id" : 0,
            "host" : "localhost:30000"
        },
        {
            "_id" : 1,
            "host" : "localhost:30001"
        },
        {
            "_id" : 2,
            "host" : "localhost:30002",
            "priority" : 0,
            "hidden" : true
        }
    ]
}
foo:PRIMARY>

Looking at the oplog...

foo:PRIMARY> use local
switched to db local
foo:PRIMARY> show tables
oplog.rs
slaves
system.indexes
system.replset
foo:PRIMARY> db.me.find()
foo:PRIMARY> use twitter
switched to db twitter
foo:PRIMARY> db.tweets.insert( { "name", "Smith" } )
Thu Sep 27 16:11:28 SyntaxError: missing : after property id (shell):1
foo:PRIMARY> use local
switched to db local
foo:PRIMARY> show collections
oplog.rs
slaves
system.indexes
system.replset
foo:PRIMARY> db.oplog.rs.find()
{ "ts" : Timestamp(1348780280000, 1), "h" : NumberLong(0), "op" : "n", "ns" : "", "o" : { "msg" : "initiating set" } }
{ "ts" : Timestamp(1348783247000, 1), "h" : NumberLong("6246938530832973841"), "op" : "n", "ns" : "", "o" : { "msg" : "Reconfig set", "version" : 2 } }
{ "ts" : Timestamp(1348783686000, 1), "h" : NumberLong("-9112696901504822284"), "op" : "n", "ns" : "", "o" : { "msg" : "Reconfig set", "version" : 3 } }

bind_ip