Docstore

The docstore package provides an abstraction layer over common document stores like Google Cloud Firestore, Amazon DynamoDB and MongoDB. This guide shows how to work with document stores in the Go CDK.

A document store is a service that stores data in semi-structured JSON-like documents grouped into collections. Like other NoSQL databases, document stores are schemaless.

The docstore package supports operations to add, retrieve, modify and delete documents.

Subpackages contain driver implementations of docstore for various services, including Cloud and on-prem solutions. You can develop your application locally using memdocstore, then deploy it to multiple Cloud providers with minimal initialization reconfiguration.

Opening a Collection🔗

The first step in interacting with a document store is to instantiate a portable *docstore.Collection for your service.

While every docstore service has the concept of a primary key that uniquely distinguishes a document in a collection, each one specifies that key in its own way. To be portable, Docstore requires that the key be part of the document’s contents. When you open a collection using one of the functions described here, you specify how to find the provider’s primary key in the document.

The easiest way to open a collection is using docstore.OpenCollection and a service-specific URL pointing to it, making sure you “blank import” the driver package to link it in.

import (
    "gocloud.dev/docstore"
    _ "gocloud.dev/docstore/<driver>"
)
...
coll, err := docstore.OpenCollection(context.Background(), "<driver-url>")
if err != nil {
    return fmt.Errorf("could not open collection: %v", err)
}
defer coll.Close()
// coll is a *docstore.Collection; see usage below
...

See Concepts: URLs for general background and the guide below for URL usage for each supported service.

Alternatively, if you need fine-grained control over the connection settings, you can call the constructor function in the driver package directly (like mongodocstore.OpenCollection).

import "gocloud.dev/docstore/<driver>"
...
coll, err := <driver>.OpenCollection(...)
...

You may find the wire package useful for managing your initialization code when switching between different backing services.

See the guide below for constructor usage for each supported service

Using a Collection🔗

Representing Documents🔗

We’ll use a collection with documents represented by this Go struct:

type Player struct {
    Name             string
    Score            int
    DocstoreRevision interface{}
}

We recommend using structs for documents because they impose some structure on your data, but Docstore also accepts map[string]interface{} values. See the docstore package documentation for more information.

The DocstoreRevision field holds information about the latest revision of the document. We discuss it below.

Actions🔗

Once you have opened a collection, you can call action methods on it to read, modify and write documents. You can execute a single action, or run multiple actions together in an [action list]({{ ref “act-list” }}).

Docstore supports six kinds of actions on documents:

• Get retrieves a document.
• Create creates a new document.
• Replace replaces an existing document.
• Put puts a document whether or not it already exists.
• Update applies a set of modifications to a document.
• Delete deletes a document.

You can create a single document with the Collection.Create method, we will use coll as the variable holding the collection throughout the guide:

err := coll.Create(ctx, &Player{Name: "Pat", Score: 10})
if err != nil {
    return err
}

Action Lists🔗

When you use an action list to perform multiple actions at once, drivers can optimize action lists by using bulk RPCs, running the actions concurrently, or employing a provider’s special features to improve efficiency and reduce cost. Here we create several documents using an action list.

import (
	"context"

	"gocloud.dev/docstore"
)

// Build an ActionList to create several new players, then execute it.
// The actions may happen in any order.
newPlayers := []string{"Pat", "Mel", "Fran"}
actionList := coll.Actions()
for _, p := range newPlayers {
	actionList.Create(&Player{Name: p, Score: 0})
}
if err := actionList.Do(ctx); err != nil {
	return err
}

ActionList has a fluent API, so you can build and execute a sequence of actions in one line of code. Here we Put a document and immediately Get its new contents.

import (
	"context"
	"fmt"

	"gocloud.dev/docstore"
)

// Add a document to the collection, then retrieve it.
// Because both the Put and the Get refer to the same document,
// they happen in order.
got := Player{Name: "Pat"}
err := coll.Actions().Put(&Player{Name: "Pat", Score: 88}).Get(&got).Do(ctx)
if err != nil {
	return err
}
fmt.Println(got.Name, got.Score)

If the underlying provider is eventually consistent, the result of the Get might not reflect the Put. Docstore only guarantees that it will perform the Get after the Put completes.

See the documentation for docstore.ActionList for the semantics of action list execution.

Updates🔗

Use Update to modify individual fields of a document. The Update action takes a set of modifications to document fields, and applies them all atomically. You can change the value of a field, increment it, or delete it.

import (
	"context"

	"gocloud.dev/docstore"
)

// Create a player.
pat := &Player{Name: "Pat", Score: 0}
if err := coll.Create(ctx, pat); err != nil {
	return err
}

// Set the score to a new value.
pat2 := &Player{Name: "Pat"}
err := coll.Actions().Update(pat, docstore.Mods{"Score": 15}).Get(pat2).Do(ctx)
if err != nil {
	return err
}

// Increment the score.
err = coll.Actions().Update(pat, docstore.Mods{"Score": docstore.Increment(5)}).Get(pat2).Do(ctx)
if err != nil {
	return err
}

Queries🔗

Docstore’s Get action lets you retrieve a single document by its primary key. Queries let you retrieve all documents that match some conditions. You can also use queries to delete or update all documents that match the conditions.

Getting Documents🔗

Like actions, queries are built up in a fluent style. Just as a Get action returns one document, the Query.Get method returns several documents, in the form of an iterator.

iter := coll.Query().Where("Score", ">", 20).Get(ctx)
defer iter.Stop() // Always call Stop on an iterator.

Repeatedly calling Next on the iterator will return all the matching documents. Like the Get action, Next will populate an empty document that you pass to it:

doc := &Player{}
err := iter.Next(ctx, doc)

The iteration is over when Next returns io.EOF.

import (
	"context"
	"fmt"
	"io"

	"gocloud.dev/docstore"
)

// Ask for all players with scores at least 20.
iter := coll.Query().Where("Score", ">=", 20).OrderBy("Score", docstore.Descending).Get(ctx)
defer iter.Stop()

// Query.Get returns an iterator. Call Next on it until io.EOF.
for {
	var p Player
	err := iter.Next(ctx, &p)
	if err == io.EOF {
		break
	} else if err != nil {
		return err
	} else {
		fmt.Printf("%s: %d\n", p.Name, p.Score)
	}
}

You can pass a list of fields to Get to reduce the amount of data transmitted.

Queries support the following methods:

• Where describes a condition on a document. You can ask whether a field is equal to, greater than, or less than a value. The “not equals” comparison isn’t supported, because it isn’t portable across providers.
• OrderBy specifies the order of the resulting documents, by field and direction. For portability, you can specify at most one OrderBy, and its field must also be mentioned in a Where clause.
• Limit limits the number of documents in the result.

If a query returns an error, the message may help you fix the problem. Some features, like full table scans, have to be enabled via constructor options, because they can be expensive. Other queries may require that you manually create an index on the collection.

Revisions🔗

Docstore maintains a revision for every document. Whenever the document is changed, the revision is too. By default, Docstore stores the revision in a field named DocstoreRevision, but you can change the field name via an option to a Collection constructor.

You can use revisions to perform optimistic locking, a technique for updating a document atomically:

1. Get a document. This reads the current revision.
2. Modify the document contents on the client (but do not change the revision).
3. Replace the document. If the document was changed since it was retrieved in step 1, the revision will be different, and Docstore will return an error instead of overwriting the document.
4. If the Replace failed, start again from step 1.

import (
	"context"
	"fmt"
	"time"

	"gocloud.dev/docstore/memdocstore"
	"gocloud.dev/gcerrors"
)

coll, err := memdocstore.OpenCollection("Name", nil)
if err != nil {
	return err
}
defer coll.Close()

// Create a player.
pat := &Player{Name: "Pat", Score: 7}
if err := coll.Create(ctx, pat); err != nil {
	return err
}
fmt.Println(pat) // memdocstore revisions are deterministic, so we can check the output.

// Double a player's score. We cannot use Update to multiply, so we use optimistic
// locking instead.

// We may have to retry a few times; put a time limit on that.
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
for {
	// Get the document.
	player := &Player{Name: "Pat"}
	if err := coll.Get(ctx, player); err != nil {
		return err
	}
	// player.DocstoreRevision is set to the document's revision.

	// Modify the document locally.
	player.Score *= 2

	// Replace the document. player.DocstoreRevision will be checked against
	// the stored document's revision.
	err := coll.Replace(ctx, player)
	if err != nil {
		code := gcerrors.Code(err)
		// On FailedPrecondition or NotFound, try again.
		if code == gcerrors.FailedPrecondition || code == gcerrors.NotFound {
			continue
		}
		return err
	}
	fmt.Println(player)
	break
}

// Output:
// &{Pat 7 1}
// &{Pat 14 2}

See the Revisions section of the package documentation for more on revisions.

Other Usage Samples🔗

• CLI Sample
• Order Processor sample
• docstore package examples

Supported Docstore Services🔗

Google Cloud Firestore🔗

The gcpfirestore package supports Google Cloud Firestore. Firestore documents are uniquely named by paths that are not part of the document content. In Docstore, these unique names are represented as part of the document. You must supply a way to extract a document’s name from its contents. This can be done by specifying a document field that holds the name, or by providing a function to extract the name from a document.

Firestore URLs provide the project and collection, as well as the field that holds the document name.

docstore.OpenCollection will use Application Default Credentials; if you have authenticated via gcloud auth application-default login, it will use those credentials. See Application Default Credentials to learn about authentication alternatives, including using environment variables.

import (
	"context"

	"gocloud.dev/docstore"
	_ "gocloud.dev/docstore/gcpfirestore"
)

// docstore.OpenCollection creates a *docstore.Collection from a URL.
const url = "firestore://projects/my-project/databases/(default)/documents/my-collection?name_field=userID"
coll, err := docstore.OpenCollection(ctx, url)
if err != nil {
	return err
}
defer coll.Close()

Full details about acceptable URLs can be found under the API reference for gcpfirestore.URLOpener.

Firestore Constructors🔗

The gcpfirestore.OpenCollection constructor opens a Cloud Firestore collection as a Docstore collection. You must first connect a Firestore client using gcpfirestore.Dial or the cloud.google.com/go/firestore/apiv1 package. In addition to a client, OpenCollection requires a Google Cloud project ID, the path to the Firestore collection, and the name of the field that holds the document name.

import (
	"context"

	"gocloud.dev/docstore/gcpfirestore"
	"gocloud.dev/gcp"
)

creds, err := gcp.DefaultCredentials(ctx)
if err != nil {
	return err
}
client, _, err := gcpfirestore.Dial(ctx, creds.TokenSource)
if err != nil {
	return err
}
resourceID := gcpfirestore.CollectionResourceID("my-project", "my-collection")
coll, err := gcpfirestore.OpenCollection(client, resourceID, "userID", nil)
if err != nil {
	return err
}
defer coll.Close()

Instead of mapping the document name to a field, you can supply a function to construct the name from the document contents with gcpfirestore.OpenCollectionWithNameFunc. This can be useful for documents whose name is the combination of two or more fields.

import (
	"context"

	"gocloud.dev/docstore"
	"gocloud.dev/docstore/gcpfirestore"
	"gocloud.dev/gcp"
)

creds, err := gcp.DefaultCredentials(ctx)
if err != nil {
	return err
}
client, _, err := gcpfirestore.Dial(ctx, creds.TokenSource)
if err != nil {
	return err
}

// The name of a document is constructed from the Game and Player fields.
nameFromDocument := func(doc docstore.Document) string {
	hs := doc.(*HighScore)
	return hs.Game + "|" + hs.Player
}

resourceID := gcpfirestore.CollectionResourceID("my-project", "my-collection")
coll, err := gcpfirestore.OpenCollectionWithNameFunc(client, resourceID, nameFromDocument, nil)
if err != nil {
	return err
}
defer coll.Close()

Amazon DynamoDB🔗

The awsdynamodb package supports Amazon DynamoDB. A Docstore collection corresponds to a DynamoDB table.

DynamoDB URLs provide the table, partition key field and optionally the sort key field for the collection.

docstore.OpenCollection will create a default AWS Session with the SharedConfigEnable option enabled; if you have authenticated with the AWS CLI, it will use those credentials. See AWS Session to learn about authentication alternatives, including using environment variables.

import (
	"context"

	"gocloud.dev/docstore"
	_ "gocloud.dev/docstore/awsdynamodb"
)

// docstore.OpenCollection creates a *docstore.Collection from a URL.
coll, err := docstore.OpenCollection(ctx, "dynamodb://my-table?partition_key=name")
if err != nil {
	return err
}
defer coll.Close()

Full details about acceptable URLs can be found under the API reference for awsdynamodb.URLOpener.

DynamoDB Constructor🔗

The awsdynamodb.OpenCollection constructor opens a DynamoDB table as a Docstore collection. You must first create an AWS session with the same region as your collection:

import (
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/dynamodb"
	"gocloud.dev/docstore/awsdynamodb"
)

sess, err := session.NewSession()
if err != nil {
	return err
}
coll, err := awsdynamodb.OpenCollection(
	dynamodb.New(sess), "docstore-test", "partitionKeyField", "", nil)
if err != nil {
	return err
}
defer coll.Close()

Azure Cosmos DB🔗

Azure Cosmos DB is compatible with the MongoDB API. You can use the mongodocstore package to connect to Cosmos DB. You must create an Azure Cosmos account and get the MongoDB connection string.

When you use MongoDB URLs to connect to Cosmos DB, specify the Mongo server URL by setting the MONGO_SERVER_URL environment variable to the connection string. See the MongoDB section for more details and examples on how to use the package.

Cosmos DB Constructors🔗

The mongodocstore.OpenCollection constructor can open a Cosmos DB collection. You must first obtain a standard MongoDB Go client with your Cosmos connections string. See the MongoDB constructor section for more details and examples.

MongoDB🔗

The mongodocstore package supports the popular MongoDB document store. MongoDB documents are uniquely identified by a field called _id. In Docstore, you can choose a different name for this field, or provide a function to extract the document ID from a document.

MongoDB URLs provide the database and collection, and optionally the field that holds the document ID. Specify the Mongo server URL by setting the MONGO_SERVER_URL environment variable.

import (
	"context"

	"gocloud.dev/docstore"
	_ "gocloud.dev/docstore/mongodocstore"
)

// docstore.OpenCollection creates a *docstore.Collection from a URL.
coll, err := docstore.OpenCollection(ctx, "mongo://my-db/my-collection?id_field=userID")
if err != nil {
	return err
}
defer coll.Close()

Full details about acceptable URLs can be found under the API reference for mongodocstore.URLOpener.

MongoDB Constructors🔗

The mongodocstore.OpenCollection constructor opens a MongoDB collection. You must first obtain a standard MongoDB Go client using mongodocstore.Dial or the package go.mongodb.org/mongo-driver/mongo. Obtain a *mongo.Collection from the client with client.Database(dbName).Collection(collName). Then pass the result to mongodocstore.OpenCollection along with the name of the ID field, or "" to use _id.

import (
	"context"

	"gocloud.dev/docstore/mongodocstore"
)

client, err := mongodocstore.Dial(ctx, "mongodb://my-host")
if err != nil {
	return err
}
mcoll := client.Database("my-db").Collection("my-coll")
coll, err := mongodocstore.OpenCollection(mcoll, "userID", nil)
if err != nil {
	return err
}
defer coll.Close()

Instead of mapping the document ID to a field, you can supply a function to construct the ID from the document contents with mongodocstore.OpenCollectionWithIDFunc. This can be useful for documents whose name is the combination of two or more fields.

import (
	"context"

	"gocloud.dev/docstore"
	"gocloud.dev/docstore/mongodocstore"
)

client, err := mongodocstore.Dial(ctx, "mongodb://my-host")
if err != nil {
	return err
}
mcoll := client.Database("my-db").Collection("my-coll")

// The name of a document is constructed from the Game and Player fields.
nameFromDocument := func(doc docstore.Document) interface{} {
	hs := doc.(*HighScore)
	return hs.Game + "|" + hs.Player
}

coll, err := mongodocstore.OpenCollectionWithIDFunc(mcoll, nameFromDocument, nil)
if err != nil {
	return err
}
defer coll.Close()

In-Memory Document Store🔗

The memdocstore package implements an in-memory document store suitable for testing and development.

URLs for the in-memory store have a mem: scheme. The URL host is used as the the collection name, and the URL path is used as the name of the document field to use as a primary key.

import (
	"context"

	"gocloud.dev/docstore"
	_ "gocloud.dev/docstore/memdocstore"
)

// docstore.OpenCollection creates a *docstore.Collection from a URL.
coll, err := docstore.OpenCollection(ctx, "mem://collection/keyField")
if err != nil {
	return err
}
defer coll.Close()

Full details about acceptable URLs can be found under the API reference for memdocstore.URLOpener.

Mem Constructors🔗

The memdocstore.OpenCollection constructor creates and opens a collection, taking the name of the key field.

import "gocloud.dev/docstore/memdocstore"

coll, err := memdocstore.OpenCollection("keyField", nil)
if err != nil {
	return err
}
defer coll.Close()

You can instead supply a function to construct the primary key from the document contents with memdocstore.OpenCollectionWithKeyFunc. This can be useful for documents whose name is the combination of two or more fields.

import (
	"gocloud.dev/docstore"
	"gocloud.dev/docstore/memdocstore"
)

// The name of a document is constructed from the Game and Player fields.
nameFromDocument := func(doc docstore.Document) interface{} {
	hs := doc.(*HighScore)
	return hs.Game + "|" + hs.Player
}

coll, err := memdocstore.OpenCollectionWithKeyFunc(nameFromDocument, nil)
if err != nil {
	return err
}
defer coll.Close()