🌐 Building Golang RESTful API with Gin, MongoDB 🌱

Building a comprehensive example of a Golang RESTful API using gin, mongo-go-driver, and MongoDB can include many advanced features like transactions, full-text search, aggregation, sharding, change streams, RBAC (Role-Based Access Control), schema validation, and more. Below is a detailed setup that demonstrates these features. It includes an initialization script, API routes, and implementations for advanced MongoDB operations.

Prerequisites:

• Gin: Web framework for building RESTful services.
• MongoDB: (Quick Setup) NoSQL database for storing JSON-like documents.
• Go MongoDB Driver: Official MongoDB Go driver.
• MongoDB should be set up in a replica set configuration for some features like change streams and transactions.

Project Structure:

go-mongo-api/
│   main.go
│   go.mod
│
├── controllers/
│   └── book_controller.go
│
├── models/
│   └── book.go
│   └── author.go
│
├── services/
│   └── book_service.go
│
├── configs/
│   └── db.go
│   └── init_db.go
│
└── scripts/
    └── init_indexes.js

MongoDB Initialization Script (scripts/init_indexes.js):

Before running the application, execute this script in MongoDB to set up indexes, schema validation, and sharding.

// scripts/init_indexes.js

// Connect to the database
db = db.getSiblingDB('library');

// Add root user for the MongoDB instance
db.createUser({
  user: "admin",
  pwd: "password",
  roles: [{ role: "root", db: "admin" }]
});

// Schema validation for the books collection
db.createCollection("books", {
  validator: {
    $jsonSchema: {
      bsonType: "object",
      required: ["title", "author_id", "published_date", "details", "category"],
      properties: {
        title: {
          bsonType: "string",
          description: "Title of the book"
        },
        author_id: {
          bsonType: "objectId",
          description: "Reference to the author"
        },
        published_date: {
          bsonType: "date",
          description: "Publication date"
        },
        details: {
          bsonType: "object",
          description: "Additional details of the book"
        },
        category: { 
          bsonType: "string",
          description: "Category of the book"
        }
      }
    }
  }
});

// This creates a text index on the title field and the details.summary field in the books collection.
// enables full-text search on both fields, allowing you to efficiently search for books based on keywords in either the title or summary.
db.books.createIndex({ title: "text", "details.summary": "text" });

// Creates an ascending index on the name field in the authors collection.
// speeds up queries that filter or sort authors by their name, making lookups for specific author names faster.
db.authors.createIndex({ name: 1 });

// Enable sharding on the "library" database
// Activates sharding for the library database, allowing its collections to be distributed across multiple shards for load balancing and scalability.
sh.enableSharding("library");

// Enables sharding specifically for the books collection in the library database.
// Uses the _id field as the shard key, with a hashed distribution. Hashed sharding evenly distributes documents across shards, which is beneficial for write-heavy workloads by minimizing "hot spots" on specific shards.
sh.shardCollection("library.books", { _id: "hashed" });

MongoDB Data Models (models/*.go):

Here, we have Book and Author models.

// models/book.go
package models

import (
    "go.mongodb.org/mongo-driver/bson/primitive"
    "time"
)

type Book struct {
    ID            primitive.ObjectID `json:"id,omitempty" bson:"_id,omitempty"`
    Title         string             `json:"title" bson:"title"`
    AuthorID      primitive.ObjectID `json:"author_id" bson:"author_id"`
    PublishedDate time.Time          `json:"published_date" bson:"published_date"`
    Details       map[string]interface{} `json:"details" bson:"details"`
    Category      string                 `json:"category" bson:"category"`
}

// models/author.go
package models

import "go.mongodb.org/mongo-driver/bson/primitive"

type Author struct {
    ID   primitive.ObjectID `json:"id,omitempty" bson:"_id,omitempty"`
    Name string             `json:"name" bson:"name"`
    Bio  string             `json:"bio" bson:"bio"`
}

MongoDB Configuration (configs/db.go):

Set up MongoDB connection and provide methods for retrieving collections.

// configs/db.go
package configs

import (
    "context"
    "log"
    "time"

    "go.mongodb.org/mongo-driver/mongo"
    "go.mongodb.org/mongo-driver/mongo/options"
)

var DB *mongo.Client

func ConnectDB(context *context.Context) *mongo.Client {
    clientOptions := options.Client().ApplyURI("mongodb://localhost:27017")
    clientOptions.SetAuth(options.Credential{
        Username: "admin",
        Password: "password",
    })

    client, err := mongo.Connect(context, clientOptions)
    if err != nil {
        log.Fatal(err)
    }

    err = client.Ping(context, nil)
    if err != nil {
        log.Fatal(err)
    }

    log.Println("Connected to MongoDB")
    DB = client
    return client
}

func GetCollection(client *mongo.Client, collectionName string) *mongo.Collection {
    return client.Database("library").Collection(collectionName)
}

CRUD Operations with Advanced Features (controllers/book_controller.go):

This controller handles CRUD operations, transactions, full-text search, and complex queries.

// controllers/book_controller.go
package controllers

import (
    "context"
    "go-mongo-api/configs"
    "go-mongo-api/models"
    "go-mongo-api/services"
    "net/http"
    "time"

    "github.com/gin-gonic/gin"
    "go.mongodb.org/mongo-driver/bson"
    "go.mongodb.org/mongo-driver/bson/primitive"
    "go.mongodb.org/mongo-driver/mongo"
    "go.mongodb.org/mongo-driver/mongo/options"
)

// CreateBook - Create a new book with transaction
func CreateBook(c *gin.Context) {
    var book models.Book
    if err := c.BindJSON(&book); err != nil {
        c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
        return
    }

    book.ID = primitive.NewObjectID()
    book.PublishedDate = time.Now()

    session, err := configs.DB.StartSession()
    if err != nil {
        c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
        return
    }
    defer session.EndSession(context.Background())

    err = mongo.WithSession(context.Background(), session, func(sessCtx mongo.SessionContext) error {
        collection := configs.GetCollection(configs.DB, "books")
        _, err := collection.InsertOne(sessCtx, book)
        if err != nil {
            return err
        }

        // Example: Update author's book count (if needed)
        authorsColl := configs.GetCollection(configs.DB, "authors")
        _, err = authorsColl.UpdateOne(
            sessCtx,
            bson.M{"_id": book.AuthorID},
            bson.M{"$inc": bson.M{"book_count": 1}},
        )
        return err
    })

    if err != nil {
        c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
        return
    }

    c.JSON(http.StatusCreated, book)
}


// GetBook - Get a book by ID
func GetBook(c *gin.Context) {
    id := c.Param("id")
    objID, _ := primitive.ObjectIDFromHex(id)

    var book models.Book
    collection := configs.GetCollection(configs.DB, "books")

    err := collection.FindOne(context.TODO(), bson.M{"_id": objID}).Decode(&book)
    if err == mongo.ErrNoDocuments {
        c.JSON(http.StatusNotFound, gin.H{"error": "Book not found"})
        return
    } else if err != nil {
        c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
        return
    }

    c.JSON(http.StatusOK, book)
}

// ListBooks - List books with cursor-based pagination
func ListBooks(c *gin.Context) {
    var books []models.Book
    collection := configs.GetCollection(configs.DB, "books")

    limit, err := primitive.ParseInt64(c.DefaultQuery("limit", "10"), 10, 64)
    if err != nil {
        limit = 10
    }

    cursorID := c.Query("cursor")
    filter := bson.M{}
    if cursorID != "" {
        objID, _ := primitive.ObjectIDFromHex(cursorID)
        filter = bson.M{"_id": bson.M{"$gt": objID}}
    }

    findOptions := options.Find().SetLimit(limit)
    cursor, err := collection.Find(context.TODO(), filter, findOptions)
    if err != nil {
        c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
        return
    }
    defer cursor.Close(context.TODO())

    for cursor.Next(context.TODO()) {
        var book models.Book
        cursor.Decode(&book)
        books = append(books, book)
    }

    c.JSON(http.StatusOK, gin.H{
        "books": books,
        "next_cursor": books[len(books)-1].ID.Hex(),
    })
}


// ListBooksByCategory - Get book counts grouped by category
func ListBooksByCategory(c *gin.Context) {
    collection := configs.GetCollection(configs.DB, "books")

    // Aggregate books by category and count them
    pipeline := []bson.M{
        {
            "$group": bson.M{
                "_id":     "$category", // Group by category
                "count":   bson.M{"$sum": 1}, // Count books in each category
            },
        },
        {
            "$sort": bson.M{
                "count": -1, // Sort by count in descending order
            },
        },
    }

    cursor, err := collection.Aggregate(context.TODO(), pipeline)
    if err != nil {
        c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
        return
    }
    defer cursor.Close(context.TODO())

    var categories []bson.M
    if err = cursor.All(context.TODO(), &categories); err != nil {
        c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
        return
    }

    c.JSON(http.StatusOK, categories)
}


// SearchBooks - Perform full-text search on books
func SearchBooks(c *gin.Context) {
    query := c.Query("q")
    collection := configs.GetCollection(configs.DB, "books")

    filter := bson.M{
        "$text": bson.M{
            "$search": query,
        },
    }

    cursor, err := collection.Find(context.TODO(), filter)
    if err != nil {
        c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
        return
    }
    defer cursor.Close(context.TODO())

    var books []models.Book
    for cursor.Next(context.TODO()) {
        var book models.Book
        cursor.Decode(&book)
        books = append(books, book)
    }

    c.JSON(http.StatusOK, books)
}

// GetAuthorBooks - Example of joining authors and books using aggregation
func GetAuthorBooks(c *gin.Context) {
    authorID := c.Param("authorId")
    objID, _ := primitive.ObjectIDFromHex(authorID)

    pipeline := mongo.Pipeline{
        bson.D{{"$match", bson.D{{"author_id", objID}}}},
        bson.D{{"$lookup", bson.D{
            {"from", "authors"},
            {"localField", "author_id"},
            {"foreignField", "_id"},
            {"as", "author_details"},
        }}},
        bson.D{{"$unwind", "$author_details"}},
    }

    collection := configs.GetCollection(configs.DB, "books")
    cursor, err := collection.Aggregate(context.TODO(), pipeline)
    if err != nil {
        c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
        return
    }
    defer cursor.Close(context.TODO())

    var books []bson.M
    for cursor.Next(context.TODO()) {
        var book bson.M
        cursor.Decode(&book)
        books = append(books, book)
    }

    c.JSON(http.StatusOK, books)
}

// UpdateBook - Update a book's info
func UpdateBook(c *gin.Context) {
    id := c.Param("id")
    objID, _ := primitive.ObjectIDFromHex(id)

    var book models.Book
    if err := c.BindJSON(&book); err != nil {
        c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
        return
    }

    collection := configs.GetCollection(configs.DB, "books")
    update := bson.M{
        "$set": bson.M{
            "title":          book.Title,
            "author_id":      book.AuthorID,
            "published_date": book.PublishedDate,
            "details":        book.Details,
        },
    }

    _, err := collection.UpdateOne(context.TODO(), bson.M{"_id": objID}, update)
    if err != nil {
        c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
        return
    }

    c.JSON(http.StatusOK, gin.H{"message": "Book updated successfully"})
}

// UpdateBookDetails - Update a field inside the JSONB 'details'
func UpdateBookDetails(c *gin.Context) {
    id := c.Param("id")
    objID, _ := primitive.ObjectIDFromHex(id)

    var updateData map[string]interface{}
    if err := c.BindJSON(&updateData); err != nil {
        c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
        return
    }

    collection := configs.GetCollection(configs.DB, "books")
    update := bson.M{
        "$set": bson.M{"details": updateData},
    }

    _, err := collection.UpdateOne(context.TODO(), bson.M{"_id": objID}, update)
    if err != nil {
        c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
        return
    }

    c.JSON(http.StatusOK, gin.H{"message": "Book details updated"})
}

// DeleteBook - Delete a book by ID
func DeleteBook(c *gin.Context) {
    id := c.Param("id")
    objID, _ := primitive.ObjectIDFromHex(id)

    collection := configs.GetCollection(configs.DB, "books")
    _, err := collection.DeleteOne(context.TODO(), bson.M{"_id": objID})
    if err != nil {
        c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
        return
    }

    c.JSON(http.StatusOK, gin.H{"message": "Book deleted successfully"})
}

Change Streams Example (services/book_service.go):

A service to demonstrate using change streams for real-time updates.

// services/book_service.go
package services

import (
    "context"
    "fmt"
    "go-mongo-api/configs"
    "go.mongodb.org/mongo-driver/mongo"
)

func WatchBooksChanges() {
    collection := configs.GetCollection(configs.DB, "books")
    stream, err := collection.Watch(context.TODO(), mongo.Pipeline{})
    if err != nil {
        fmt.Println("Error starting change stream:", err)
        return
    }
    defer stream.Close(context.Background())

    fmt.Println("Watching changes on books collection...")

    for stream.Next(context.Background()) {
        var changeEvent map[string]interface{}
        if err := stream.Decode(&changeEvent); err != nil {
            fmt.Println("Error decoding change event:", err)
            continue
        }

        fmt.Println("Change detected:", changeEvent)
    }
}

Main Entry (main.go):

Initialize the server and routes.

// main.go
package main

import (
    "go-mongo-api/configs"
    "go-mongo-api/controllers"
    "go-mongo-api/services"
    "github.com/gin-gonic/gin"
)

func main() {
    router := gin.Default()
    ctx:=context.Background()
    
    configs.ConnectDB(ctx)
    go services.WatchBooksChanges() // Start watching changes

    // Routes
    router.POST("/books", controllers.CreateBook)
    router.GET("/books/:id", controllers.GetBook)
    router.GET("/list-books", controllers.ListBooks)
    router.GET("/books/count-by-category", controllers.ListBooksByCategory) 
    router.GET("/search-books", controllers.SearchBooks)
    router.GET("/books/author/:authorId", controllers.GetAuthorBooks)
    router.PUT("/books/:id", controllers.UpdateBook)
    router.PUT("/books/:id/details", controllers.UpdateBookDetails)
    router.DELETE("/books/:id", controllers.DeleteBook)

    router.Run(":8080")
}

Features Covered:

1. CRUD Operations: Create, read, update, and delete books with MongoDB.
2. Cursor-based Pagination: List books with support for pagination.
3. Aggregation with grouping, sorting: Function aggregates book counts by category and sorts the results in descending order before returning them in JSON format.
4. JSON Handling: Use the details field to store nested JSON data.
5. Transactions: Example of a multi-document transaction while creating a book.
6. Full-Text Search: Search books using MongoDB's text indexes.
7. Joining Data: Use the $lookup aggregation stage to join books with author details.
8. Update JSON Fields: Modify nested JSON fields within the details object.
9. Change Streams: Monitor real-time changes to the books collection.
10. Schema Validation: Enforce document structure using jsonSchema.
11. Indexes: Create text and compound indexes.
12. Sharding: Enable sharding for distributed data across clusters.
13. RBAC: Example uses MongoDB connection with authentication.

This setup gives you a full-fledged REST API with MongoDB using Golang, covering many advanced MongoDB features. Make sure MongoDB is set up as a replica set for transactions and change streams to work properly.

More On Replica Set

MongoDB's replica set configuration is essential for enabling certain advanced features, particularly transactions and change streams. Here's why it's important:

1. Replica Set Requirement for Transactions

• Transactions in MongoDB, which allow multiple operations to be executed atomically (i.e., all or nothing), are only available in replica sets.
• Transactions ensure data consistency across operations by bundling them into a single unit. If one operation fails, the transaction can roll back all changes.
• Replica sets in MongoDB provide the infrastructure necessary to maintain this atomicity, as they keep multiple copies (replicas) of data across servers. This allows MongoDB to manage and roll back transactions efficiently.

2. Replica Set Requirement for Change Streams

• Change streams allow applications to subscribe to real-time data changes in MongoDB.
• They notify your application whenever an insert, update, or delete operation occurs on a collection, making it useful for event-driven architectures, data synchronization, and caching.
• Change streams rely on oplog (operation log), which is available only in replica sets. The oplog records changes and propagates them across the replica set, allowing MongoDB to replay these operations and stream them to the client.

Setting up a Replica Set in MongoDB

A basic setup for local development involves:

1. Starting multiple MongoDB instances with replica set configuration.
2. Using the rs.initiate() command to initialize the replica set and add members.

For production, you typically set up a replica set across different servers or data centers to improve fault tolerance and availability. In case a primary instance fails, a secondary instance can take over to maintain uninterrupted service.

If you found this helpful, let me know by leaving a 👍 or a comment!, or if you think this post could help someone, feel free to share it! Thank you very much! 😃