Sentinel Logo

Collection

Learn about Collections, namespaces for organizing documents in Sentinel.

Collections are namespaces for organizing related documents within a Sentinel store. Each collection corresponds to a directory on your filesystem, making it easy to manage and browse your data using standard tools. Collections provide comprehensive CRUD operations, advanced querying capabilities, and flexible verification options.

Creating and Accessing Collections

Collections are created automatically when you access them through a store:

use sentinel_dbms::Store;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = Store::new("./my-data", None).await?;

    // Creates into collection directory if it doesn't exist
    let users = store.collection("users").await?;
    let products = store.collection("products").await?;
    let audit_logs = store.collection("audit_logs").await?;

    Ok(())
}

Each call to collection() returns a Collection instance that provides CRUD operations for documents within that namespace.

Collection Structure

A collection is represented as a directory within your store’s data/ folder:

my-data/
└── data/
    └── users/              # Collection directory
        ├── alice.json      # Document: "alice"
        ├── bob.json        # Document: "bob"
        └── charlie.json    # Document: "charlie"
        └── .deleted/      # Soft-deleted documents
            ├── old-user.json
            └── archived.json

Document IDs become filenames with a .json extension. The .deleted/ subdirectory stores soft-deleted documents for recovery purposes. This means you can browse your data with any file manager or command-line tool.

CRUD Operations

Collections provide four primary operations for document management.

Insert

The insert method creates a new document or overwrites an existing one:

use sentinel_dbms::Store;
use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = Store::new("./my-data", None).await?;
    let users = store.collection("users").await?;

    // Insert a new document
    users.insert("alice", json!({
        "name": "Alice Johnson",
        "email": "[email protected]",
        "role": "admin"
    })).await?;

    // Overwriting is allowed - same ID replaces document
    users.insert("alice", json!({
        "name": "Alice Smith",  // Updated name
        "email": "[email protected]",
        "role": "admin"
    })).await?;

    Ok(())
}

When you insert a document, Sentinel automatically:

  • Computes a BLAKE3 hash of the data for integrity verification
  • Adds metadata fields (id, version, created_at, updated_at, hash)
  • Optionally signs the document with Ed25519 if the store was created with a passphrase
  • Writes the document atomically to disk as {id}.json

Bulk Insert

For inserting multiple documents efficiently, use the bulk_insert method:

use sentinel_dbms::Store;
use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = Store::new("./my-data", None).await?;
    let users = store.collection("users").await?;

    // Prepare documents for bulk insert
    let documents = vec![
        ("user-123", json!({"name": "Alice", "role": "admin"})),
        ("user-456", json!({"name": "Bob", "role": "user"})),
        ("user-789", json!({"name": "Charlie", "role": "user"})),
    ];

    // Insert all documents in one operation
    users.bulk_insert(documents).await?;

    Ok(())
}

The bulk_insert method inserts documents in the order provided. If any document fails to insert, the operation stops and returns an error. In case of failure, some documents may have been inserted before the error occurred.

Get

The get method retrieves a document by its ID:

use sentinel_dbms::Store;
use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = Store::new("./my-data", None).await?;
    let users = store.collection("users").await?;

    // Insert first
    users.insert("alice", json!({"name": "Alice"})).await?;

    // Get returns Option<Document>
    match users.get("alice").await? {
        Some(doc) => {
            println!("Found: {}", doc.data()["name"]);
            println!("ID: {}", doc.id());
            println!("Hash: {}", doc.hash());
        }
        None => {
            println!("Document not found");
        }
    }

    // Non-existent documents return None, not an error
    let missing = users.get("nonexistent").await?;
    assert!(missing.is_none());

    Ok(())
}

By default, get verifies both hash and signature with strict mode. Use get_with_verification() to customize verification behavior:

use sentinel_dbms::{Store, VerificationOptions, VerificationMode};
use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = Store::new("./my-data", Some("passphrase")).await?;
    let users = store.collection("users").await?;

    users.insert("alice", json!({"name": "Alice"})).await?;

    // Get with custom verification options
    let options = VerificationOptions {
        verify_signature: true,
        verify_hash: true,
        signature_verification_mode: VerificationMode::Warn,
        empty_signature_mode: VerificationMode::Warn,
        hash_verification_mode: VerificationMode::Warn,
    };

    let doc = users.get_with_verification("alice", &options).await?;
    assert!(doc.is_some());

    Ok(())
}

Update

The update method replaces a document’s contents:

use sentinel_dbms::Store;
use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = Store::new("./my-data", None).await?;
    let users = store.collection("users").await?;

    // Create initial document
    users.insert("alice", json!({
        "name": "Alice",
        "level": 1
    })).await?;

    // Update with new data (full replacement)
    users.update("alice", json!({
        "name": "Alice",
        "level": 2,  // Leveled up!
        "badges": ["early_adopter"]
    })).await?;

    Ok(())
}

Note that update performs a full replacement, not a partial merge. If you want to preserve existing fields, read the document first, modify the data, then update.

Delete

The delete method removes a document from the collection:

use sentinel_dbms::Store;
use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = Store::new("./my-data", None).await?;
    let users = store.collection("users").await?;

    users.insert("temp-user", json!({"name": "Temporary"})).await?;

    // Delete the document
    users.delete("temp-user").await?;

    // Deleting again is safe (idempotent)
    users.delete("temp-user").await?;

    // Verify it's gone
    assert!(users.get("temp-user").await?.is_none());

    Ok(())
}

The delete operation performs a soft delete by moving the file to a .deleted/ subdirectory within the collection. This makes the operation idempotent and allows for recovery of accidentally deleted documents. If the collection directory or the .deleted/ subdirectory doesn’t exist, they are created automatically.

Document ID Validation

Document IDs must be valid filenames across all platforms. Sentinel validates IDs before any operation:

Valid characters include:

  • Letters: a-z, A-Z
  • Numbers: 0-9
  • Underscore: _
  • Hyphen: -

Invalid characters include:

  • Path separators: /, \
  • Windows reserved characters: < > : " | ? *
  • Control characters: ASCII 0x00-0x1F, 0x7F
  • Dots (to avoid confusion with file extensions)

Examples of valid IDs:

  • user-123
  • alice
  • DOCUMENT_001
  • config-v2-backup

Examples of invalid IDs:

  • user/admin (contains path separator)
  • file.name (contains dot)
  • CON (Windows reserved name)
  • empty string
use sentinel_dbms::Store;
use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = Store::new("./my-data", None).await?;
    let users = store.collection("users").await?;

    // This works
    users.insert("valid-id-123", json!({})).await?;

    // This fails with InvalidDocumentId error
    match users.insert("invalid/id", json!({})).await {
        Err(sentinel_dbms::SentinelError::InvalidDocumentId { id }) => {
            println!("Invalid ID: {}", id);
        }
        _ => {}
    }

    Ok(())
}

Listing Documents

The list method returns a stream of all document IDs in the collection:

use sentinel_dbms::Store;
use futures::TryStreamExt;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = Store::new("./my-data", None).await?;
    let users = store.collection("users").await?;

    // Stream all document IDs
    let ids: Vec<_> = users.list().try_collect().await?;

    println!("Found {} documents", ids.len());
    for id in ids {
        println!("  - {}", id);
    }

    Ok(())
}

The count method provides the total number of documents:

use sentinel_dbms::Store;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = Store::new("./my-data", None).await?;
    let users = store.collection("users").await?;

    let count = users.count().await?;
    println!("Total documents: {}", count);

    Ok(())
}

Streaming All Documents

Stream all documents in a collection without loading them all into memory:

use sentinel_dbms::{Store, VerificationOptions};
use futures::StreamExt;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = Store::new("./my-data", None).await?;
    let users = store.collection("users").await?;

    // Stream all documents with default verification
    let mut all_docs = users.all();

    while let Some(doc_result) = all_docs.next().await {
        match doc_result {
            Ok(doc) => {
                println!("Document: {}", doc.id());
                println!("  Data: {}", doc.data());
            }
            Err(e) => {
                eprintln!("Error reading document: {}", e);
            }
        }
    }

    Ok(())
}

Use all_with_verification() for custom verification options:

use sentinel_dbms::{Store, VerificationOptions};
use futures::StreamExt;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = Store::new("./my-data", None).await?;
    let users = store.collection("users").await?;

    // Stream with warnings instead of errors
    let options = VerificationOptions::warn();
    let mut all_docs = users.all_with_verification(&options);

    while let Some(doc_result) = all_docs.next().await {
        // Process documents...
    }

    Ok(())
}

Filtering Documents

The filter method applies a predicate function to all documents:

use sentinel_dbms::Store;
use futures::StreamExt;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = Store::new("./my-data", None).await?;
    let users = store.collection("users").await?;

    // Stream only documents where role is "admin"
    let mut admins = users.filter(|doc| {
        doc.data().get("role")
            .and_then(|v| v.as_str())
            .map_or(false, |role| role == "admin")
    });

    while let Some(doc) = admins.next().await {
        let doc = doc?;
        println!("Admin: {}", doc.data()["name"]);
    }

    Ok(())
}

For custom verification options during filtering, use filter_with_verification():

use sentinel_dbms::{Store, VerificationOptions};
use futures::StreamExt;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = Store::new("./my-data", None).await?;
    let users = store.collection("users").await?;

    // Filter with custom verification
    let options = VerificationOptions::warn();
    let mut admins = users.filter_with_verification(
        |doc| {
            doc.data().get("role")
                .and_then(|v| v.as_str())
                .map_or(false, |role| role == "admin")
        },
        &options
    );

    while let Some(doc) = admins.next().await {
        let doc = doc?;
        println!("Admin: {}", doc.data()["name"]);
    }

    Ok(())
}

Querying Documents

For complex queries with filters, sorting, pagination, and field projection, use the query method with QueryBuilder:

use sentinel_dbms::{Store, QueryBuilder, Operator, SortOrder};
use serde_json::json;
use futures::TryStreamExt;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = Store::new("./my-data", None).await?;
    let users = store.collection("users").await?;

    let query = QueryBuilder::new()
        .filter("role", Operator::Equals, json!("admin"))
        .sort("name", SortOrder::Ascending)
        .limit(10)
        .build();

    let result = users.query(query).await?;
    let documents: Vec<_> = result.documents.try_collect().await?;

    println!("Found {} admins", documents.len());
    for doc in documents {
        println!("  - {}", doc.data()["name"]);
    }

    Ok(())
}

Use query_with_verification() for custom verification options during queries:

use sentinel_dbms::{Store, QueryBuilder, Operator, VerificationOptions};
use serde_json::json;
use futures::TryStreamExt;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = Store::new("./my-data", None).await?;
    let users = store.collection("users").await?;

    let options = VerificationOptions::warn();
    let query = QueryBuilder::new()
        .filter("role", Operator::Equals, json!("admin"))
        .build();

    let result = users.query_with_verification(query, &options).await?;
    let documents: Vec<_> = result.documents.try_collect().await?;

    println!("Found {} admins", documents.len());

    Ok(())
}

See the Querying and Filtering guide for comprehensive details on query building and operators.

Working with JSON Data

Sentinel uses serde_json::Value for document data, giving you flexibility in what you store:

use sentinel_dbms::Store;
use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = Store::new("./my-data", None).await?;
    let data = store.collection("data").await?;

    // Store various JSON types
    data.insert("string-example", json!("just a string")).await?;
    data.insert("number-example", json!(42)).await?;
    data.insert("array-example", json!([1, 2, 3, 4, 5])).await?;
    data.insert("nested-example", json!({
        "level1": {
            "level2": {
                "level3": "deep value"
            }
        }
    })).await?;

    // Read and navigate nested data
    if let Some(doc) = data.get("nested-example").await? {
        let deep = &doc.data()["level1"]["level2"]["level3"];
        println!("Deep value: {}", deep);
    }

    Ok(())
}

Collection Names

Like document IDs, collection names must be valid directory names:

use sentinel_dbms::Store;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = Store::new("./my-data", None).await?;

    // Valid collection names
    store.collection("users").await?;
    store.collection("audit_logs").await?;
    store.collection("products-v2").await?;
    store.collection("config.backup").await?;  // Dots allowed in collections

    // Invalid - would fail with InvalidCollectionName
    // store.collection("users/admin").await?;
    // store.collection("CON").await?;

    Ok(())
}

Note that collection names allow dots (for patterns like config.backup), while document IDs do not (to avoid confusion with the .json extension).

Verification Options

Collection methods that read documents support flexible verification options through VerificationOptions:

use sentinel_dbms::{VerificationOptions, VerificationMode};

// Default: strict verification of both hash and signature
let default = VerificationOptions::default();

// Strict mode: fail on any verification error
let strict = VerificationOptions::strict();

// Disabled: skip all verifications
let disabled = VerificationOptions::disabled();

// Warning mode: log warnings but continue
let warn = VerificationOptions::warn();

// Custom configuration
let custom = VerificationOptions {
    verify_signature: true,
    verify_hash: false,  // Only verify signature
    signature_verification_mode: VerificationMode::Strict,
    empty_signature_mode: VerificationMode::Warn,
    hash_verification_mode: VerificationMode::Silent,
};

See the Cryptography and Error Handling guides for more details on verification behavior.

Error Handling

Collection operations return Result<T, SentinelError>. Handle errors appropriately:

use sentinel_dbms::{Store, SentinelError};
use serde_json::json;

#[tokio::main]
async fn main() {
    let store = Store::new("./my-data", None).await.unwrap();
    let users = store.collection("users").await.unwrap();

    match users.insert("test", json!({})).await {
        Ok(()) => println!("Document created"),
        Err(SentinelError::InvalidDocumentId { id }) => {
            eprintln!("Invalid document ID: {}", id);
        }
        Err(SentinelError::Io { source }) => {
            eprintln!("I/O error: {}", source);
        }
        Err(SentinelError::HashVerificationFailed { id, reason }) => {
            eprintln!("Hash verification failed for document {}: {}", id, reason);
        }
        Err(SentinelError::SignatureVerificationFailed { id, reason }) => {
            eprintln!("Signature verification failed for document {}: {}", id, reason);
        }
        Err(e) => {
            eprintln!("Other error: {}", e);
        }
    }
}

Best Practices

When working with collections, consider these recommendations:

  • Use meaningful collection names. Choose names that describe data: users, audit_logs, certificates, api_keys.

  • Keep related documents together. Group documents by type and access patterns in the same collection.

  • Use consistent ID schemes. Adopt a naming convention for IDs: user-{uuid}, log-{timestamp}, cert-{fingerprint}.

  • Validate data before insertion. Sentinel stores whatever JSON you give it—validate your data first.

  • Consider using prefixes for large collections. If you expect millions of documents, use ID prefixes to help with manual navigation: a-alice, b-bob.

  • Choose appropriate verification mode. Use strict mode for production, warn mode for auditing, and silent mode for performance-critical scenarios.

  • Leverage streaming for large datasets. Use filter, all, and query methods instead of collecting all documents into memory.

  • Use bulk insert for multiple documents. The bulk_insert method is more efficient than multiple individual insert calls.

Next Steps

Now that you understand collections, learn about:

Related Pages

Store
Learn about Store, the top-level container for collections in Sentinel.
Document
Learn about Document structure, metadata fields, and integrity verification in Sentinel.
Quick Start
Create your first Sentinel store, collection, and documents in minutes.