Docs ingest
A document ingest has both a content tree and a run that tracks selected documents. This example builds the tree with child blueprints and the run as an aggregate, with separate local and library-wide chunk numbering.
A searchable corpus: Library → Document → Section → Chunk, and an ingest run modelled as an aggregate (one run plus one status row per document). Chunk ordinals restart per section, while a library-wide ordinal keeps counting. Source: tests/Zeeq.Biome.Tests/DocsIngest.
Two ordinals on one entity
One reader may need a chunk's position within its section; another may need its position across the library. Keep those as separate sequences with the scope each use requires:
// Chunk ordinals restart for each section.
internal static readonly Sequence<int> ChunkOrdinals = Sequence
.Of(1, n => n + 1)
.PerParent();
// This contrasting scope numbers chunks across
// every document/section in one library.
internal static readonly Sequence<int> LibraryOrdinals = Sequence
.Of(1, n => n + 1)
.Per<Library>();
// Reach Library through Section -> Document; keep
// local and library-wide ordinals distinct.
internal static readonly ChildBlueprint<IngestSeed, Section, Chunk> Chunk =
Blueprint
.ChildOf<IngestSeed, Section, Chunk>(
(section, ctx) =>
new() { Section = section, Library = ctx.Ancestor<Library>() }
)
.Derive(c => c.Ordinal, ChunkOrdinals)
.Derive(c => c.LibraryOrdinal, LibraryOrdinals)
.Derive(
c => c.Text,
(c, _) => $"{c.Section.Heading}: chunk {c.Ordinal}"
);
A corpus and an ingest run
The ingest run is built from an input that selects which documents it covers. At save time, the run and its status rows expand after the document rows.
await biome
.AddLibraries(out var libraries, 2)
.AddDocuments(out var documents, 3) // 6 documents
.AddSections(out var sections, 4) // 24 sections
.AddChunks(out var chunks, 3) // 72 chunks
// Under documents marks the conceptual sibling
// branch; input customizers select its rows.
.Under(documents)
.AddIngestRun(out var graph, i => i.Documents = documents)
.BuildAsync();
var now = DateTimeOffset.UtcNow;
var libraries = new[]
{
new Library { Owner = "owner" },
new Library { Owner = "owner" },
};
var documents = new List<Document>();
var sections = new List<Section>();
var chunks = new List<Chunk>();
foreach (var library in libraries)
{
var version = 1; // per library
var libraryOrdinal = 1; // per library, across documents and sections
for (var d = 0; d < 3; d++)
{
var document = new Document
{
Library = library,
Version = version,
Path = $"guide-v{version}.md",
UpdatedAt = now.AddHours(-1),
};
version++;
documents.Add(document);
for (var s = 1; s <= 4; s++)
{
var section = new Section
{
Document = document,
Number = s,
Heading = $"Section {s}",
};
sections.Add(section);
for (var c = 1; c <= 3; c++)
{
chunks.Add(
new Chunk
{
Section = section,
Library = library,
Ordinal = c,
LibraryOrdinal = libraryOrdinal++,
Text = $"{section.Heading}: chunk {c}",
}
);
}
}
}
}
var run = new IngestRun("ingest", now, now.AddMinutes(5));
var statuses = documents.Select(d => new DocumentStatus(run, d)).ToArray();
db.AddRange(libraries);
db.AddRange(documents);
db.AddRange(sections);
db.AddRange(chunks);
db.Add(run);
db.AddRange(statuses);
await db.SaveChangesAsync();
Assertions in the real test: chunks[3].Ordinal == 1 and chunks[3].LibraryOrdinal == 4 (local restarts, library-wide continues), chunks[36].LibraryOrdinal == 1 (a new library restarts), and the store receives 2 + 6 + 24 + 72 + 1 + 6 rows. The run and its six statuses follow the documents, and the IngestGraph wrapper is never saved.
The aggregate
internal sealed record IngestGraph(IngestRun Run, DocumentStatus[] Statuses)
: IBiomeAggregate
{
// Expansion preserves run-before-status order;
// the wrapper itself is never a row.
public IEnumerable<object> GetEntities() => [Run, .. Statuses];
}
// Input customizers choose documents; expansion
// yields one run and one status per document.
internal static readonly InputBlueprint<
IngestSeed,
IngestInput,
IngestGraph
> Ingest = Blueprint
.FromInput<IngestSeed, IngestInput, IngestGraph>(
() => new(),
(input, ctx) =>
{
var run = new IngestRun(
input.Label!,
ctx.Now,
ctx.FromNow(TimeSpan.FromMinutes(5))
);
return new(
run,
input
.Documents.Select(d => new DocumentStatus(run, d))
.ToArray()
);
}
)
.Derive(i => i.Label, (_, _) => "ingest");
Derive(i => i.Label, …) fills the input before the factory runs, so the factory can use input.Label! here.Archived, sensitive and versioned
Stacked traits, an explicit version that's reserved, and a later sibling ingest under both libraries:
await biome
.AddLibraries(out var libraries, 2)
.AddDocuments(
out var documents,
Documents.Archived + Documents.Sensitive,
// Explicit 1 is reserved first; the other
// document gets 2 and derives guide-v2.md.
d =>
{
d.Version = 1;
d.Path = "contract.md";
}
)
.AddSections(out var sections, 2)
.AddChunks(out _, 2)
// Resume both libraries after their deep descendant branches.
.Under(libraries)
.AddDocuments(out var later, 1)
// Revisit one section: local ordinals continue,
// rather than resetting on this Add call.
.Under(sections[0])
.AddChunk(out var extra)
.BuildAsync();
// documents[0].Version == 2, Path "guide-v2.md"
// documents[1].Path == "contract.md" (call site wins)
// later.All(d => d.Version == 3); extra.Ordinal == 3; extra.LibraryOrdinal == 9
A transient preview aggregate
A preview ingest run reuses a real document but is never saved, and its parts inherit that exclusion:
biome.AddSearchableChunk(out var leaf);
time.Advance(TimeSpan.FromMinutes(10));
await biome
.Under(leaf.Section.Document)
.AddIngestRun(out var preview, i => i.Documents = [leaf.Section.Document])
// Built and returned, but NOT saved; aggregate
// parts inherit the excluded branch.
.Transient()
.BuildAsync();
// store: no IngestRun, no DocumentStatus;
// preview.Run.StartedAt == instant.AddMinutes(10)