/
Ingestion Pipeline Logging, Auditability, and Anomaly Detection

Ingestion Pipeline Logging, Auditability, and Anomaly Detection

Overview

This page shows representative examples of logging generated during document ingestion and indexing for retrieval-augmented generation (RAG) workflows. The examples are illustrative, but they reflect the level of detail typically captured for operational monitoring, troubleshooting, access-control validation, and audit review.

The examples below use SharePoint Online as the source system, but the same logging pattern generally applies to other supported enterprise content sources.

Logging Scope

Ingestion and processing logs generally record:

  • source system, site, library, and document identifiers

  • processing state transitions and timestamps

  • parser, chunking, and embedding outcomes

  • retry and failure details

  • source-system permission assignments as observed at ingestion time

  • mapped access-control records written to the retrieval layer

  • service identity or user context associated with the action

  • audit events and security-relevant flags

This information is used to confirm that content was ingested from an expected source, processed successfully, and published with the correct access controls.

Example 1: Ingestion Event

This example shows a document being picked up from SharePoint and placed into the ingestion pipeline.

2026-03-11T14:19:10.551Z INFO ingestion-worker event=source_item_queued correlation_id=ing-7f1d3f8e-8f47-4e12-b4e9-91d9d8fd0d55 request_id=req-b9ce6b74b5b94f4ea2f4 environment=prod service=gateway-embedding-service source_system=sharepoint_online tenant=contoso.sharepoint.com site_path=/sites/Finance library=Shared Documents source_path="/sites/Finance/Shared Documents/FY26/Budget/Board-Approved-Budget-v3.xlsx" source_item_id=482991 source_etag="{6A1F4EAA-2A8D-4D0B-9CFA-6D8F5A1D1A9E},17" last_modified_utc=2026-03-11T14:18:02.001Z content_type=application/vnd.openxmlformats-officedocument.spreadsheetml.sheet file_size_bytes=284193 sync_mode=incremental queued_by=svc_sharepoint_ingester queue_status=accepted retry_count=0

Example 2: Processing Completion Event

This example shows the same document after parsing, chunking, embedding, and publication.

2026-03-11T14:22:48.219Z INFO embedding-pipeline event=content_published correlation_id=ing-7f1d3f8e-8f47-4e12-b4e9-91d9d8fd0d55 content_id=cnt_7c0c9a2d account_id=acct_1024 project_id=proj_88 pipeline=parse>chunk>embed>permission_map>publish parse_status=success detected_sheets=6 text_chars_extracted=48217 chunking_strategy=recursive_character chunk_count=74 estimated_token_count=12984 embedding_provider=openai embedding_model=text-embedding-3-large embedding_status=success embedded_chunk_count=74 queue_submitted_utc=2026-03-11T14:19:10.551Z processing_started_utc=2026-03-11T14:20:01.090Z completed_utc=2026-03-11T14:22:48.219Z duration_ms=41872 publish_status=success

Example 3: Permission Mapping Event

This example shows the permissions observed on the source file and the resulting access-control mapping applied to indexed content.

2026-03-11T14:22:47.903Z INFO permission-mapper event=permission_mapping_completed correlation_id=ing-7f1d3f8e-8f47-4e12-b4e9-91d9d8fd0d55 content_id=cnt_7c0c9a2d permission_source=sharepoint_acl inheritance_broken=true access_mode=allow_list principal[0].type=user principal[0].id=aad:9f0c7f11-12ab-41da-845f-2b5c8e7a3d10 principal[0].display_name="Jane Doe" principal[0].role=read principal[1].type=group principal[1].id=aadgrp:3c2f20b0-98c5-4ff7-8c84-5d2a8b4f0a13 principal[1].display_name="Finance Leadership" principal[1].role=read principal[2].type=group principal[2].id=aadgrp:6c1202cc-a261-4f3e-a915-22d6d212ab77 principal[2].display_name="Budget Editors" principal[2].role=read principal_count=3 mapping_status=success gateway_permission_records_written=3 permission_mismatch_detected=false

Example 4: Audit Record

This is the corresponding audit entry for the ingestion and publication action.

2026-03-11T14:22:48.230Z INFO audit-log audit_event=content_ingested result=success actor_type=service actor_id=svc_sharepoint_ingester account_id=acct_1024 project_id=proj_88 content_id=cnt_7c0c9a2d source_system=sharepoint_online source_path="/sites/Finance/Shared Documents/FY26/Budget/Board-Approved-Budget-v3.xlsx" action=ingest_and_publish review_required=false

Example 5: Security-Relevant Exception

This example shows a file that was not published because the access-control state at ingestion time was inconsistent with prior observations.

2026-03-11T15:03:11.004Z WARN ingestion-guard event=content_quarantined correlation_id=ing-1e2d4470-2d0d-4ef6-a365-8d2ec5e82f40 severity=high source_system=sharepoint_online source_path="/sites/Finance/Shared Documents/M&A/Target-List.xlsx" content_id=cnt_99ab41f2 detected_issue=permission_mapping_mismatch details="source ACL principal count changed from 12 to 0 before publish" previous_principal_count=12 current_principal_count=0 embedding_status=completed publish_status=blocked automatic_action=quarantined_not_published alert_targets="Security Operations; Platform Owner" review_required=true

Operational Use

These logs are typically used for three purposes.

First, they support operational troubleshooting. Teams can trace a document from source pickup through parsing, chunking, embedding, permission mapping, and publication.

Second, they support access-control validation. The logs show both the permissions observed on the source item and the access-control records written to the indexed representation.

Third, they support security monitoring. Alerts can be generated for conditions such as:

  • a document being ingested from an unexpected site or library

  • repeated ingestion failures or excessive retries

  • a source item’s permission set changing unexpectedly

  • a file reaching embedding completion but failing permission mapping

  • a previously restricted file appearing with an empty or expanded ACL

  • unusual ingestion volume outside normal baselines