Async CVM PDF Text Extraction with Webhook Callbacks

The problem

Synchronous PDF-to-text over HTTP does not scale for large DFP or FRE files:

• Multi-hundred-page PDFs exceed reasonable HTTP timeouts
• Page-level progress matters for streaming into RAG indexes
• Retries and partial failure need a durable pattern

apicvm handles extraction in a background worker. Progress arrives only through callbacks — there is no HTTP job-status endpoint.

The async flow

1. GET /v1/documents?ticker=PETR4&type=DFP&year=2024  → pick document.id
2. POST /v1/document-text-extractions  → 202 { id, status: "queued", document_id }
3. Page workers process pages in parallel → POST callback_url per page (status: success)
4. Parent job finishes → POST callback_url with status: completed

Step 1: Enqueue extraction

export APICVM_KEY='apicvm_...'
export APICVM_URL='https://apicvm.dev'

curl -X POST \
  -H "Authorization: Bearer $APICVM_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "callback_url": "https://your-app.example.com/callbacks/apicvm",
    "document": { "id": "<document-uuid>" }
  }' \
  "$APICVM_URL/v1/document-text-extractions"

Response 202:

{
  "id": "42",
  "status": "queued",
  "document_id": "550e8400-e29b-41d4-a716-446655440000"
}

Note: job response fields use snake_case (document_id), unlike most of the API.

If a page already has cached ocr_text, that page job reuses it (credits are still debited at enqueue for the full page count). Pages may arrive out of order.

Step 2: Handle page callbacks

Your endpoint receives one POST per page (up to 3 retries with backoff). Do not treat page order as completion — wait for status: "completed".

Success payload (per page):

{
  "status": "success",
  "document_id": "550e8400-e29b-41d4-a716-446655440000",
  "job_id": "42",
  "page": {
    "number": 1,
    "markdown": "# Section title\n\nParagraph text..."
  },
  "total_pages": 120
}

Completed payload (document ready):

{
  "status": "completed",
  "document_id": "550e8400-e29b-41d4-a716-446655440000",
  "job_id": "42",
  "total_pages": 120
}

Error payload:

{
  "status": "error",
  "document_id": "550e8400-e29b-41d4-a716-446655440000",
  "job_id": "42",
  "error_message": "Description of the error"
}

Design your handler to be idempotent — the worker may retry callbacks.

Example: minimal Python receiver (Flask)

from flask import Flask, request

app = Flask(__name__)
pages = {}

@app.post("/callbacks/apicvm")
def apicvm_callback():
    body = request.json
    doc_id = body["document_id"]
    if body["status"] == "error":
        print(f"Job failed: {body['error_message']}")
        return "", 200

    if body["status"] == "completed":
        ordered = [pages[doc_id][i] for i in sorted(pages.get(doc_id, {}))]
        full_text = "\n\n".join(ordered)
        # chunk → embed → index
        print(f"Done: {doc_id}, {len(ordered)} pages")
        return "", 200

    page_num = body["page"]["number"]
    pages.setdefault(doc_id, {})[page_num] = body["page"]["markdown"]
    return "", 200

Run behind HTTPS in production. Local development may allow http:// when the server has APICVM_CALLBACK_ALLOW_HTTP=true — production requires HTTPS.

Callback URL restrictions

In staging and production:

Invalid URLs return 422 INVALID_CALLBACK_URL with a reason code (PRIVATE_IP, LOCALHOST, etc.).

Wiring into a RAG pipeline

Typical agent loop:

1. Resolve company → list DFP/ITR → pick filing
2. POST extraction with your agent server's callback URL
3. On each page callback, chunk markdown and upsert to vector store
4. On status: "completed", mark document ready for queries

See also: Build a RAG Pipeline with Brazil CVM Filings.

Current limitations

• No job status endpoint — track progress only via callbacks or your own state.
• Callback timeout is 30 seconds per POST; keep handlers fast.
• Extraction quality depends on PDF layout; complex tables may need post-processing.
• Corpus coverage depends on the Hold ingestion pipeline — not every filing is available immediately after CVM publication.

Next steps

• Read the API docs
• Get an API key
• Extract Text from CVM PDFs for AI Agents
• Brazil CVM Filings for AI Agents

Related guides

• Build a RAG Pipeline with Brazil CVM Filings
• Download DFP and ITR Filings from Brazil