Loading request...
Request for guidance on scaling document extraction pipelines in AWS, specifically for processing large batches of files efficiently.
I'm building a document extraction pipeline on AWS for a client. PDFs go into S3, which triggers a Lambda chain: PDF concatenation -> text extraction (Textract + Bedrock VLM fallback) -> PII redaction (Comprehend) -> structured LLM extraction (Gemini via Fargate). Currently working with \~10 docs and it runs fine, but we need to scale to 500+ docs uploaded in bulk. What should I be thinking about? Main concerns are API rate limits, Lambda concurrency, and whether Fargate-per-file makes sense at scale.