Engineering Note
Our Rails app was handling file uploads. Then file processing started affecting everything else.
Initially, the flow was simple.
User uploads a file → Rails receives it → process the file → save the result.
But as file sizes and upload volumes grew, the application had more work to do.
Large uploads consumed server resources.
Processing delayed requests.
Retries became difficult to manage.
We needed to separate uploading from processing.
The architecture becomes:
1. Upload directly to S3
Rails generates a presigned URL, and the browser uploads the file directly to S3.
The file doesn't need to pass through the Rails server.
2. Trigger processing
An S3 event can trigger a Lambda function for suitable workloads.
3. Handle failures safely
Processing should be retryable, with clear status tracking and protection against duplicate events.
4. Keep Rails responsive
Rails manages permissions, metadata and processing status without doing every heavy operation itself.
But Lambda isn't the answer for every workload.
Large files, long-running jobs or complex processing may need ECS tasks or background workers instead.
Important learning:
Scaling file uploads isn't just about increasing server capacity.
It's about separating upload traffic, processing workloads and customer-facing requests so they don't compete for the same resources.
Let S3 handle the files.
Let the right compute service handle the processing.
Let Rails coordinate the workflow.
#Rails #RubyOnRails #AWS #AmazonS3 #AWSLambda #SystemDesign #SaaS #CloudArchitecture
User uploads a file → Rails receives it → process the file → save the result.
But as file sizes and upload volumes grew, the application had more work to do.
Large uploads consumed server resources.
Processing delayed requests.
Retries became difficult to manage.
We needed to separate uploading from processing.
The architecture becomes:
1. Upload directly to S3
Rails generates a presigned URL, and the browser uploads the file directly to S3.
The file doesn't need to pass through the Rails server.
2. Trigger processing
An S3 event can trigger a Lambda function for suitable workloads.
3. Handle failures safely
Processing should be retryable, with clear status tracking and protection against duplicate events.
4. Keep Rails responsive
Rails manages permissions, metadata and processing status without doing every heavy operation itself.
But Lambda isn't the answer for every workload.
Large files, long-running jobs or complex processing may need ECS tasks or background workers instead.
Important learning:
Scaling file uploads isn't just about increasing server capacity.
It's about separating upload traffic, processing workloads and customer-facing requests so they don't compete for the same resources.
Let S3 handle the files.
Let the right compute service handle the processing.
Let Rails coordinate the workflow.
#Rails #RubyOnRails #AWS #AmazonS3 #AWSLambda #SystemDesign #SaaS #CloudArchitecture