Introduction

SageMaker does not run cron jobs by itself, so scheduling is usually built with surrounding AWS services. The most common pattern is EventBridge schedule triggers a target, and that target starts a SageMaker Pipeline, processing job, or training job. Reliable scheduling depends on idempotent job starts, clear execution names, and observability.

Pick the Right Trigger Pattern

You have three common options:

• EventBridge rule that invokes Lambda, then Lambda starts SageMaker work
• EventBridge rule that starts Step Functions, then workflow controls SageMaker steps
• EventBridge Scheduler for timezone-aware schedules and one-off schedules

For most teams, Lambda plus EventBridge is enough and easy to debug.

Lambda Function to Start a SageMaker Pipeline

This example Lambda starts a pipeline execution and uses a timestamp in the execution display name so each run is traceable.

1import os
2import boto3
3from datetime import datetime, timezone
4
5sm = boto3.client("sagemaker")
6
7PIPELINE_NAME = os.environ["PIPELINE_NAME"]
8
9def lambda_handler(event, context):
10    stamp = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
11    exec_name = f"scheduled-{stamp}"
12
13    response = sm.start_pipeline_execution(
14        PipelineName=PIPELINE_NAME,
15        PipelineExecutionDisplayName=exec_name,
16        PipelineParameters=[
17            {"Name": "run_date", "Value": stamp}
18        ],
19    )
20
21    return {
22        "pipeline": PIPELINE_NAME,
23        "execution_arn": response["PipelineExecutionArn"],
24        "event_id": event.get("id"),
25    }

Set PIPELINE_NAME as a Lambda environment variable. Give the Lambda role permission for sagemaker:StartPipelineExecution and logging permissions for CloudWatch Logs.

Create a Daily EventBridge Rule

Use boto3 to create a rule and connect it to your Lambda target.

1import json
2import boto3
3
4events = boto3.client("events")
5
6RULE_NAME = "daily-ml-pipeline"
7LAMBDA_ARN = "arn:aws:lambda:us-east-1:123456789012:function:start-ml-pipeline"
8
9events.put_rule(
10    Name=RULE_NAME,
11    ScheduleExpression="cron(0 12 * * ? *)",
12    State="ENABLED",
13    Description="Run SageMaker pipeline daily at 12:00 UTC",
14)
15
16events.put_targets(
17    Rule=RULE_NAME,
18    Targets=[
19        {
20            "Id": "startPipelineLambda",
21            "Arn": LAMBDA_ARN,
22            "Input": json.dumps({"source": "eventbridge-schedule"}),
23        }
24    ],
25)
26
27print("Rule and target configured")

After setting the target, add Lambda invoke permission so EventBridge can call the function.

1aws lambda add-permission \
2  --function-name start-ml-pipeline \
3  --statement-id allow-eventbridge \
4  --action lambda:InvokeFunction \
5  --principal events.amazonaws.com \
6  --source-arn arn:aws:events:us-east-1:123456789012:rule/daily-ml-pipeline

Pass Dynamic Parameters Per Run

Scheduled runs are more useful when each execution has context. Include date ranges, input S3 prefixes, or model variant names. Keep parameter construction in Lambda so schedule remains simple.

If you run multiple variants, avoid separate rules for each unless schedules differ. One rule can invoke Lambda with payload indicating variant key, and Lambda can map that key to pipeline parameters.

Reliability and Cost Controls

Scheduled ML jobs can fail silently if you only inspect SageMaker manually. Add these controls:

• CloudWatch alarm for Lambda errors
• alarm or notification on pipeline execution failure states
• dead-letter queue for Lambda async failures
• execution timeout and retry policy tuned for expected duration
• idempotency check to avoid duplicate runs during retries

A practical idempotency rule is one run per logical schedule window, stored in DynamoDB. Lambda checks the table before starting execution.

Validate the Schedule End to End

Do one manual validation before declaring success:

• trigger Lambda manually and confirm pipeline starts
• trigger EventBridge rule and verify Lambda log entry
• verify pipeline parameters match expected schedule window
• check alert path by simulating a controlled failure

This catches permission gaps early and avoids silent non-execution in production.

Common Pitfalls

• Creating EventBridge rule but forgetting Lambda invoke permission.
• Hardcoding execution names and causing name collisions.
• Scheduling in UTC without documenting local-time expectation.
• Retrying failed runs without idempotency, causing duplicate processing.
• Monitoring only Lambda success and not actual pipeline outcome.

Summary

• SageMaker scheduling is typically EventBridge plus Lambda or Step Functions.
• Start with a small Lambda that triggers pipeline execution with traceable names.
• Add EventBridge rule, target, and explicit invoke permission.
• Pass run-specific parameters so scheduled jobs are context aware.
• Use alarms and idempotency checks to keep scheduled workflows reliable.