Step-Functions on The Edge Cases

Using Step Functions for Fan-Out or Scatter-Gather

Mon, 01 Jul 2024 20:57:17 +0000

You can use a Map State for this. You define it like so in CDK:

from aws_cdk import aws_stepfunctions as sfn, aws_stepfunctions_tasks as sfn_tasks

map_lambda_task = sfn_tasks.LambdaInvoke(
    self,
    "task-name",
    lambda_function=function_lambda,
    payload=sfn.TaskInput.from_object(
        {
            "item_key": sfn.JsonPath.string_at("$.item_key")
        }
    ),
    output_path="$.Payload"
)
map_state = sfn.Map(
    self,
    "state-name",
    items_path="$.item_key",
    input_path="$,
    parameters={
        "item_key.$": "$$.Map.Item.Value"
    }
)
map_state.item_processor(processor=map_lambda_task)

This sets up a lambda task (using an existing Lambda function, not shown) to be invoked as a Step Function task, and the map state that calls it in the iterator. The iterator iterates through each item in the Map’s input, then calls the task for each item in the list passed to the state as input, using a single item (whether it’s a primitive or another container like a list or dict).

Catching and Responding To Failures in AWS Step Functions

Tue, 07 Jun 2022 18:40:26 +0000

To handle errors within a Step Function, you can use a Catch block that hands off execution to an Error State. This State can be a Task type, which can (among other things) trigger a Lambda that does the actual error handling.

Example

## Step function failure handler - in the future this will be more complex than writing to DB
failure_handler = lambda_python.PythonFunction(
 self,
 f"failure-handler",
 entry="lambda/",
 index="failure_handler.py",
 runtime=_lambda.Runtime.PYTHON_3_9,
)
failure_task = sfn_tasks.LambdaInvoke(
 self,
 f"failure-lambda-task",
 lambda_function=training_pipeline_failure_handler,
 output_path="$",
)

## State machine definition
task_one.add_catch(failure_task, result_path="$.error")
task_two.add_catch(failure_task, result_path="$.error")
state_machine_def = task_one.next(task_two)
state_machine = sfn.StateMachine(
 self, f"failing-state-machine", definition=state_machine_def
)

Sources

AWS Step Function Documentation - handling error conditions

How to Pass Arguments to Training Code in Sagemaker via a Step Function

Tue, 29 Mar 2022 03:43:22 +0000

To pass arguments to training code in Sagemaker by itself or via a Step Function, set up an argument parser in the Python script with keyword arguments.

Example

Implement an argument parser in the Sagemaker entry point script. For example, in a Python script:

 import argparse

 if __name__ == "__main__":
 parser = argparse.ArgumentParser

 parser.add_argument("--learning-rate", type=int, default=1)
 parser.add_argument("--batch-size", type=int, default=64)
 parser.add_argument("--communicator", type=str)
 parser.add_argument("--frequency", type=int, default=20)

 args = parser.parse_args
...

Start a training job with hyperparameters.