Sagemaker on The Edge Cases

import os Causing Counterintuitive Errors in Sagemaker

Mon, 10 Apr 2023 00:52:17 +0000

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/function/awslambdaric/__main__.py", line 25, in <module>
    main(sys.argv)
  File "/function/awslambdaric/__main__.py", line 19, in main
    lambda_runtime_api_addr = os.environ["AWS_LAMBDA_RUNTIME_API"]
  File "/usr/lib/python3.8/os.py", line 675, in __getitem__
    raise KeyError(key) from None
KeyError: 'AWS_LAMBDA_RUNTIME_API'

~~My suspicion is that the our internal package requires os as a dependency to open a file, and that causes some kind of clash with AWS.~~

But it gets weirder! The dependency’s Constants module has an import os that works just fine! BUT it uses this to get the assets path:

Logging from Custom Docker Code in Sagemaker

Tue, 10 Jan 2023 18:05:43 +0000

Unlike in AWS Lambda code, in Sagemaker training jobs using custom Python code in Docker containers logging won’t work unless a stream handler is specifically added to the logger.

logger = logging.getLogger("__main__")
stream_handler = logging.StreamHandler(sys.stdout)
logger.addHandler(stream_handler)

SAGEMAKER_PROGRAM in Custom Dockerfile

Tue, 10 Jan 2023 03:47:47 +0000

At the end of a Dockerfile that defines a container for a Sagemaker training job, you have to define the SAGEMAKER_PROGRAM environment variable. It needs to point directly to the script that acts as the entry point as it exists within /opt/ml/code, rather than the full path.

ENV SAGEMAKER_PROGRAM train.py

How to Pass Arguments to Training Code in Sagemaker via a Step Function

Tue, 29 Mar 2022 03:43:22 +0000

To pass arguments to training code in Sagemaker by itself or via a Step Function, set up an argument parser in the Python script with keyword arguments.

Example

Implement an argument parser in the Sagemaker entry point script. For example, in a Python script:

 import argparse

 if __name__ == "__main__":
 parser = argparse.ArgumentParser

 parser.add_argument("--learning-rate", type=int, default=1)
 parser.add_argument("--batch-size", type=int, default=64)
 parser.add_argument("--communicator", type=str)
 parser.add_argument("--frequency", type=int, default=20)

 args = parser.parse_args
...

Start a training job with hyperparameters.

Input and Output Channels for Sagemaker Training Jobs

Wed, 27 Oct 2021 19:08:01 +0000

Inputs

With SageMaker training jobs, you specify an “input channel” and configure it in the job definition. Each channel is a dictionary that specifies the data split. Each of these items will be made available on the instance in /opt/ml/input/data/<stage>.

JSON Example

{
 "train": {
 "ContentType": "trainingContentType",
 "TrainingInputMode": "File",
 "S3DistributionType": "FullyReplicated",
 "RecordWrapperType": "None"
 },
 "evaluation": {
 "ContentType": "evalContentType",
 "TrainingInputMode": "File",
 "S3DistributionType": "FullyReplicated",
 "RecordWrapperType": "None"
 },
 "validation": {
 "TrainingInputMode": "File",
 "S3DistributionType": "FullyReplicated",
 "RecordWrapperType": "None"
 }
}

CDK Example

Source

Outputs

With SageMaker training jobs, model artifacts should be saved to /opt/ml/model/ so that, with the training job output configuration, they will be zipped and added to S3. Failures can be written to the file /opt/ml/output/failure and the first 1024 characters will be returned as FailureReason.