In the SageMaker SDK Is there any way to combine / interpolate a ParameterString in a ProcessingInput/Processor/ProcessingStep? - amazon-sagemaker

Question

When using the SageMaker SDK, I would like to use a pipeline parameter ParameterString to construct a S3 path, thus I need to interpolate the ParameterString somehow, python str.format() and f-strings do not properly work with ParameterString.
import sagemaker
from sagemaker.processing import ProcessingInput, ProcessingOutput
from sagemaker.sklearn.processing import SKLearnProcessor
date_parameter = ParameterString(name="date")
p_input = ProcessingInput(
source=f"s3://my-bucket/date={date_parameter}",
destination="/opt/ml/processing/input"),)
What can be used to compose / combine / interpolate pipeline parameters?

Answer 1

The closest equivalent to string interpolation that you can use in a SageMaker Pipeline is sagemaker.workflow.functions.Join
from sagemaker.workflow.execution_variables import ExecutionVariables
from sagemaker.workflow.functions import Join
from sagemaker.workflow.parameters import ParameterString
date_parameter = ParameterString(name="date")
source_variable = Join(on='', values=['s3://bucket-name/date=', date_parameter])
p_input = ProcessingInput(
input_name="xxx",
source=source_variable,
destination="/opt/ml/processing/input",
)
The source_variable in the code above will translate to {'Std:Join': {'On': '', 'Values': ['s3://bucket-name/date=', {'Get': 'Parameters.date'}]}} on the CreatePipeline > PipelineDefinition pipeline definition.
And when the SageMaker Pipeline is actually started that will be evaluated by SageMaker to a literal string