Run SageMaker Batch transform failed on loading model - amazon-sagemaker

Question

I am trying to run batch transform job with HuggineFace class and fine-tuned model and custom inference file.
The job failed on loading the model but I could load it locally.
I need to make custom inference file because i need to keep the input file as is, so i had to change the input key from the input json file.
Here are the exception :
PredictionException(str(e), 400)
2022-05-08 16:49:45,499 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: Can't load config for '/.sagemaker/mms/models/model'. Make sure that:
2022-05-08 16:49:45,499 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2022-05-08 16:49:45,499 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - - '/.sagemaker/mms/models/model' is a correct model identifier listed on 'https://huggingface.co/models'
2022-05-08 16:49:45,499 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2022-05-08 16:49:45,500 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - - or '/.sagemaker/mms/models/model' is the correct path to a directory containing a config.json file
2022-05-08 16:49:45,500 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2022-05-08 16:49:45,500 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - : 400
I am running on script mode:
from sagemaker.huggingface.model import HuggingFaceModel
hub = {
# 'HF_MODEL_ID':'cardiffnlp/twitter-roberta-base-sentiment',
'HF_TASK':'text-classification',
'INPUT_TEXTS': 'Description'
}
huggingface_model = HuggingFaceModel(model_data='../model/model.tar.gz',
role=role,
source_dir="../model/pytorch_model/code",
transformers_version="4.6",
pytorch_version="1.7",
py_version="py36",
entry_point="inference.py",
env=hub
)
batch_job = huggingface_model.transformer(
instance_count=1,
instance_type='ml.p3.2xlarge',
output_path=output_s3_path, # we are using the same s3 path to save the output with the input
strategy='SingleRecord',
accept='application/json',
assemble_with='Line'
)
batch_job.transform(
data=s3_file_uri,
content_type='application/json',
split_type='Line',
#input_filter='$[1:]',
join_source='Input'
)
Custom inference.py
import json
import os
from transformers import pipeline
import torch
DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'
def model_fn(model_dir):
model = pipeline(task=os.environ.get('HF_TASK', 'text-classification'), model=model_dir, tokenizer=model_dir)
return model
def transform_fn(model, input_data, content_type, accept):
input_data = json.loads(input_data)
input_text = os.environ.get('INPUT_TEXTS', 'inputs')
inputs = input_data.pop(input_text, None)
parameters = input_data.pop("parameters", None)
# pass inputs with all kwargs in data
if parameters is not None:
prediction = model(inputs, **parameters)
else:
prediction = model(inputs)
return json.dumps(
prediction,
ensure_ascii=False,
allow_nan=False,
indent=None,
separators=(",", ":"),
)

Answer 1

I think the issue is with the "model_data" parameter. It should point to an S3 object(model.tar.gz).
Then the transform job will download the model file from S3 and load it.

Answer 2

The solution is change the "task" in the pipline to "sentiment-analysis"
hub = {
# 'HF_MODEL_ID':'cardiffnlp/twitter-roberta-base-sentiment',
'HF_TASK':'sentiment-analysis',
'INPUT_TEXTS': 'Description'
}