Sagemaker: DeepAR Hyperparameter Tuning Error - amazon-sagemaker

Question

Running into a new issue with tuning DeepAR on SageMaker when trying to initialize a hyperparameter tuning job - this error also occurs when calling the test:mean_wQuantileLoss. I've upgraded the sagemaker package, restarted my instance, restarted the kernel (using a juptyer notebook), and yet the problem persists.
ClientError: An error occurred (ValidationException) when calling the
CreateHyperParameterTuningJob operation: The objective metric type, [Maximize], that you specified for objective metric, [test:RMSE], isn’t valid for the [156387875391.dkr.ecr.us-west-2.amazonaws.com/forecasting-deepar:1] algorithm. Choose a valid objective metric type.
Code:
my_tuner = HyperparameterTuner(estimator=estimator,
objective_metric_name="test:RMSE",
hyperparameter_ranges=hyperparams,
max_jobs=20,
max_parallel_jobs=2)
# Start hyperparameter tuning job
my_tuner.fit(inputs=data_channels)
Stack Trace:
ClientError Traceback (most recent call last)
<ipython-input-66-9d6d8de89536> in <module>()
7
8 # Start hyperparameter tuning job
----> 9 my_tuner.fit(inputs=data_channels)
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/tuner.py in fit(self, inputs, job_name, include_cls_metadata, **kwargs)
255
256 self._prepare_for_training(job_name=job_name, include_cls_metadata=include_cls_metadata)
--> 257 self.latest_tuning_job = _TuningJob.start_new(self, inputs)
258
259 #classmethod
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/tuner.py in start_new(cls, tuner, inputs)
525 output_config=(config['output_config']),
526 resource_config=(config['resource_config']),
--> 527 stop_condition=(config['stop_condition']), tags=tuner.tags)
528
529 return cls(tuner.sagemaker_session, tuner._current_job_name)
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in tune(self, job_name, strategy, objective_type, objective_metric_name, max_jobs, max_parallel_jobs, parameter_ranges, static_hyperparameters, image, input_mode, metric_definitions, role, input_config, output_config, resource_config, stop_condition, tags)
348 LOGGER.info('Creating hyperparameter tuning job with name: {}'.format(job_name))
349 LOGGER.debug('tune request: {}'.format(json.dumps(tune_request, indent=4)))
--> 350 self.sagemaker_client.create_hyper_parameter_tuning_job(**tune_request)
351
352 def stop_tuning_job(self, name):
~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
312 "%s() only accepts keyword arguments." % py_operation_name)
313 # The "self" in this scope is referring to the BaseClient.
--> 314 return self._make_api_call(operation_name, kwargs)
315
316 _api_call.__name__ = str(py_operation_name)
~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
610 error_code = parsed_response.get("Error", {}).get("Code")
611 error_class = self.exceptions.from_code(error_code)
--> 612 raise error_class(parsed_response, operation_name)
613 else:
614 return parsed_response
ClientError: An error occurred (ValidationException) when calling the CreateHyperParameterTuningJob operation:
The objective metric type, [Maximize], that you specified for objective metric, [test:RMSE], isn’t valid for the [156387875391.dkr.ecr.us-west-2.amazonaws.com/forecasting-deepar:1] algorithm.
Choose a valid objective metric type.

Answer 1

It looks like you are trying to maximize this metric, test:RMSE can only be minimized by SageMaker HyperParameter Tuning.
To achieve this in the SageMaker Python SDK, create your HyperparameterTuner with objective_type='Minimize'. You can see the signature of the init method here.
Here is the change you should make to your call to HyperparameterTuner:
my_tuner = HyperparameterTuner(estimator=estimator,
objective_metric_name="test:RMSE",
objective_type='Minimize',
hyperparameter_ranges=hyperparams,
max_jobs=20,
max_parallel_jobs=2)