Troubles with BlazingText jsonlines Batch Transform - amazon-sagemaker

Question

I have a jsonlines file that looks like this:
{"id":123,"source":"this is a text string"}
{"id":456,"source":"this is another text string"}
{"id":789,"source":"yet another string"}
When I run a BlazingText Batch Transform job on a file that just contains the source, it works. When trying to join the inputs and outputs, I get Customer Error: Unable to decode payload: Incorrect data format. (caused by AttributeError).
Any suggestions?
Code:
bt_transformer = bt_model.transformer(
instance_count = 1,
instance_type = "ml.m4.xlarge",
assemble_with = "Line",
output_path = s3_batch_out_data,
accept = "application/jsonlines"
)
bt_transformer.transform(
s3_batch_in_data,
content_type = "application/jsonlines",
split_type = "Line",
input_filter = "$.source",
join_source = "Input",
output_filter = "$['id', 'SageMakerOutput']"
)
bt_transformer.wait()

Answer 1

When applying "$.source" on {"id":123,"source":"this is a text string"}, the output is "this is a text string" instead of {"source":"this is a text string"} which is probably why you got a format error. I'm wondering why you need to do such filtering on a JSON input - doesn't the algorithm ignore unrecognized JSON fields automatically?