Amazon SageMaker kMeans won't take sparse matrix (csr_matrix) as input, any alternatives before using a dense matrix? - sparse-matrix

Question

I want to apply sagemaker's kMeans algorithm to a sparse matrix, obtained with TfidfVectorizer from sklearn's library.
Ideally I would like to provide the input data to Sagemaker's kMeans implementation as sparse matrix scipy.sparse.csr.csr_matrix, but when I this (kmeans.fit(kmeans.record_set(train_data))) I get the following error:
TypeError: must be real number, not csr_matrix
Of course, if I pass a dense matrix the algorithm will work (train_data.toarray()) but the amount of memory it would need is ginormous. Any possible alternatives before I incur into using supersized amazon instances?

Answer 1

The key was in the SageMaker python SDK. There you can find a function that transforms a scipy sparse matrix to a sparse tensor (write_spmatrix_to_sparse_tensor).
The complete code that solved the problem without having to incur into a dense matrix is the following:
from sagemaker.amazon.common import write_spmatrix_to_sparse_tensor
tfidf_matrix = tfidf_vectorizer.fit_transform('your_train_data') # output: sparse scipy matrix
sagemaker_bucket = 'your-bucket'
data_key = 'kmeans_lowlevel/data'
data_location = f"s3://{sagemaker_bucket}/{data_key}"
buf = io.BytesIO()
write_spmatrix_to_sparse_tensor(buf, tfidf_matrix)
buf.seek(0)
boto3.resource('s3').Bucket(sagemaker_bucket).Object(data_key).upload_fileobj(buf)
After doing this, in the create_training_params configuration you'll have to feed the S3Uri field with the data location you have provided to store the sparse matrix in S3:
create_training_params = \
{
... # all other params
"InputDataConfig": [
{
"ChannelName": "train",
"DataSource": {
"S3DataSource": {
"S3DataType": "S3Prefix",
"S3Uri": data_location, # YOUR_DATA_LOCATION_GOES_HERE
"S3DataDistributionType": "FullyReplicated"
}
},
"CompressionType": "None",
"RecordWrapperType": "None"
}
]
}