What's the simplest reliable way to encode multiple jpeg images in a single byte string? - multipartform-data

Question

I need to publish a Google Cloud Pub/Sub message with multiple jpeg images. It needs to go in the data body. Putting it as a base64-encoded string in an attribute won't work, because attribute values are limited to 1024 bytes:
https://cloud.google.com/pubsub/quotas#resource_limits
What's a simple and reliable pattern for doing that? It might seem possible to choose some fixed delimiter, but I want to avoid the possibility of that delimiter occurring inside an image. Is it possible something like |||| might occur in a jpeg byte array? Another possibility might seem to encode as multi-part mime, but I haven't found any general-purpose non-http libraries to do that. I need implementations in both Java/Scala and Python. Or maybe can I just concatenate the jpeg byte arrays without any external delimiter, and split them based on header identifiers?

Answer 1

You probably want to store data in some kind of schema-based message using something like Avro or Protocol Buffers. Both can generate code that can be used to serialize and deserialize messages in Java/Scala and Python.
For example, in protocol buffers, you could create a message in a file image.proto:
syntax = "proto3";
message Images {
bytes images = 1;
}
You could generate the python code for this with protoc compiler:
$ protoc -I=. --python_out=. image.proto
In Python3, to add images, serialize the message, and send it, you would do the following:
import image_pb2
from google.cloud import pubsub_v1
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(<project name>, <topic name>)
def send_images(images):
img_msg = image_pb2.Images()
for i in images:
img_msg.images.append(i)
msg_data = img_msg.SerializeToString()
message_future = publisher.publish(topic_path, data=msg_data)
print(message_future.result())
To receive the images and process them:
import image_pb2
from google.cloud import pubsub_v1
def receive(message):
images = image_pb2.Images()
images.ParseFromString(message.data)
for i in images.images:
# Process the image
message.ack()
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(<project name>, <subscription name>)
subscribe_future = subscriber.subscribe(subscription_path, receive)
print(subscribe_future.result())

Answer 2

It looks like the following approach may work, written in Scala, using only natural delimiters:
def serializeJpegs(jpegs: Seq[Array[Byte]]): Array[Byte] =
jpegs.foldLeft(Array.empty[Byte])(_ ++ _)
def deserializeJpegs(bytes: Array[Byte]): Seq[Array[Byte]] = {
val JpegHeader = Array(0xFF.toByte, 0xD8.toByte)
val JpegFooter = Array(0xFF.toByte, 0xD9.toByte)
val Delimiter = JpegFooter ++ JpegHeader
val jpegs: mutable.Buffer[Array[Byte]] = mutable.Buffer.empty
var (start, end) = (0, 0)
end = bytes.indexOfSlice(Delimiter, start) + JpegFooter.length
while (end > JpegFooter.length) {
jpegs += bytes.slice(start, end)
start = end
end = bytes.indexOfSlice(Delimiter, start) + JpegFooter.length
}
if (start < bytes.length) {
jpegs += bytes.drop(start)
}
jpegs
}
I'm sure there's a more efficient and functional implementation, but that's for another day!