How to upload image and pass in to tensorflowjs model to get prediction using reactjs? - reactjs

Question

EDIT
Using the graph model format and the updated code example, I've managed to get it to return a prediction. Issue is now it always returns 1, no matter which image I feed it, so wondering if I am not passing in the right image data?
Second EDIT: Changed the way I was passing in the img object, but still getting 1 for every image I feed it.
I have only just started looking into tensorflowjs and am using a prebuilt keras model I have been given access to. This model is a binary classifier. The model has been saved as an .h5 file and I have been asked to run it in the browser using tensorflowjs and react. Essentially I want to select an image from my local storage or an sd card and feed it to the model to get a yes or no classification.
I’ve followed the tensorflowjs docs in converting the keras model to a TF.js Layers format, but then can’t load the model. I’m getting an error about an unknown layer: RandomFlip. So I then tried converting the model to a graph model as I couldn’t find a solution to the error and thought I’d give it a try. This loaded the model but then there were more issues when feeding it the image. The shape of dict['image_tensor'] provided in model.execute(dict) must be [-1,380,380,1], but was [380, 380] . Then I searched for that and got it to resize to [-1,380,380,1] , but then it was complaining about size not being the expected, so I thought maybe I've messed up in some of the previous steps.
To convert to a graph model I used the following command: tensorflowjs_converter --input_format keras --output_format tfjs_layers_model /Users/myUser/Documents/save_at_45.h5 /Users/myUser/Documents/convert-keras-model and in my code loading it with the loadGraphModel method. Following this path has at least allowed me to load the model.
I also tried converting it to a Layers format with: tensorflowjs_converter --input_format keras --output_format tfjs_layers_model /Users/myUser/Documents/save_at_45.h5 /myUser/mariomendes/Documents/convert-keras-model and in my code loading it with the loadLayersModel. This returns the error Unknown layer: RandomFlip. I've tried searching for a solution to this, but haven't been able to find one.
Does knowing it is a .h5 file mean I should know if it needs to be converted to a tf Graph format or Layers format or is there something else that determines which format it should be converted to?
I've stored the converted model in both formats and it's weights in S3 and am getting it from there.
For my react code I have done the following:
import React, { useState, useEffect } from "react";
import "./index.css";
import * as tf from "#tensorflow/tfjs";
function ImgImporter() {
const [file, setFile] = useState(null);
const [model, setModel] = useState(null);
const [processing, setProcessing] = useState(false);
const [prediction, setPrediction] = useState(null);
const [imageLoaded, setImageLoaded] = useState(false);
function readImage(file) {
return new Promise((rs, rj) => {
const fileReader = new FileReader();
fileReader.onload = () => rs(fileReader.result);
fileReader.onerror = () => rj(fileReader.error);
fileReader.readAsDataURL(file);
});
}
async function handleImgUpload(event) {
const {
target: { files },
} = event;
const _file = files[0];
const fileData = await readImage(_file);
setFile(fileData);
setProcessing(true);
}
useEffect(() => {
async function loadModel() {
if (!model) {
const _model = await tf.loadGraphModel("/model.json");
setModel(_model);
}
}
loadModel();
});
useEffect(() => {
async function predict() {
if (imageLoaded && file) {
const imageElement = document.createElement("img");
imageElement.src = file;
imageElement.onload = async () => {
const tensor = tf.browser
.fromPixels(imageElement, 1)
.resizeNearestNeighbor([380, 380])
.expandDims()
.toFloat();
const prediction = await model.predict(tensor).data();
setPrediction(parseInt(prediction, 10));
setProcessing(false);
setImageLoaded(false);
};
}
}
predict();
}, [imageLoaded, model, file]);
return (
<div className="File-input-container">
<form className="Form">
<label htmlFor="upload-image">Upload image</label>
<input
id="image-selector"
type="file"
name="upload-image"
accept="image/*"
className="File-selector"
onChange={handleImgUpload}
disabled={!model || processing}
/>
</form>
<div className="Img-display-container">
<img
onLoad={() => {
setImageLoaded(true);
}}
alt=""
src={file}
/>
</div>
<div className="Img-processing-container">
{processing ? (
<p>Loading ...</p>
) : prediction !== null ? (
<div>
<p>{prediction === 1 ? "Yes" : "No"}</p>
</div>
) : null}
</div>
</div>
);
}
export default ImgImporter;
When I upload an image this is returning the following result in the console as the value of prediction:
dataId: {id: 195}
dtype: "float32"
id: 94
isDisposedInternal: false
kept: false
rankType: "2"
scopeId: 6
shape: (2) [1, 1]
size: 1
strides: [1]
Would be great if someone could shed some light on this or help me finding the right direction.

Answer 1

If you want to get the value, you can use prediction.dataSync() or its promise counterpart await prediction.data()

Answer 2

Regarding your second edit. You're converting the image to a float, but does the model expect a normalized float? You might need to append .div(255) or whatever normalization is needed. Please post the specs for your model.
Also, as stated by edkeveked, you can used dataSync() to get your data, but it's worth noting you could have also used arraySync too, which would maintain the returned tensor depth.
Also, I noticed you didn't do any cleanup. So your tensors will build up in GPU memory. Don't forget to dispose.
I hope these things help.