I converted a keras model into a tensorflowjs model using the simple tensorflowjs_converter --input_format keras ./model/L_keypoint_classifier_final.h5 L_layer_model. I managed to get this model working on a .ts (TypeScript) file.
Now I am focused on deploying this model using React and Typescript (in a .tsx file). My app component is loading the models as such:
const [models, setModels] = useState<Models>({
L_Model: undefined,
R_Model: undefined,});
useEffect(() => {
loadModel().then((models) => {
setModels(models);
setIsLoading(false);
}); }
The loadModel() function is exported from another file and it is:
export async function loadModel() {
let result: Models = { R_Model: undefined, L_Model: undefined };
result.R_Model = await tf.loadLayersModel("/right/model.json");
result.L_Model = await tf.loadLayersModel("/left/model.json");
return result;
}
That directory of the models is in the public folder of my project. After loading the models in the app component, I pass them to a child component using props.
<Camera models={models}></Camera>
They are received in the camera component as:
const Camera: FunctionComponent<CameraProps> = (props) => {
const { R_Model, L_Model } = props.models;
In the camera component I pass in a tf.Tensor2D. This tensor does in fact contain values that I checked. But when I pass them to the model.predict() function, it just returns a tensor full of NaNs.
This is my code for preprocessing the input and passing it to the model:
//Preprocess Landmarks
//#ts-ignore
let landmark_list = calc_landmark_list(landmarks);
landmark_list = pre_process_landmarks(landmark_list);
//#ts-ignore
landmark_list = tf.tensor2d([landmark_list]);
console.log(landmarks_list.dataSync());
let prediction;
if(isRightHand){
prediction = R_Model?.predict(landmark_list);
}else{
prediction = L_Model?.predict(landmark_list);
}
const scores = prediction.arraySync()[0];
After that, I try to find the maxScore of the predictions, but since arraySync() returns a NaN array, it does not work. My team and me have try searchig for different options. Some include wrapping the predict function inside an aasync function, but that doesn't seem to work either (or maybe we have done it incorrectly, although we have followed the examples thoroughly).
The console.log of the landmark_list.dataSync() prints out:
Float32Array(42) [0, 0, -0.2683601677417755, -0.1023331806063652, -0.4781370162963867, -0.397993803024292, -0.5191399455070496, -0.6676312685012817, -0.46050554513931274, -0.8477477431297302, -0.30691489577293396, -0.9023468494415283, -0.49582260847091675, -1, -0.5734853148460388, -0.7551659941673279, -0.5509241223335266, -0.5708747506141663, -0.15572300553321838, -0.9109046459197998, -0.38624173402786255, -0.9391834735870361, -0.4641483426094055, -0.6930190920829773, -0.4609870910644531, -0.49743616580963135, -0.00984301045536995, -0.8530527353286743, -0.25299814343452454, -0.7750100493431091, -0.32405075430870056, -0.5182365775108337, -0.32825687527656555, -0.3154793083667755, 0.11740472167730331, -0.7356364130973816, -0.12479904294013977, -0.6477926969528198, -0.21985816955566406, -0.43255504965782166, -0.24492989480495453, -0.25398018956184387, buffer: ArrayBuffer(168), byteLength: 168, byteOffset: 0, length: 42, Symbol(Symbol.toStringTag): 'Float32Array']
Related
I'm new to tensorflowjs and I'm struggling to implement some custom layers, if someone could point me in the right direction that would be really helpful!
For example, I have a layer in InceptionResnetV1 architecture where I'm multiplying the layer by a constant scale (this was originally an unsupported Lambda layer which I'm switching out for a custom layer), but the value of this scale changes per block. This works fine in Keras with an implementation such as below, and using load_model with ScaleLayer in the custom objects
class ScaleLayer(tensorflow.keras.layers.Layer):
def __init__(self, **kwargs):
super(ScaleLayer, self).__init__(**kwargs)
def call(self, inputs, **kwargs):
return tensorflow.multiply(inputs, kwargs.get('scale'))
def get_config(self):
return {}
x = ScaleLayer()(x, scale = tensorflow.constant(scale))
I tried defining this in a similar way in javascript and then registered the class
class ScaleLayer extends tf.layers.Layer {
constructor(config?: any) {
super(config || {});
}
call(input: tf.Tensor, kwargs: Kwargs) {
return tf.tidy(() => {
this.invokeCallHook(input, kwargs);
const a = input;
const b = kwargs['scale'];
return tf.mul(a, b);
});
}
static get className() {
return 'ScaleLayer';
}
}
tf.serialization.registerClass(ScaleLayer);
However I'm finding that the kwargs are always empty. I tried another similar method where I passed scale as another dimension of the input, then did input[0] * input[1], which again worked fine for the keras model but not in javascript.
I feel like I'm missing something key on the way to defining this kind of custom layer with a changing value per block on the javascript end, so if someone would be able to point me in the right direction it would be much appreciated! Thanks.
constructor(config?: any) {
super(config || {});
}
The config are passed to the parent constructor. But as indicated by the question, the ScaleLayer layer also needs to keep some config properties
constructor(config?: any) {
super(config || {});
// this.propertyOfInterest = config.propertyOfInterest
// make sure that config is an object;
this.scale = config.scale
}
Then for the computation, the ScaleLayer property propertyOfInterest can be used
call(input: tf.Tensor) {
return tf.tidy(() => {
this.invokeCallHook(input, kwargs);
const a = input;
return tf.mul(a, this.scale);
});
}
Use the layer this way:
const model = tf.sequential();
...
model.add(new ScaleLayer({scale: 1}));
...
I have this Firebase structure:
Firebase Structure.
Then I have a function in my Code, which adds a map called "set".
My Structure is looking like this after: New structure.
Now i want an onUpdate Firebase function, which is called after the map "set" is added in any document.
This function should shuffle my "question" array.
I tried something like this:
exports.shuffleSet = functions.firestore
.document('duell/{duell_id}/set/questions')
.onUpdate((change, context) => {
const data = change.after.data();
const previousData = change.before.data();
if (data.name == previousData.name) {
return null;
}
//shuffle code here
});
But Im not sure if .document('duell/{duell_id}/set/questions') is the correct way to navigate to the question array. And at the beginning the "set" is not even existing as explained at the top.
How do I navigate to the question array correctly, that I can pull it & update it shuffled?
You should pass a document path to functions.firestore.document(). You cannot pass a field name, since Firestore Cloud Functions are triggered by documents events.
So you should do as follows:
exports.shuffleSet = functions.firestore
.document('duell/{duell_id}')
.onUpdate((change, context) => {
if (!change.after.data().shuffledSet) {
const data = change.after.data();
const question = data.set.question; // get the value of the question field
const shuffledSet = shuffle(question); // generate the new, suffled set. It’s up to you to write this function
return change.after.ref.update({shuffledSet});
} else {
return null; // Nothing to do, the shuffled field is already calculated
}
});
EDIT
Using the graph model format and the updated code example, I've managed to get it to return a prediction. Issue is now it always returns 1, no matter which image I feed it, so wondering if I am not passing in the right image data?
Second EDIT: Changed the way I was passing in the img object, but still getting 1 for every image I feed it.
I have only just started looking into tensorflowjs and am using a prebuilt keras model I have been given access to. This model is a binary classifier. The model has been saved as an .h5 file and I have been asked to run it in the browser using tensorflowjs and react. Essentially I want to select an image from my local storage or an sd card and feed it to the model to get a yes or no classification.
I’ve followed the tensorflowjs docs in converting the keras model to a TF.js Layers format, but then can’t load the model. I’m getting an error about an unknown layer: RandomFlip. So I then tried converting the model to a graph model as I couldn’t find a solution to the error and thought I’d give it a try. This loaded the model but then there were more issues when feeding it the image. The shape of dict['image_tensor'] provided in model.execute(dict) must be [-1,380,380,1], but was [380, 380] . Then I searched for that and got it to resize to [-1,380,380,1] , but then it was complaining about size not being the expected, so I thought maybe I've messed up in some of the previous steps.
To convert to a graph model I used the following command: tensorflowjs_converter --input_format keras --output_format tfjs_layers_model /Users/myUser/Documents/save_at_45.h5 /Users/myUser/Documents/convert-keras-model and in my code loading it with the loadGraphModel method. Following this path has at least allowed me to load the model.
I also tried converting it to a Layers format with: tensorflowjs_converter --input_format keras --output_format tfjs_layers_model /Users/myUser/Documents/save_at_45.h5 /myUser/mariomendes/Documents/convert-keras-model and in my code loading it with the loadLayersModel. This returns the error Unknown layer: RandomFlip. I've tried searching for a solution to this, but haven't been able to find one.
Does knowing it is a .h5 file mean I should know if it needs to be converted to a tf Graph format or Layers format or is there something else that determines which format it should be converted to?
I've stored the converted model in both formats and it's weights in S3 and am getting it from there.
For my react code I have done the following:
import React, { useState, useEffect } from "react";
import "./index.css";
import * as tf from "#tensorflow/tfjs";
function ImgImporter() {
const [file, setFile] = useState(null);
const [model, setModel] = useState(null);
const [processing, setProcessing] = useState(false);
const [prediction, setPrediction] = useState(null);
const [imageLoaded, setImageLoaded] = useState(false);
function readImage(file) {
return new Promise((rs, rj) => {
const fileReader = new FileReader();
fileReader.onload = () => rs(fileReader.result);
fileReader.onerror = () => rj(fileReader.error);
fileReader.readAsDataURL(file);
});
}
async function handleImgUpload(event) {
const {
target: { files },
} = event;
const _file = files[0];
const fileData = await readImage(_file);
setFile(fileData);
setProcessing(true);
}
useEffect(() => {
async function loadModel() {
if (!model) {
const _model = await tf.loadGraphModel("/model.json");
setModel(_model);
}
}
loadModel();
});
useEffect(() => {
async function predict() {
if (imageLoaded && file) {
const imageElement = document.createElement("img");
imageElement.src = file;
imageElement.onload = async () => {
const tensor = tf.browser
.fromPixels(imageElement, 1)
.resizeNearestNeighbor([380, 380])
.expandDims()
.toFloat();
const prediction = await model.predict(tensor).data();
setPrediction(parseInt(prediction, 10));
setProcessing(false);
setImageLoaded(false);
};
}
}
predict();
}, [imageLoaded, model, file]);
return (
<div className="File-input-container">
<form className="Form">
<label htmlFor="upload-image">Upload image</label>
<input
id="image-selector"
type="file"
name="upload-image"
accept="image/*"
className="File-selector"
onChange={handleImgUpload}
disabled={!model || processing}
/>
</form>
<div className="Img-display-container">
<img
onLoad={() => {
setImageLoaded(true);
}}
alt=""
src={file}
/>
</div>
<div className="Img-processing-container">
{processing ? (
<p>Loading ...</p>
) : prediction !== null ? (
<div>
<p>{prediction === 1 ? "Yes" : "No"}</p>
</div>
) : null}
</div>
</div>
);
}
export default ImgImporter;
When I upload an image this is returning the following result in the console as the value of prediction:
dataId: {id: 195}
dtype: "float32"
id: 94
isDisposedInternal: false
kept: false
rankType: "2"
scopeId: 6
shape: (2) [1, 1]
size: 1
strides: [1]
Would be great if someone could shed some light on this or help me finding the right direction.
If you want to get the value, you can use prediction.dataSync() or its promise counterpart await prediction.data()
Regarding your second edit. You're converting the image to a float, but does the model expect a normalized float? You might need to append .div(255) or whatever normalization is needed. Please post the specs for your model.
Also, as stated by edkeveked, you can used dataSync() to get your data, but it's worth noting you could have also used arraySync too, which would maintain the returned tensor depth.
Also, I noticed you didn't do any cleanup. So your tensors will build up in GPU memory. Don't forget to dispose.
I hope these things help.
I want to add a custom meta field to the gutenberg document panel and used this doc. For the custom meta field I used this tutorial.
The problem I have occurs when trying to put them together.
Here is my code so far:
const { __ } = wp.i18n;
const { registerBlockType } = wp.blocks;
const { InspectorControls } = wp.editor;
const { registerPlugin } = wp.plugins
const { PluginDocumentSettingPanel } = wp.editPost
const { PanelBody, PanelRow, TextControl } = wp.components
const PluginDocumentSettingPanelDemo = () => (
<PluginDocumentSettingPanel
name="custom-panel"
title="Custom Panel"
className="custom-panel"
>
<TextControl
value={wp.data.select('core/editor').getEditedPostAttribute('meta')['_myprefix_text_metafield']}
label={ "Text Meta" }
onChange={(value) => wp.data.dispatch('core/editor').editPost({meta: {_myprefix_text_metafield: value}})}
/>
</PluginDocumentSettingPanel>
)
registerPlugin('plugin-document-setting-panel-demo', {
render: PluginDocumentSettingPanelDemo
})
Edit: Thanks to Ivan I solved this side issue :)
My Sidebar looks okay at first:
But when I try to change the inputs value it isn't updated (but the storage in wp.data is). I can't delete it, too. It stays at it's initial value. When I remove the part where I set the initial value it works like it should be but since I need the initial value this isn't an option for me ;)
Here an example log from the console when I add an "x" to the end of the input (as mentioned above the text in the input itself doesn't change)
Does anyone know how to make the input field working properly?
First of all, make sure you have https://wordpress.org/plugins/gutenberg/ plugin installed, because PluginDocumentSettingPanel is not fully implemented in core WP yet. It should be for 5.3 version, as per these tweets.
Second, you don't need the interval function for the wp.plugins. The reason it is undefined at first is that WordPress doesn't know that you need the wp-plugins loaded first. From the WordPress documentation
If you wanted to use the PlainText component from the editor module, first you would specify wp-editor as a dependency when you enqueue your script
The same applies for all other modules (read scripts, like 'wp-plugins').
You have to add the 'wp-plugins' script as a dependency, when registering your js plugin script:
<?php
/*
Plugin Name: Sidebar plugin
*/
function sidebar_plugin_register() {
wp_register_script(
'plugin-sidebar-js',
plugins_url( 'plugin-sidebar.js', __FILE__ ),
array( 'wp-plugins', 'wp-edit-post', 'wp-element' ) // <== the dependencies array is important!
);
}
add_action( 'init', 'sidebar_plugin_register' );
function sidebar_plugin_script_enqueue() {
wp_enqueue_script( 'plugin-sidebar-js' );
}
add_action( 'enqueue_block_editor_assets', 'sidebar_plugin_script_enqueue' );
The PHP above is taken from the official WP documentation.
I would also suggest reading thoroughly this awesome tutorial from Css Tricks. It goes in depth about setting up an ESNext environment with only the #wordpress/scripts package. It goes over the dependencies, adding meta fields and much more :) I hope this helps!
--------------- Initial answer ends here ---------------
Edit: After testing the code from the author, I found out a couple of issues. First of all, there was a missing closing tag for the TextControl. Second, I added Higher order components from the wp-data package, which I then used to "enhance" the TextControl, so that it doesn't manipulate or read data directly, but abstract that logic into it's higher order components. The code looks like so:
const { __ } = wp.i18n;
const { registerPlugin } = wp.plugins;
const { PluginDocumentSettingPanel } = wp.editPost;
const { TextControl } = wp.components;
const { withSelect, withDispatch, dispatch, select } = wp.data;
// All the necessary code is pulled from the wp global variable,
// so you don't have to install anything
// import { withSelect, withDispatch, dispatch, select } from "#wordpress/data";
// !!! You should install all the packages locally,
// so your editor could access the files so you could
// look up the functions and classes directly.
// It will not add to the final bundle if you
// run it through wp-scripts. If not, you can
// still use the wp global variable, like you have done so far.
let TextController = props => (
<TextControl
value={props.text_metafield}
label={__("Text Meta", "textdomain")}
onChange={(value) => props.onMetaFieldChange(value)}
/>
);
TextController = withSelect(
(select) => {
return {
text_metafield: select('core/editor').getEditedPostAttribute('meta')['_myprefix_text_metafield']
}
}
)(TextController);
TextController = withDispatch(
(dispatch) => {
return {
onMetaFieldChange: (value) => {
dispatch('core/editor').editPost({ meta: { _myprefix_text_metafield: value } })
}
}
}
)(TextController);
const PluginDocumentSettingPanelDemo = () => {
// Check if a value has been set
// This is for editing a post, because you don't want to override it everytime
if (!select('core/editor').getEditedPostAttribute('meta')['_myprefix_text_metafield']) {
// Set initial value
dispatch('core/editor').editPost({ meta: { _myprefix_text_metafield: 'Your custom value' } });
}
return (
<PluginDocumentSettingPanel
name="custom-panel"
title="Custom Panel"
className="custom-panel"
>
<TextController />
</PluginDocumentSettingPanel>
)
};
registerPlugin('plugin-document-setting-panel-demo', {
render: PluginDocumentSettingPanelDemo
})
Since the meta field is registered with an underscore as a first symbol in the name, WordPress will not allow you to save it, because it treats it as a private value, so you need to add extra code, when registering the field:
function myprefix_register_meta()
{
register_post_meta('post', '_myprefix_text_metafield', array(
'show_in_rest' => true,
'type' => 'string',
'single' => true,
'sanitize_callback' => 'sanitize_text_field',
'auth_callback' => function () {
return current_user_can('edit_posts');
}
));
}
add_action('init', 'myprefix_register_meta');
Again, all of this is explained in the Css tricks tutorial.
I had the same problem - values were not being updated in the field and in the database - and, after some research, I have found that the reason for this is that you should add 'custom-fields' to the 'supports' array in your call to register_post_type(), like this:
register_post_type(
'my_post_type_slug',
array(
...
'supports' => array( 'title', 'editor', 'custom-fields' ),
...
)
);
You can test that this works by calling wp.data.select( 'core/editor' ).getCurrentPost().meta in your JavaScript console, when the block editor is loaded. If your post type does not add support for 'custom-fields', this call will return undefined; if it does, it will return an empty array (or rather, an array with the already existing meta from the database). This behavior is mentioned in the Gutenberg docs, in a note on registering your post meta:
To make sure the field has been loaded, query the block editor internal data structures, also known as stores. Open your browser’s console, and execute this piece of code:
wp.data.select( 'core/editor' ).getCurrentPost().meta;
Before adding the register_post_meta function to the plugin, this code returns a void array, because WordPress hasn’t been told to load any meta field yet. After registering the field, the same code will return an object containing the registered meta field you registered.
I worked on a similar implementation recently, and worked through a bunch of examples as well. Between the above-mentioned articles, and this great series by one of the Automattic devs, I got a working version of the above example using the newer useSelect and useDispatch custom hooks. It's really quite similar, but utilizes custom hooks from React 16.8 for a slightly more concise dev experience:
(Also, using #wordpress/scripts, so the imports are from the npm packages instead of the wp object directly, but either would work.)
import { __ } from '#wordpress/i18n';
import { useSelect, useDispatch } from '#wordpress/data';
import { PluginDocumentSettingPanel } from '#wordpress/edit-post';
import { TextControl } from '#wordpress/components';
const TextController = (props) => {
const meta = useSelect(
(select) =>
select('core/editor').getEditedPostAttribute('meta')['_myprefix_text_metafield']
);
const { editPost } = useDispatch('core/editor');
return (
<TextControl
label={__("Text Meta", "textdomain")}
value={meta}
onChange={(value) => editPost({ meta: { _myprefix_text_metafield: value } })}
/>
);
};
const PluginDocumentSettingPanelDemo = () => {
return (
<PluginDocumentSettingPanel
name="custom-panel"
title="Custom Panel"
className="custom-panel"
>
<TextController />
</PluginDocumentSettingPanel>
);
};
export default PluginDocumentSettingPanelDemo;
Hopefully that helps someone else searching.
I am new to both React and Firebase. I struggled a bit to get data from the database, even though the instructions on the Firebase website were pretty straightforward.
I managed to print data in the view by using this code:
Get data from DB and save it in state:
INSTRUMENTS_DB.once('value').then(function(snapshot) {
this.state.instruments.push(snapshot.val());
this.setState({
instruments: this.state.instruments
});
From Firebase, I receive and Object containing several objects, which correspond to the differen instruments, like shown in the following snippet:
Object {
Object {
name: "Electric guitar",
image: "img/guitar.svg"
}
Object {
name: "Bass guitar",
image: "img/bass.svg"
}
// and so on..
}
Currently, I print data by populating an array like this:
var rows = [];
for (var obj in this.state.instruments[0]) {
rows.push(<Instrument name={this.state.instruments[0][obj].name}
image={this.state.instruments[0][obj].image}/>);
}
I feel like there's a better way to do it, can somedody give a hint? Thanks
I user firebase a lot and mu solution is little ES6 helper function
const toArray = function (firebaseObj) {
return Object.keys(firebaseObj).map((key)=> {
return Object.assign(firebaseObj[key], {key});
})
};
I also assign the firebase key to object key property, so later I can work with the keys.
The native map function only works for arrays, so using directly it on this object won't work.
What you can do instead is:
Call the map function on the keys of your object using Object.keys():
getInstrumentRows() {
const instruments = this.state.instruments;
Object.keys(instruments).map((key, index) => {
let instrument = instruments[key];
// You can now use instrument.name and instrument.image
return <Instrument name={instrument.name} image={instrument.image}/>
});
}
Alternatively, you can also import the lodash library and use its map method which would allow you to refactor the above code into:
getInstrumentRowsUsingLodash() {
const instruments = this.state.instruments;
_.map(instruments, (key, index) => {
let instrument = instruments[key];
// You can now use instrument.name and instrument.image
return <Instrument name={instrument.name} image={instrument.image}/>
});
}
Side note:
When you retrieve you data from Firebase you attempt to update the state directly with a call on this.state.instruments. The state in React should be treated as Immutable and should not be mutated with direct calls to it like push.
I would use map function:
_getInstrumentRows() {
const instruments = this.state.instruments[0];
if (instruments) {
return instruments.map((instrument) =>
<Instrument name={instrument.name}
image={instrument.image}/>);
}
}
In your render() method you just use {_getInstrumentRows()} wherever you need it.