How to extract keypoints from BodyPix model in React? - reactjs

I need help on this matter. I am trying to extract the keypoints from BodyPix to be saved in a JSON file. I can see the keypoints in the console whenever I inspect it from the browser. I also want to save the keypoints into an array or a variable maybe so that I can use it throughout the code. Thank you very much. Here's the code:
function App() {
const bodyPixProperties = {
architecture: 'MobileNetV1',
outputStride: 16,
multiplier: 0.75,
quantBytes: 4
};
const runBodysegment = async () => {
const net = await bodyPix.load(bodyPixProperties);
console.log("BodyPix model loaded.");
// Loop and detect hands
setInterval(() => {
detect(net);
}, 100);
};
const detect = async (net) => {
// Check data is available
if (
...
const person = await net.segmentPersonParts(video, {
flipHorizontal: false,
internalResolution: 'medium',
segmentationThreshold: 0.7
});
console.log(person);
// const coloredPartImage = bodyPix.toMask(person);
const coloredPartImage = bodyPix.toColoredPartMask(person);
const opacity = 0.7;
const flipHorizontal = false;
const maskBlurAmount = 0;
const canvas = canvasRef.current;
bodyPix.drawMask(
canvas,
video,
coloredPartImage,
opacity,
maskBlurAmount,
flipHorizontal
);
}
};
runBodysegment();

Related

Tensorflow.js Model Prediction Output Never Changes

I have a custom trained tensorflow.js graph model (link can be found in the getModel method) to recognize images of fly agarics. However, the prediction output (result from model.executeAsync()) never changes - this is evident in the console.log in the renderPredictions method.
Greatly appreciate any help.
webcamView.tsx
import { useEffect, useRef, useState } from "react";
import * as tf from '#tensorflow/tfjs-core';
import { loadGraphModel } from "#tensorflow/tfjs-converter";
import '#tensorflow/tfjs-backend-cpu';
export default function WebcamView() {
const videoWidth = 640;
const videoHeight = 500;
const videoRef = useRef<HTMLVideoElement>(null);
const [loading, setLoading] = useState<Boolean>(true);
let model: any = undefined;
const getModel = async () => {
model = await loadGraphModel('https://raw.githubusercontent.com/AlanChen4/FungEye/main/data/web_model_old/model.json');
return model;
};
const startVideo = async () => {
navigator.mediaDevices.getUserMedia({
video: {
facingMode: 'environment',
}
}).then(stream => {
if (videoRef.current !== null) {
const video = videoRef.current;
video.srcObject = stream;
video.onloadedmetadata = () => {
video.play();
video.addEventListener('loadeddata', processVideoInput);
}
}
})
};
const renderPredictions = (predictions: any) => {
const predictionBoxes = predictions[6].dataSync();
const predictionClasses = predictions[2].dataSync();
const predictionScores = predictions[4].dataSync();
// this always prints the same scores
console.log(predictionScores);
};
const processVideoInput = () => {
// classify the frame in the video stream and then repeat to process the next frame
if (videoRef.current !== null) {
classifyVideoInput(videoRef.current).then(() => {
window.requestAnimationFrame(processVideoInput);
})
}
}
const classifyVideoInput = async (videoImage: HTMLVideoElement) => {
// get next video frame
await tf.nextFrame();
// convert tensor to image
const tfImage = tf.browser.fromPixels(videoImage);
// convert image to smaller image in order to match detection size
const smallerImage = tf.image.resizeBilinear(tfImage, [videoHeight, videoWidth]);
// convert smaller image to format usable by model
const resizedImage = tf.cast(smallerImage, 'int32');
let tf4d_ = tf.tensor4d(Array.from(resizedImage.dataSync()), [1, videoHeight, videoWidth, 3]);
const tf4d = tf.cast(tf4d_, 'int32');
// generate predictions from model
let predictions = await model.executeAsync(tf4d);
renderPredictions(predictions);
tfImage.dispose();
smallerImage.dispose();
resizedImage.dispose();
tf4d.dispose();
};
useEffect(() => {
const start = async () => {
await getModel();
await startVideo();
setLoading(false);
};
start();
}, []);
return (
<div className="p-8">
<div className="flex justify-center p-5">
<div id="videoView" style={{ cursor: 'pointer' }}>
{loading
? <p className="text-center text-semibold p-5">Loading Model...</p>
: <video ref={videoRef} playsInline autoPlay muted width={videoWidth} height={videoHeight}></video>
}
</div>
</div>
</div>
);
};
I have checked to make sure that video processing only occurs after the video has loaded. I've checked numerous threads and posts but can not seem to find the cause of this issue. I am using tensorflow.js 3.1.0.

React native custom hook return null in the first time using react-native-geolocation-services custom

I am trying to create a custom hook to get user's current location. I am using react-native-geolocation-services.
It returns null for the first time.
However, when I try to re-run the app. The geo data shows again.
Is this issue happening in asyn data?
Am I wrongly implemented the usestate so that the data didn't show in the first time?
Map component
import {useCurrentLocation} from '../queries/getCurrentLocation';
const Map = () => {
const {coordinate, watchError} = useCurrentLocation();
console.log('data',coordinate)
return <View style={styles.container}><MapView /></View>;
};
Custome Hook
import React, {useRef, useState, useEffect } from 'react';
import Geolocation, {watchPosition} from 'react-native-geolocation-service';
import useLocationPermission from '../hooks/useLocationPermission';
export const useCurrentLocation = () => {
const [coordinate, setCoordinate] = useState(null);
const [watchError, setWatchError] = useState(null);
const watchId = useRef(null);
const {hasPermission, hasPermissionError} = useLocationPermission();
const startWatch = () => {
if (!hasPermission) return;
watchId.current = Geolocation.watchPosition(
position => {
const latitude = position.coords.latitude;
const longitude = position.coords.longitude;
const speed = position.coords.speed;
setCoordinate({latitude, longitude, speed});
},
error => {
setWatchError(error);
},
{
accuracy: {
android: 'high',
//TODO config to ios
//ios: 'best',
},
enableHighAccuracy: true,
distanceFilter: 0,
interval: 20000,
fastestInterval: 2000,
},
);
};
const stopWatch = () => {
if (watchId.current == null) return;
Geolocation.clearWatch(watchId.current);
watchId.current = null;
};
useEffect(() => {
if (hasPermission) {
getCurrentCoordinate(coordinate);
}
startWatch();
return () => {
stopWatch();
};
}, [coordinate]);
return {coordinate, watchError};
};
const getCurrentCoordinate = coordinate => {
Geolocation.getCurrentPosition(position => {
coordinate = position;
});
return coordinate;
};

(Deck.GL - Trips Layer ) How can I set a right loop number for various route lengths?

I am trying to render a specific path on deck gl map, but every time, I fail to set the loop correctly.
Either loop is short and the trip stops in the middle of the road, or the loop is too long and the trip makes ugly straight line from the last coordinate to the first and starts the drawing again.
I need the trip to stop at the last coordinate and smoothly start again.
const [animation] = useState({});
const [time, setTime] = useState(0);
const [loopLength, setLoopLength] = useState(0);
const [startTime, setStartTime] = useState(Date.now());
// update loop length and render layers when routes change (when api returns a response)
useEffect(() => {
reset();
if (routes.length) {
const maxRoute = max(routes, ({ timestamps }) => max(timestamps));
setLoopLength(maxRoute);
// setLoopLength(10000);
setTime(0);
setStartTime(Date.now());
}
}, [routes]);
//
useEffect(() => {
if (routes.length) {
animate();
}
}, [startTime]);
// start animation
const animate = () => {
window.cancelAnimationFrame(animation.id);
const timestamp = (Date.now() - startTime) / 1000;
const loopTime = loopLength / SPEED; // how many units is whole loop
const time = ((timestamp % loopTime) / loopTime) * loopLength;
setTime(time);
animation.id = window.requestAnimationFrame(animate);
};
// method for rendering layers
const renderLayers = () => {
return [
new TripsLayer({
id: "trips",
data: routes,
getPath: (d) => d.routes,
getTimestamps: (d) => d.timestamps,
getColor: () => DEFAULT_THEME.trailColor0,
opacity: 1,
widthMinPixels: 1,
widthScale: 0.5,
rounded: true,
trailLength: 1000,
currentTime: time,
}),
];
};
Docs : https://deck.gl/docs/api-reference/geo-layers/trips-layer
Example: https://deck.gl/examples/trips-layer/

How to convert PDF to Image in ReactJS

I am developing a course platform using ReactJS. When the user finishes a course he can download the PDF file.
I need a version of the same file as an image (png or jpg), but I haven't found any way to do that. Can someone help me?
To generate the PDF certificate I'm using the lib: React-PDF.
This is my code to generate pdf file:
<PDFDownloadLink
document={
<Certificate course={course} name={name} date={today()} />
}
fileName="somename.pdf"
>
{({ blob, url, loading, error }) => {
return loading ? 'Loading document...' : 'Download now!';
}}
</PDFDownloadLink>
I created a helper function: convertPdfToImages which takes in the pdf file and returns an array of images encoded in base64, using the pdfjs package.
npm install pdfjs-dist -S
const PDFJS = require("pdfjs-dist/webpack");
const readFileData = (file) => {
return new Promise((resolve, reject) => {
const reader = new FileReader();
reader.onload = (e) => {
resolve(e.target.result);
};
reader.onerror = (err) => {
reject(err);
};
reader.readAsDataURL(file);
});
};
//param: file -> the input file (e.g. event.target.files[0])
//return: images -> an array of images encoded in base64
const convertPdfToImages = async (file) => {
const images = [];
const data = await readFileData(file);
const pdf = await PDFJS.getDocument(data).promise;
const canvas = document.createElement("canvas");
for (let i = 0; i < pdf.numPages; i++) {
const page = await pdf.getPage(i + 1);
const viewport = page.getViewport({ scale: 1 });
const context = canvas.getContext("2d");
canvas.height = viewport.height;
canvas.width = viewport.width;
await page.render({ canvasContext: context, viewport: viewport }).promise;
images.append(canvas.toDataURL());
}
canvas.remove();
return images;
}
Please use this library
https://www.npmjs.com/package/react-pdf-to-image
It is pretty straight forward. It will return the list of images (each page in the pdf as one image)
import React from 'react';
import {PDFtoIMG} from 'react-pdf-to-image';
import file from './pdf-sample.pdf';
const App = () =>
<div>
<PDFtoIMG file={file}>
{({pages}) => {
if (!pages.length) return 'Loading...';
return pages.map((page, index)=>
<img key={index} src={page}/>
);
}}
</PDFtoIMG>
</div>
export default App;
if you want to just download the each pdf page as image instead of component please follow below code
import PDFJS from 'pdfjs-dist/webpack';
this is the dependency library for react-pdf-to-image. Then read the pdf file(I'm giving base64 as input)
PDFJS.getDocument(blob).promise.then(pdf => {
const pages = [];
this.pdf = pdf;
for (let i = 0; i < this.pdf.numPages; i++) {
this.getPage(i + 1).then(result => {
// the result is the base 64 version of image
});
}
})
after reading each page, read each page as image from getPage method as below
getPage = (num) => {
return new Promise((resolve, reject) => {
this.pdf.getPage(num).then(page => {
const scale = "1.5";
const viewport = page.getViewport({
scale: scale
});
const canvas = document.createElement('canvas');
const canvasContext = canvas.getContext('2d');
canvas.height = viewport.height || viewport.viewBox[3]; /* viewport.height is NaN */
canvas.width = viewport.width || viewport.viewBox[2]; /* viewport.width is also NaN */
page.render({
canvasContext, viewport
}).promise.then((res) => {
resolve(canvas.toDataURL());
})
})
})
}

Can't access my state in my requestAnimationFrame function

I'm trying to build a space invader game from scratch using React/React-Hook & HTML5 canvas.
So far i achieved to draw my ship on the canvas but i can't figure out how to access my states in the "requestAnimationFrame" function. I did succeed to access REFs but i don't want all my vars to be refs.
So far my code looks like this :
import React from 'react';
import {makeStyles} from '#material-ui/core/styles';
const spaceInvaderStyles = makeStyles((theme) => ({
canvas: {
display: 'block',
margin: 'auto',
imageRendering: 'optimizeSpeed',
imageRendering: '-moz-crisp-edges',
imageRendering: '-webkit-optimize-contrast',
imageRendering: 'optimize-contrast',
backgroundColor: 'black',
}
}))
// GAME CONSTANTS
const GAME = {
shape: {
w:'640px',
h:'640px',
},
shipRow: '600',
shipColor: 'rgba(0,252,0)',
spritesSrc: 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAD4AAADuCAYAAABh/7RrAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAM4SURBVHhe7ZhRctswDAWrHCH3P6OPkNTuqB0FwxEAEqTkvt0feyYUJCRvX5xsvwr4frK/7WJ7sr9dxsf+KgeLqyG7eLNURstqNhVlSNTV0HX87j5HyXpP1NXA8QyeT97M0etbeDMtRF0NHH93cDwIi6uR8uIvXi94vo1eXwFRVwPHz8g6Oft8BURdDV3HPb9ezHAsw4xnJOpq6C5+tb8V9OzQtXSkbEZY8cPAcTVYXI1QicwuM48ZZUfU1cDxI1c77VHhPFFXQ9fxHp+tY6OdUDHPzvAg6mrg+LuD40FYXI2mF5731qfRnsjOs+d7IOpq6Doe8bPCqRFmPCNRV0PX8f31FM+x0Q6YPb8FUVeDxdXoKg2vjEaZUWYWoq4Gi6tR8sltNnxyK4TF1Wi6c7XTHhXOE3U1dB3v8dk6NtoJFfPsDA+irgaOvzs4HoTF1ej6zGt7IeuXpXpeBKKuBo4fsc5ZrIPeeY/sPHu+B6Kuhq7jET+rHcxe751/4d3TQtTV0F0868Yd6dmhZOlI+ZxxxTcfx9VgcTWapTJaVrOpKEOiroau43f3OUrWe6KuBo5n8HzyZo5e38KbaSHqauD4u4PjQVhcjZAX2R7wfKue1wNRVwPHj3gOjjo8+/oIRF0NHD9jtnMrnLYQdTVYXI2ucptRNkdW3I+oq8HiajT/y+qVSXX5ZOfZ8y+yz0DU1cDxI9aX1pmVRJ7HnvEg6mrgeIaIcxkq5tkZHkRdDRx/d3A8CIur0fTi7t5nfW5B1NXA8TOs8xWOnbHifkRdDRw/stppjxnPQ9TVYHE1QiWxuuxW3I+oq8HianSVRnX5ZOc9j3/ub//xvOSxvw1B1NXA8TOsg6sZ7ZAWRF0NHD/iOW2dG+2A7Dx7vgeiroau46N+3oWs90RdDRw/Yn25ugciz2PPeBB1NXD8iOdU1qcs3v3s119kn4moq6Hr+P6aouXYEc+30esrIOpqsLga00ukxcf3549y+9oey5+DqKvB4mosKRVbZh4ryo6oq8HiarC4GrKLN39fZn/v3p3W5wKiroau4/+bz1GIuhosrgaLq8HiarC4GiyuBourIbv4n/9Fqf1N/rU9tt/3UY/ZXED3UQAAAABJRU5ErkJggg==',
sprites: {
ship: {
x: 0,
y: 204,
w: 61,
h: 237,
}
},
player: {
initialPos: {
x: 290,
y: 580,
}
}
}
const KEYS = {
left:81,
right:68,
down: 83,
up: 90,
arrowLeft: 37,
arrowRight: 39,
arrowDown: 40,
arrowUp: 38,
}
const SpaceInvader = (props) => {
const classes = spaceInvaderStyles();
const canvasRef = React.useRef();
const [cctx, setCctx] = React.useState(null);
const [sprites, setSprites] = React.useState(null);
const [player, setPlayer] = React.useState({
pos: GAME.player.initialPos.x,
})
// keys
const [currentKey, setCurrentKey] = React.useState(null);
//time
let lastTime = 0;
const [counter, setCounter] = React.useState(0);
React.useEffect(() => {
// INIT
// context
const canvas = canvasRef.current;
const context = canvas.getContext('2d');
setCctx(context)
// sprites
loadSprites();
}, [])
// key handler
React.useEffect(() => {
window.addEventListener('keydown', (event) => handleUserKeyPress(event,true));
window.addEventListener('keyup', (event) => handleUserKeyPress(event,false))
return () => {
window.removeEventListener('keydown', (event) => handleUserKeyPress(event,true));
window.removeEventListener('keyup', (event) => handleUserKeyPress(event,false))
};
}, [cctx, sprites, player, currentKey])
React.useEffect(() => {
if(!cctx) return
animate();
}, [cctx])
React.useEffect(() => {
if(spritesAreLoaded()){
cctx.drawImage(sprites, GAME.sprites.ship.x, GAME.sprites.ship.y, GAME.sprites.ship.w, GAME.sprites.ship.h, GAME.player.initialPos.x, GAME.player.initialPos.y , GAME.sprites.ship.w, GAME.sprites.ship.h)
}
}, [sprites])
React.useEffect(() => {
console.log(counter)
}, [counter])
// utils
const clearCanvas = () => {
cctx.clearRect(0,0, 640, 640);
}
const saveCanvas = () => {
cctx.save();
}
const drawImage = (image, sx, sy, sw, dx, dy, sh, dw, dh) => {
cctx.drawImage(image, sx, sy, sw, dx, dy, sh, dw, dh);
}
const restore = () => {
cctx.restore();
}
const loadSprites = () => {
var spritesImg = new Image();
spritesImg.src = GAME.spritesSrc;
spritesImg.onload = function() {
// sprites are loaded at this point
setSprites(spritesImg);
}
}
const spritesAreLoaded = () => {
return sprites !== null;
}
const move = (direction) => {
// cctx, sprites and all others state vars are at default value here too
clearCanvas();
saveCanvas();
drawImage(sprites, GAME.sprites.ship.x, GAME.sprites.ship.y, GAME.sprites.ship.w, GAME.sprites.ship.h, player.pos + (10 * direction), GAME.player.initialPos.y , GAME.sprites.ship.w, GAME.sprites.ship.h);
restore();
setPlayer({...player, pos: player.pos + (10 * direction)});
}
const handleUserKeyPress = React.useCallback( (event, isDown) => {
event.preventDefault();
const {key, keyCode} = event;
setCurrentKey(isDown ? keyCode : null);
}, [cctx, sprites, player, currentKey])
const updatePlayer = () => {
// currentKey is at default value here...
const direction = currentKey === KEYS.left ? -1 : currentKey === KEYS.right ? 1 : null;
if(direction !== null) move(direction)
}
const animate = (time) => {
var now = window.performance.now();
var dt = now - lastTime;
if(dt > 100) {
lastTime = now;
updatePlayer();
};
requestAnimationFrame(animate);
}
return (
<canvas
className={classes.canvas}
ref={canvasRef}
width={GAME.shape.w}
height={GAME.shape.h}
/>
)
}
export default SpaceInvader;
I'm trying to access "currentKey" in the thread function but it always return "null" (the default state value),
I found on some topic that you need to bind a context to the animate function but i don't know how to do it with a functional component (with a class component i would do a .bind(this))
I'm pretty new at HTML5 canvas so I might not be able to see the problem here.
All tips are appreciated,
Thanks in advance !
Finally after a lot of testing I did that :
React.useEffect(() => {
console.log("use Effect");
if(!cctx) return
requestRef.current = requestAnimationFrame(animate);
return () => cancelAnimationFrame(requestRef.current);
})
and it works...
If anyone have a cleaner solution
EDIT 1 : After some more testing, it might not be a viable solution, it causes too many re-render

Resources