I am trying to fetch full HTML page content of a single page web application (AngularJS) using puppeteer. But I got only HTML content of header and footer. This code working fine on my local machine but not working on AMI server.
//function for timeout
async function timeout(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
// function to crawl a html page using a url
async function crawler(url) {
const browser = await puppeteer.launch({
headless: true,
args: ['--no-sandbox', '--disable-setuid-sandbox']
});
const page = await browser.newPage();
await page.goto(url); //fetch url
await timeout(5000); // wait for 5 second to load full page
const html = await page.evaluate(() => document.documentElement.outerHTML);
await browser.close();
return html;
}
Also tried waitFor and networkidle2 but didn't get proper HTML in result.
Related
I am getting a data with an image link and I want to show those images by using the link in my img tag. I have disabled CORS on my chrome and I am receiving the data successfully but when I try to render the image it gives me an error "Cross-Origin Read Blocking (CORB) blocked cross-origin response https://commons.wikimedia.org/wiki/File:A_Conversation_With_Oscar_Wilde_-_London_-_240404.jpg with MIME type text/html. See https://www.chromestatus.com/feature/5629709824032768 for more details"
Can you please help on how can I bypass this. I am using axios for to make the fetch request and React to render the image.
async function getData(location) {
try {
// Make API call to get coordinates of location
const geonameResponse = await axios.get(`https://api.opentripmap.com/0.1/en/places/geoname?name=${location}&apikey=${API_KEY}`)
//console.log(geonameResponse)
// Make API call to get features within 1000 meter radius of location
const radiusResponse = await axios.get(
`https://api.opentripmap.com/0.1/en/places/radius?radius=1000&lon=${geonameResponse.data.lon}&lat=${geonameResponse.data.lat}&limit=10&apikey=${API_KEY}`)
//console.log(radiusResponse)
// Make API call for each feature to get more detailed information
// Make API call for each feature to get more detailed information
const xidResponses = await Promise.all(
radiusResponse.data.features.map(async (item) => {
return new Promise((resolve) => {
setTimeout(async () => {
resolve(await axios.get(
`https://api.opentripmap.com/0.1/en/places/xid/${item.properties.xid}?apikey=${API_KEY}`
));
}, 2000);
});
})
);
// Set data to array of xidResponses
setData(xidResponses);
} catch (error) {
console.error(error);
}
}
I'm getting an image from a user and storing it with in Cloud Storage through Firebase. The storing works fine.
Right after that I'm requesting all images from the storage to display them, but I got the response without the last submit. So I need to refresh the page, then useEffect make one more request and everything works.
UPD. Here's the the complete logic:
This function is uploading image to the storage:
const uploadImage = async (image,imageName) => {
if (image == null) return;
const imageRef = ref(storage,`images/${imageName}`);
try {
uploadBytes(imageRef, image);
} catch (err) {
alert(`${err}`)
}
}
This function does request to firestore, but doesn't return last uploaded image every time, just randomly. sometimes it does, usually not :
const getImages = async () => {
try {
const imagesListData = await listAll(imagesListRef);
setImagesList([]);
imagesListData.items.forEach(item => {
getDownloadURL(item).then(url => setImagesList(prev => [...prev,url]));
})
} catch(err){
alert(err.message);
}
}
after refreshing the page, useEffect does the job :
useEffect(() => {
getImages();
},[])
As I said above sometimes it works as I expected from the first try without me changing the code(which is the most confusing),most of the times I need to refresh the page to get the last image.
p.s. list() instead listAll() give same results
I'm building this website where users can sometimes upload one audio or one image and sometimes both at the same time to cloudinary. I'm able to upload from the front-end (react.js) one or the other (image or audio) but not both at the same time.
I saw this post that says that it is not possible except if we "write a script and use multi-threads (up to 10 threads) to upload many files at once. " I have no idea what it means in the context of react.js | JavaScript.
My code is the following:
I first call the handleUploadCloudinary with its parameter. The function is being called once the data is ready to be published.
const publishTheEntry = () => {
const {
mainText,
mediaAudio,
mediaImg
} = formData;
if(mediaAudio !== ""){
handleUploadCloudinary("audio");
};
if(mediaImg !== ""){
handleUploadCloudinary("image");
};
};
The handleUploadCloudinary() is the following:
const handleUploadCloudinary = (mediaType) => {
const {
mediaAudio,
mediaImg
} = formData;
const formDataCloudinary = new FormData();
formDataCloudinary.append("upload_preset", my_var_here);
formDataCloudinary.append("timestamp", (Date.now() / 1000) | 0);
if(mediaType === "img"){
formDataCloudinary.append("file", mediaImg);
axios.post(
"https://api.cloudinary.com/v1_1/sismographie-cloud/image/upload",
formDataCloudinary
).then((response) => {
console.log(response);
let url = response.data.secure_url;
setFormData({...formData, mediaImg: url});
})
}else if(mediaType === "audio"){
formDataCloudinary.append("file", mediaAudio);
axios.post(
"https://api.cloudinary.com/v1_1/sismographie-cloud/upload",
formDataCloudinary
).then((response) => {
let url = response.data.secure_url;
setFormData({...formData, mediaAudio: url});
})
}
};
Even if, for when both audio + image are stored into the state, I can console.log() both of the conditions, the state won't bot update the formData with both. One will successfully sate the state with a new cloudinary link while the other one will remain a buffer.
You can loop through your resources list and upload assets one by one, or create new threads at the backend (best practice).
This link is a demo for uploading multiple files using Axios and React:
https://codesandbox.io/s/chunked-client-side-upload-forked-vqx6mp?file=/src/CldCustUploadLgRestApi.js
I'm trying to fetch a .json file from local, and
I get response 200 but with a body response of HTML: "You need to enable JavaScript to run this app."
I have javascript enabled of course.
I don't want to import the file to simulate the real fetch.
How does local fetch work in react? How do I know if the fetch route is right? It doesn't give any useful error hint.
useEffect(() => {
const getData = async () => {
const dataFromLocal = await fetchData();
console.log('dataFromLocal', dataFromLocal);
}
getData();
}, [])
const fetchData = async () => {
const response = await fetch('data.json');
const data = await response.json();
return data;
}
I found how it works:
const response = await fetch('data.json',
{headers:
{'Content-Type': 'application/json','Accept': 'application/json'}
});
just add this headers object to the fetch method and it works
There are only two possibilities based on the code you've shown:
Your server is responding with the contents of an HTML resource (likely index.html) as the response to the request for data.json, or
data.json looks like this, the JSON string you provided:
"You need to enable JavaScript to run this app."
Is data.json in your project's ./public folder?
I have the below code in my ReactJS file that is to upLoad a file to Firebase.
async function uploadFile() {
console.log('starting UPLOAD ========');
const blob = await fetch(mediaBlobUrl).then((r) => r.blob());
const path = '/recordings/one';
firebase
.storage()
.ref(path)
.put(blob)
.then(function (snapshot) {
console.log('Uploaded complete');
});
console.log(`====> setURL is ${setURL} <=======`);
storage.ref(path).getDownloadURL().then(setURL);
}
The first time the button is clicked I get this error, but the the second time I click it then it works. Not sure what is going on?
Firebase Storage: Object 'recordings/one' does not exist. (storage/object-not-found)
I did notice when it fails this is the URL it is trying to hit (404). Notice the %2 instead of "/"
https://firebasestorage.googleapis.com/v0/b/buerce.appspot.com/o/recordings%2Fone 404
That's because your getDownloadURL method runs before actually uploading file. Both the put and getDownloadURL method returns promises. Your function is async so I'd recommend using await for both upload and getDownloadURL methods just like you are doing on fetch method.
async function uploadFile() {
console.log('starting UPLOAD ========');
const blob = await fetch(mediaBlobUrl).then((r) => r.blob());
const path = '/recordings/one';
await firebase.storage().ref(path).put(blob)
const setURL = await storage.ref(path).getDownloadURL()
console.log(`====> setURL is ${setURL} <=======`);
}