Cheerio not working as expected in a Google Cloud function - request

I have this piece of code :
const response = await request.get(
`https://www.youtube.com/watch?v=yURRmWtbTbo&sp=EgIQAQ%253D%253D`
);
const $ = cheerio.load(response, {
decodeEntities: false
});
console.log(
$(
".video-list-item.related-list-item.show-video-time.related-list-item-compact-video"
).html()
);
It works great on my local machine but if I execute this code in a cloud function, it returns null. I checked this is the exact same code, packages also have the same version. I'm not sure what's going on here. I also tried with axios...
What's even stranger is that it works well on both machines with a different youtube ID: I_izvAbhExY
EDIT:
Here is a diff check of the response from my local machine (to the left) and from the cloud function machine (to the right) :
https://www.diffchecker.com/dKaN5kBx

I don't see any class with "video-list-item" in the code on the right. Maybe that's the problem?
Some parts of the webpage are missing in the source code on the right. So the selector
$(
".video-list-item.related-list-item.show-video-time.related-list-item-compact-video"
)
will return null or undefined

Related

getInitialProps causing ERR_TOO_MANY_REDIRECTS error on my live site, but not on my local version

So, I've been trying to implement cookies on my website, to keep track of a list of JavaScript objects, so the page stays consistent when the user comes back to it. I've been following this tutorial here.
On my local machine, using npm run dev on localhost:3000, it works absolutely perfect. However, when I push the commit to GitHub, it builds on Vercel without any issue, but when I try and access the live website on the internet, it gives me a 'ERR_TOO_MANY_REDIRECTS' error.
I'm pretty confused as to why it would work perfectly fine on my locally hosted site, but freaks out and does not work when it's put into production.
I think I have narrowed the problem down to getInitialProps because when I comment out the implementation in my index.js file, it still doesn't work, but when I comment out getInitialProps, it works again.
Here is the code I think may be the problem.
Home.getInitialProps = async ({req, res}) => {
const data = parseCookies(req)
if (res) {
if (Object.keys(data).length === 0 && data.constructor === Object) {
res.writeHead(301, { Location: "/" })
res.end()
}
}
return {
data: data && data,
}
}
And here is the code for that parseCookies method, which is imported as
import { parseCookies } from "../helpers/index"
within my index.js
import cookie from "cookie"
export function parseCookies(req) {
return cookie.parse(req ? req.headers.cookie || "" : document.cookie)
}
I'm super confused at this point, I've walked myself through the code a dozen times now and still have no idea what I might be doing wrong. Any help would be much appreciated! And please lemme know if there's anymore info I can provide!
The ERR_TOO_MANY_REDIRECTS error occurs because Object.keys(data).length === 0 && data.constructor === Object returns true when no cookies are set and you access the homepage. When this happens the redirect takes you back to / (the homepage) which then makes the check again and a new redirect occurs, and so on.
Locally, you probably have cookies set, so you don't experience the issue. However, when you access the website hosted on Vercel, no cookies are present initially, which triggers the infinite redirect cycle.
To fix the issue simply remove the logic from the homepage, since that's the redirect destination. You can still have it on other pages and redirect to the homepage, though.

SAD PANDA: TypeError: failed to fetch

​ === SAD PANDA ===
TypeError: Failed to fetch
=== SAD PANDA ===
While executing a flow cadence transaction in react.js, I got the above error.
My intention is when I click the minttoken button, this transaction has to execute so as to mint the NFT.
const mintToken = async() => {
console.log(form.name)
const encoded = await fcl.send([
fcl.proposer(fcl.currentUser().authorization),
fcl.payer(fcl.authz),
fcl.authorizations([fcl.authz]),
fcl.limit(50),
fcl.args([
fcl.arg(form.name,t.String),
fcl.arg(form.velocity,t.String),
fcl.arg(form.angle,t.String),
fcl.arg(form.rating,t.String),
fcl.arg(form.uri,t.String)
]),
fcl.transaction`
import commitContract from 0xf8d6e0586b0a20c7
transaction {
let receiverRef: &{commitContract.NFTReceiver}
let minterRef: &commitContract.NFTMinter
prepare(acct: AuthAccount) {
self.receiverRef = acct.getCapability<&{commitContract.NFTReceiver}>(/public/NFTReceiver)
.borrow()
?? panic("Could not borrow receiver reference")
self.minterRef = acct.borrow<&commitContract.NFTMinter>(from: /storage/NFTMinter)
?? panic("could not borrow minter reference")
}
execute {
let metadata : {String : String} = {
"name": name,
"swing_velocity": velocity,
"swing_angle": angle,
"rating": rating,
"uri": uri
}
let newNFT <- self.minterRef.mintNFT()
self.receiverRef.deposit(token: <-newNFT, metadata: metadata)
log("NFT Minted and deposited to Account 2's Collection")
}
}
`
]);
await fcl.decode(encoded);
}
this error being so useless is my fault, but I can explain what is happening here because it also only happens in a really specific situation.
Sad Panda error is a catch all error that happens when there is a catastrophic failure when fcl tries to resolve the signatures and it fails in a completely unexpected way. At the time of writing this it usually shows up when people are writing their own authorization functions so that was the first thing i looked at in your code example. Since you are using fcl.authz and fcl.currentUser().authorization (both of those are the same by the way) your situation here isnt because of a custom authorization function, which leads me to believe this is either a configuration issue (fcl.authz is having a hard time doing its job correctly) or what fcl is getting back from the wallet doesn't line up with what it is expecting internally (most likely because of an out of date version of fcl).
I have also seen this come up when the version of the sdk that fcl uses doesnt line up with the version of the sdk that is there (because some people have added #onflow/sdk as well as #onflow/fcl) so would also maybe check to make sure you only have fcl in your package.json and not the sdk as well (everything you should need from the sdk should be exposed from fcl directly, meaning you shouldnt need the sdk as a direct dependency of your application)
I would first recommend making sure you are using the latest version of fcl (your code should still all work), then i would make sure you are only using fcl and not inadvertently using an older version of the sdk. If you are still getting the same error after that could you create an issue on the github so we can dedicate some resources to helping sort this out (and make it so you and others dont see this cryptic error in future versions of fcl)

Loading PIXI textures, handling failures

I'm working on a map project under React, using react-leaflet, and leaflet-pixi-overlay. Markers are implemented using the PIXI overlay (React 16.13.1, pixi.js 5.3.0, leaflet 1.6.0, leaflet-pixi-overlay 1.8.1).
I am struggling a bit with the PIXI documentation. I would like to use this PIXI.Texture.fromURL method (http://pixijs.download/release/docs/PIXI.Texture.html#.fromURL)
However neither my VS Code environment, nor my compiled source can access this method.
I am using instead PIXI.Texture.from(imageUrl), as well as PIXI.Texture.fromLoader(imageUrl). Both seem to work, but I don't get the difference between the two? The docs don't show these as being promises, yet they seem to work well with an async await?
Then, when a load fails, I don't see how to tell that things went wrong. Actually what I don't see is how to tell that things went right!
If I do:
let failed = false;
let newTexture;
try {
newTexture = await PIXI.Texture.from(url);
} catch (err) {
console.log(`FAILED loading texture from ${url}, err=${err}`);
failed = true;
}
console.log(`valid=${texture.valid}`);
Then:
texture.valid is always false, even when the texture loaded and displays just fine
no error is ever thrown when the url points to nowhere
Any pointers, is there a site with good (recent as of 2020) PIXI references? Am I missing something basic? Thanks.
Edit 07/06/2020:
Issues were largely due to both my IDE and webpack not 'seeing' that I had update pixi.js to 5.3.0, restart of both gave me access to Texture.fromURL.
The Texture.from call is a synchronous one. My understanding is that by default, it will load a 'valid' texture of 1x1 px in case of failure. Texture.fromURL was added to provide an async solution, see https://github.com/pixijs/pixi.js/issues/6514
Looks to me like Texture.fromURL still needs a bit of work, as it doesn't seem to ever return on a failed fetch (at least for relative paths, which is what I use). I see the same thing when using the following approach:
const texture = PIXI.Texture.from(url, {
resourceOptions: { autoLoad: false }}
);
await texture.baseTexture.resource.load();
With a bad relative path, the load function never returns in my test environment.
Then, when a load fails, I don't see how to tell that things went wrong. Actually what I don't see is how to tell that things went right! If I do:
...
Then:
texture.valid is always false, even when the texture loaded and displays just fine
no error is ever thrown when the url points to nowhere
Ok, first thing: please use the newest version of PIXI which is now 5.3.0: https://github.com/pixijs/pixi.js/releases/tag/v5.3.0
Texture.fromUrl was added in this PR: https://github.com/pixijs/pixi.js/pull/6687/files . Please read description of this PR: https://github.com/pixijs/pixi.js/pull/6687 - user bigtimebuddy describes 3 ways to load Texture synchronously. From this you should understand why it didnt worked in your code.
Also see: https://gamedev.stackexchange.com/questions/175313/determine-when-a-pixi-texture-is-loaded-to-clone-it
About error handling, catching errors and checking if Texture is "valid": please try running following example (modified version of yours) :
let failed = false;
let newTexture;
try {
newTexture = await PIXI.Texture.fromURL('https://loremflickr.com/100/100');
// to see how failure works comment above line and uncomment line below:
// newTexture = await PIXI.Texture.fromURL('http://not-existing-site-0986756.com/not_existing.jpg');
} catch (err) {
console.log(`FAILED loading texture`);
console.log(err);
failed = true;
}
console.log('failed: ' + (failed ? 'yes' : 'no'));
console.log(`valid=${typeof newTexture !== 'undefined' ? newTexture.valid : 'variable newTexture is undefined'}`);
console.log(newTexture ? newTexture : 'n/a');
And lastly about method not found in IDE:
I would like to use this PIXI.Texture.fromURL method (http://pixijs.download/release/docs/PIXI.Texture.html#.fromURL)
However neither my VS Code environment, nor my compiled source can access this method.
I use PhpStorm (but other IntelliJ editor should be similar - for example: WebStorm) and it finds this method:
/**
* Useful for loading textures via URLs. Use instead of `Texture.from` because
* it does a better job of handling failed URLs more effectively. This also ignores
* `PIXI.settings.STRICT_TEXTURE_CACHE`. Works for Videos, SVGs, Images.
* #param {string} url The remote URL to load.
* #param {object} [options] Optional options to include
* #return {Promise<PIXI.Texture>} A Promise that resolves to a Texture.
*/
Texture.fromURL = function (url, options) {
var resourceOptions = Object.assign({ autoLoad: false }, options === null || options === void 0 ? void 0 : options.resourceOptions);
var texture = Texture.from(url, Object.assign({ resourceOptions: resourceOptions }, options), false);
var resource = texture.baseTexture.resource;
// The texture was already loaded
if (texture.baseTexture.valid) {
return Promise.resolve(texture);
}
// Manually load the texture, this should allow users to handle load errors
return resource.load().then(function () { return Promise.resolve(texture); });
};
Do you use the development build of Pixi, or production one? ( https://github.com/pixijs/pixi.js/releases ).
Update 2020-07-06:
Your comment:
One thing still not clear to me though: when using an approach based on PIXI.Loader, do the sprites using a given texture get automatically refreshed once the texture has been loaded, or is there a manual refresh process required?
If you use "PIXI.Loader" approach then you can set the "load" callback - in which you should have all resources / textures already loaded. See: https://pixijs.download/dev/docs/PIXI.Loader.html
First you define which resources need to be loaded:
// Chainable `add` to enqueue a resource
loader.add('bunny', 'data/bunny.png')
.add('spaceship', 'assets/spritesheet.json');
loader.add('scoreFont', 'assets/score.fnt');
and then you define the callback:
// The `load` method loads the queue of resources, and calls the passed in callback called once all
// resources have loaded.
loader.load((loader, resources) => {
// resources is an object where the key is the name of the resource loaded and the value is the resource object.
// They have a couple default properties:
// - `url`: The URL that the resource was loaded from
// - `error`: The error that happened when trying to load (if any)
// - `data`: The raw data that was loaded
// also may contain other properties based on the middleware that runs.
sprites.bunny = new PIXI.TilingSprite(resources.bunny.texture);
sprites.spaceship = new PIXI.TilingSprite(resources.spaceship.texture);
sprites.scoreFont = new PIXI.TilingSprite(resources.scoreFont.texture);
});
You can try this way and inside this callback you can observe that texture of each resource is valid - for example: resources.bunny.texture.valid - it should be true.
Also, as you see in that doc, you can use other more advanced features like middleware or other callbacks for error handling etc.

Cortana ran into an issue

I have created a javascript application (aka UWA) in order to play with my Belkin wemo and then turn on or turn off the ligth with Cortana. The following function is well called but Cortana ends up with an issue. If I remove the call to the HTTP call, the program works fine. Who can tell me what's wrong with the following function because no more details are exposed unfortunately (of course in the real program is replaced with the right URL):
function setWemo(status) {
WinJS.xhr({ url: "<url>" }).then(function () {
var userMessage = new voiceCommands.VoiceCommandUserMessage();
userMessage.spokenMessage = "Light is now turned " + status;
var statusContentTiles = [];
var statusTile = new voiceCommands.VoiceCommandContentTile();
statusTile.contentTileType = voiceCommands.VoiceCommandContentTileType.titleOnly;
statusTile.title = "Light is set to: " + status;
statusContentTiles.push(statusTile);
var response = voiceCommands.VoiceCommandResponse.createResponse(userMessage, statusContentTiles);
return voiceServiceConnection.reportSuccessAsync(response);
}).done();
}
Make sure that your background task has access to the WinJS namespace. For background tasks, since there isn't any default.html, base.js won't be getting imported automatically unless you explicitly do it.
I had to update winjs to version 4.2 from here (or the source repository on git), then add that to my project to update from the released version that comes with VS 2015. WinJS 4.0 has a bug where it complains about gamepad controls if you try to import it this way (see this MSDN forum post)
Then I added a line like
importScripts("/Microsoft.WinJS.4.0/js/base.js");
to the top of your script's starting code to import WinJS. Without this, you're probably getting an error like "WinJS is undefined" popping up in your debug console, but for some reason, whenever I hit that, I wasn't getting a debug break in visual studio. This was causing the Cortana session to just hang doing nothing, never sending a final response.
I'd also add that you should be handling errors and handling progress, so that you can periodically send progress reports to Cortana to ensure that it does not time you out (which is why it gives you the error, probably after around 5 seconds):
WinJS.xhr({ url: "http://urlhere/", responseType: "text" }).done(function completed(webResponse) {
... handle response here
},
function error(errorResponse) {
... error handling
},
function progress(requestProgress) {
... <some kind of check to see if it's been longer than a second or two here since the last progress report>
var userProgressMessage = new voiceCommands.VoiceCommandUserMessage();
userProgressMessage.DisplayMessage = "Still working on it!";
userProgressMessage.SpokenMessage = "Still working on it";
var response = voiceCommands.VoiceCommandResponse.createResponse(userProgressMessage);
return voiceServiceConnection.reportProgressAsync(response);
});

Weird behavior with Restlet and GAE

I have the following piece of code with Restlet in Google AppEngine from an Android client.
ClientResource clientResource = new ClientResource(RESTLET_TEST_URL);
ProductResource productResource = clientResource.wrap(ProductResource.class);
productResource.store(mProduct);
Status status = clientResource.getResponse().getStatus();
Toast.makeText(this, "Status: "+ status.getDescription(), Toast.LENGTH_SHORT).show();
clientResource.release();
The .store() method is analogous to a PUT request. The weird thing is, this works fine when I connect to the development server but on the actual AppEngine site, nothing happens. I just get Status: OK indicating that the request went through.
I can't troubleshoot cause I can only do that in the Dev Server and that is working fine.
Any ideas on what the problem may be or how to approach this ?
For reference, the code at the server end is :
if (product != null ) {
if (new DataStore().putToDataStore(product) ) {
log.warning("Product written to datastore");
} else {
log.warning("Product not found in datastore");
}
}
This is just a simple write to the datastore using Objectify.
Turns out this is a known issue. See here
The solution is to use clientResource.setEntityBuffering(true);. However, please note that this method is only available in the Release Candidate for Android Client and not in the stable release.

Resources