404 after 43 seconds TTFB - http-status-code-404

I have script which uses simple_html_dom to parse different site data. It looks through my table of users, grabs the various sites needed, and then parses the data and stores them into my db.
The problem is that when I iterate through more than 3 users I get a 404 error. After a lot of debugging (much of which I'm learning as I go) it looks like as soon as my TTFB hits 40 seconds I get a 404 not found error. Anything under that the page returns fine.
I included the following in my php file to extend the time but this problem seems to ignore these statements.
// It may take a whils to crawl a site ...
ini_set("memory_limit", "-1");
ini_set('max_execution_time', 300); //300 seconds = 5 minutes
ini_set('max_input_time', -1); //300 seconds = 5 minutes
set_time_limit(0);
But I've never had this problem before where I get a 404 for a page that exists. I'm somewhat new to simple_html_dom and crawling through different pages but is the problem that the wait time is too long? If so how do I can I fix that? Thanks

So it did not have to do with execution time or any setting I could change with the php script. For anyone having the same issue this was fixed by changing the way simple_html_dom loads the script from:
$html = new simple_html_dom();
$html->load_file($url_link);
To:
$html = #file_get_contents($url_link);
$html = str_get_html($html);
Hope this helps someone else!

Related

Offline HLS Fairplay playback error when the app is closed, code 16227

I'm implementing Offline Playback with HLS Fairplay following the demo in the FairPlay Streaming Server SDK v4.0.1 that uses AVContentSessionKey.
I download three contents, each content is downloaded and persisted correctly, both the .movpkg and its content key on the documents directory, when I turn off the WIFI these three contents downloaded plays correctly without any problems, before playing Im using this code:
let urlAsset = element.urlAsset!
ContentKeyManager.shared.contentKeySession.addContentKeyRecipient(urlAsset)
if !urlAsset.resourceLoader.preloadsEligibleContentKeys {
urlAsset.resourceLoader.preloadsEligibleContentKeys = true
}
self.present(playerViewController, animated: true, completion: {
AssetPlaybackManager.sharedManager.setAssetForPlayback(urlAsset)
})
So far so good. But the problem is when I close the application (Home button to close applications) and then play the downloaded contents, only the last content downloaded plays correctly, the other ones (first and second) send these error on the console.
Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed"
UserInfo={NSUnderlyingError=0x1c065d760 {Error Domain=NSOSStatusErrorDomain Code=-16227 "(null)"},
NSLocalizedFailureReason=An unknown error occurred (-16227),
NSURL=file:///private/var/mobile/Containers/Data/Application/A950D8DB-B711-47E3-AAF5-C95CC9682430/Library/com.apple.UserManagedAssets.kkG8Ih/644986_7798B8476A473F68.movpkg/, NSLocalizedDescription=The operation could not be completed}
I double check the .movpkg with the keys in the documents directory and appears correctly
/Documents/.keys/one-key
/Documents/.keys/two-key
/Documents/.keys/three-key
Before the error occurs the ContentKeyDelegate is called and the key is loaded and passed to the request correctly.
if persistableContentKeyExistsOnDisk(withContentKeyIdentifier: assetIDString) {
let urlToPersistableKey = urlForPersistableContentKey(withContentKeyIdentifier: assetIDString)
guard let contentKey = FileManager.default.contents(atPath: urlToPersistableKey.path) else {
/
pendingPersistableContentKeyIdentifiers.remove(assetIDString)
return
}
/
Create an AVContentKeyResponse from the persistent key data to use for requesting a key for
decrypting content.
*/
let keyResponse = AVContentKeyResponse(fairPlayStreamingKeyResponseData: contentKey)
/
keyRequest.processContentKeyResponse(keyResponse)
return
}
If I print the contentKeyRecipients the three contents appears correctly
- (lldb) po
ContentKeyManager.shared.contentKeySession.contentKeyRecipients ▿ 3
elements
- 0 : AVURLAsset: 0x1c0234d40, URL = file:///private/var/mobile/Containers/Data/Application/E791A4DE-4261-46B7-A84D-D10B27035FAE/Library/com.apple.UserManagedAssets.kkG8Ih/539628_20469336224AA388.movpkg
- 1 : AVURLAsset: 0x1c0234fa0, URL = file:///private/var/mobile/Containers/Data/Application/E791A4DE-4261-46B7-A84D-D10B27035FAE/Library/com.apple.UserManagedAssets.kkG8Ih/644986_7798B8476A473F68.movpkg
- 2 : AVURLAsset: 0x1c42391c0, URL = file:///private/var/mobile/Containers/Data/Application/E791A4DE-4261-46B7-A84D-D10B27035FAE/Library/com.apple.UserManagedAssets.kkG8Ih/573744_62377F9549C45B93.movpkg
My tests are in iOS 11.1.2 and iOS 11.2 beta 2
I'm not sure what is happening, but seems to be a problem with the persisted key, I don't how if each content needs to be associated with one AVContentKeySession at time.
If someone faced a similar problem, any help would be appreciated.
Thanks in advance
I'm having similar issue.
however, since I need to support iOS 10, I'm not using the new AVContentKeyResponse class. Instead, I'm loading the persistent content key myself, and pass it to the loading request.
Anyway, I'm getting exact the same error as you and same behavior. One thing to note is that if I remove the code that loads persistent content key from disk, and always fetch the key from server, then everything works. But this defeats the purpose of "offline" playback...
So it seems like the system thinks the persistent content key is invalid...
Which TLLV you used on the server side to specify Rental Duration of the downloaded content? Did you use Content key duration TLLV or Offline Key TLLV? If you used Offline Key TLLV you need to double check that "Content ID" field is different for every downloaded movie.
We had encounter this error message, too.
It will happen when content exceed over expiration date which set in server side.
For example :
We give 10 minutes of expiration date for Video A
Download this Video A, and verify CKC delivery correctly (print log)
Play Video A without connection
Take a break (after 11 minutes), close App, and launch App again, select Video A to Play
Show below error message from AVPlayerItem.error.description:
Error Domain=AVFoundationErrorDomain Code=-11800
"The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-16227),
NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x1d4257310
{Error Domain=NSOSStatusErrorDomain Code=-16227 "(null)"}}
You can refresh encrypted data again by
AVAssetResourceLoaderDelegate
or use AVContentSessionKey
Reference : https://developer.apple.com/videos/play/wwdc2018/507/
Make sure you set correct offline content identifier on a serverside. The identifier you set should be associated with the specific rendition/stream allowed by the license. This helped me.

JSON post, browser and application different times

Before i start eating my keyboard, maybe someone can help me out here...
Im using Angular Bootstrap to set a time. After posting the time, and checking my console, i see that the selected time is posted. But after checking my log file, i see that the time is posted minus 1 hour. I've set my app => timezone to Europe/Amsterdam (ECT if im right) but still cant find what is going wrong.
Can someone help me please?
Controller:
public function update( EventRequest $request, $eventGuid )
{
Log::info($request->all());
$this->event->updateById($request, $eventGuid);
}
Angular post will be done as: 2016-01-27 03:45:39
Console log: 2016-01-27 03:45:39
Laravel log: 2016-01-27 02:45:39

Spontaneous Server Errors During AngularJS $http calls

I'm building an SPA in AngularJS served by a Laravel (5.1) backend. Of late I've been encountering an annoying error, a server 500 or code 0 error which is abit hard to explain how it comes but let me try to may be someone will understand the dental formula of my problem.
When i start my AngularJS controller, I make several server calls (via independent $http calls from services) to retrieve information i might later need in the controller. For example,
Functions.getGrades()
.then(function(response)
{
$scope.grades = response.data;
});
Subjects.offered()
.then(function(response)
{
$scope.subjects = response.data;
});
Later on i pass these variables (grades or subjects) to a service where they are used for processing. However, these functions are randomly returning code 500 server errors after they run, and sometimes returning status code 0 after running. This happens in a random way and it is hard for me to point out the circumstances leading to their popping up. This leaves me with frequent empty Laravel-ised error screens like the ones shown below.
Anyone reading my mind?
Ok, after a suggestion given in a comment above that I check my Laravel log files (located in storage/logs/laravel.log- Laravel 5.1), i found out that the main error most of these times was this one: 'PDOException' with message 'SQLSTATE[HY000] [1044] Access denied for user ''#'localhost' to database 'forge'' in ..., plus another one that paraphrased something like No valid encrypter found. These were the key opener.
On reading another SO thread here, it said in part:
I solved, sometimes laravel not read APP_KEY in .ENV. And returns a value "SomeRandomString" (default is defined in config / app.php), and have the error "key length is invalid", so the solution is to copy the value of APP_KEY, to the value 'key 'in config / app.php, that's all! I solved!
That was exactly the issue! When loading the DB params from the .env to config/database.php, Laravel was sometimes unable to read the environment variables and went for the fallback default fallback options (forge for DB name and username and SomeRandomString for the APP_KEY). So, to solve this i just did as advised: copied the APP_KEY in .env to the config/app.php and edited the default DB parameters to the actual DB name and username/password I'm using. Just that and i was free from pollution. Hope someone finds this helpful.

Weird (intermittent) Can't set headers after they are sent error

Have a very strange intermittent problem, and one that I just can not understand at all. Not certain it is code related, bug in express or just me missing something.
Have an app, all in MEAN, that like so many other apps around does a bunch of API calls. Some even in parallel.
It all worked perfect this morning, actually, read the "where it gets weird", but then all of a sudden it stops working and the server starts falling over with the below error:
GET /api/skillList 304 45.345 ms - -
_http_outgoing.js:335
throw new Error('Can\'t set headers after they are sent.');
^
Error: Can't set headers after they are sent.
at ServerResponse.OutgoingMessage.setHeader (_http_outgoing.js:335:11)
at ServerResponse.header (/Users/bengtbjorkberg/WebstormProjects/ResourceEdge/node_modules/express/lib/response.js:695:10)
at ServerResponse.json (/Users/bengtbjorkberg/WebstormProjects/ResourceEdge/node_modules/express/lib/response.js:232:10)
at /Users/bengtbjorkberg/WebstormProjects/ResourceEdge/routes/api.js:78:9
at /Users/bengtbjorkberg/WebstormProjects/ResourceEdge/node_modules/mongoose/node_modules/kareem/index.js:160:11
at Query._findOne (/Users/bengtbjorkberg/WebstormProjects/ResourceEdge/node_modules/mongoose/lib/query.js:1145:12)
at /Users/bengtbjorkberg/WebstormProjects/ResourceEdge/node_modules/mongoose/node_modules/kareem/index.js:156:8
at /Users/bengtbjorkberg/WebstormProjects/ResourceEdge/node_modules/mongoose/node_modules/kareem/index.js:18:7
at process._tickCallback (node.js:355:11)
Process finished with exit code 1
What I have checked.
Threw tons of log messages, looks like it falls over at different places, or I am missing one of the places
Spent a lot of time looking at the last call (/api/skilllist), to be working quite OK.
Where it gets REALLY weird
If I start the developer console in Chrome, the problem DOES NOT OCCURE, which is probably why it worked all day but then stopped working when I wanted to show it to someone...
While I was writing this, I realised that I could use safari, and it falls over with the java console on, and it is the same line server side. And its the database call below. But if I turn the console on in Chrome, it starts working... What am I missing
exports.canlist = function (req, res) {
// use mongoose to get all profiles in the database
console.log("Canlist called");
Profile.find( {}, {'_id':1, 'alias':1, 'img':1, 'summary':1, 'keys':1}, function(err, profiles) {
// if there is an error retrieving, send the error. nothing after res.send(err) will execute
if (err) {
console.log("Error " + err)
res.send(err)
}
console.log("Sending back " + profiles.length + " profiles")
res.json(profiles); // return all todos in JSON format
});
};
So, after much faffing about, I figured it out. Not why it works when using java console it open on Chrome.
Basically the res.send(err) command is wrong.
First its missing return.
Secondly I THINK you are suppose to use the longer form like this:
return res.status(500).send(err);
return res.json(profile);
Think the problem was that it was confusing express, which probably do some pre processing, which may explain why it "seems" to blow up at all kind of times.
(Would have kept quiet about this, as it is a bit of a ridiculous mistake, but it saves someone a few hours staring at the screen I might get a christmas card this year. (or at least get to tell santa that I've been a good boy)

$window.open called in a promise breaks the second promise

I am pretty new to angularjs. Love it but sometimes get surprises that I just can't find an answer.
I have a function that lets the user build a list to a file by clicking a link. This can take up to a minute so the user can build different lists at the same time. The meaning is that once the file is build a popup appears to download it.
All of this works when only 1 file is build. After a half a minute the download popup shows up. But when 2 are build at the same time the second POST is cancelled (it arrives in the error callback) once the pop-up of the first appears.
If I remove $window.open both POST are finished succesfully and I get both fileId's in the console log.
Clearly the problem lies with $window.open. But why? What am I doing wrong or how can I solve this? Anyone an idea? I have been searching for hours already but can't seem to find anything on this case.
$scope.idPromise = $http.post(restPath, $scope.data)
.then(function(data){
console.log(data.data);
console.log(data.data.fileId);
$window.open(restPath + 'files/' + data.data.fileId, '_self');
},function(error){
console.log('promise error')
});
So when two lists are build without window.open in the code I get something like this in my console:
POST http://localhost:8080/xxx-rest/rest/lijsten/vastelijsten/gezinshoofdenMetLeden 200 OK 1m
POST http://localhost:8080/xxx-rest/rest/lijsten/vastelijsten/gezinshoofdenMetLeden 200 OK 1m 40s
report3545743463669473959.xlsx
report4733168386603499105.xlsx
when window.open is placed in the code:
POST
http://localhost:8080/xxx-rest/rest/lijsten/vastelijsten/gezinshoofdenMetLeden
200 OK 1m
POST
http://localhost:8080/xxx-rest/rest/lijsten/vastelijsten/gezinshoofdenMetLeden
x 56,39s
report1588183186872251177.xlsx
promise error
As you can see the last POST is stopped as soon as the first POST gets a response, so for the second POST no response is gotten.
Update
It seems the problem lies with the grunt server and the liveReload Protocol. My college had always told me to ignore this error but it is now clear that because the page loses connection that the promise fails.
Does anyone have experience with grunt and liveReload? Can I fix this? Will it work on a production server?
The console error I get:
De verbinding met ws://localhost:35729/livereload werd onderbroken
tijdens het laden van de pagina. this.socket = new
this.WebSocket(this._uri);
(translated: the connection with... was interrupted)

Resources