Mechanize/Ruby read source code of 404 page - http-status-code-404

All I'm doing is loading mechanize, and getting a page that returns 404. But that's exactly what I want. The 404 page has plenty of html I'd like to use in my example.
a = mechanize.new
a.get('http://www.youtube.com/watch?v=e4g8jriw4rg')
a.page
=> nil
I can't seem to find any further info on this.

You need to handle the exception:
begin
page = a.get 'http://www.youtube.com/watch?v=e4g8jriw4rg'
rescue Mechanize::ResponseCodeError => e
puts e.response_code # the status code as a string, e.g. "404"
page = e.page
end
puts page.title

This may have been the case when the answer was written (the code changed about 5 years ago) but it's no longer the case. You can now set allowed_error_codes on the agent instance to an array of Integers or Strings with the values set to the HTTP Response codes you wish to handle without an exception. The docs (as of writing this) note that "2xx, 3xx and 401 status codes will be handled without checking this list."

Related

WebApi and Ampersands in name

So my angular website has a webapi with the following method.
[Route("items/{itemName}")]
public object GetMcguffinsByItem(string itemName)
{
return _mcguffinsService.GetAllByItemName(itemName);
}
However, an item name can have an ampersand as a valid character. However when attempting to use items that do have an ampersand, the method will return a 400 badrequest.
Im not sure how to go about fixing this problem.
For more verification: I was under the impression that encoding and using %26 is all required to pass an ampersand to part of the URI. It seems to be a common answer when searching my problem. I have excluded the angular as I can verify that it builds the string correctly, and other names produce the desired result.
The javascript method encodeURIComponent() followed by using the angular service double encodes the item name, and returns a 404.
EDIT:
Sample Input:
A&B 266
After Encoding:
A%26B%20266
Console:
angular.js:10722 GET http://localhost:60894/api/v1/mcguffins/items/A%26B%20266 404 (Not Found)
Using the browser on api directly with same input gives this error:
[HttpException (0x80004005): A potentially dangerous Request.Path value was detected from the client (&).]
System.Web.HttpRequest.ValidateInputIfRequiredByConfig() +11944671
System.Web.PipelineStepManager.ValidateHelper(HttpContext context) +55

php file() triggers no Exception on valid URL

try {
$antwort = file_get_contents('http://not_existing.notnotnot', false);
if($antwort===false) echo 'ERROR';
} catch(Exception $e) {
$e->getMessage();
}
var_dump($antwort); // returns string(0) ""
I get no Exception, no false, just empty content for every URL. A valid URL returns with this snippet the right content. Why can't I get exceptions for an invalid URL?
I came to this question because a wget on the same server leads to a valid return, but with a php script I can't file() the same URL. Really weird and I have no idea how to debug it.
It won't throw an exception if the file isn't found; it will raise a warning-level error. Those are different things. From the docs:
An E_WARNING level error is generated if filename cannot be found, maxlength is less than zero, or if seeking to the specified offset in the stream fails.
You should check for a false return, as you do, and not expect to catch an exception.
Also keep in mind when fetching a URL that the remote server may return an incorrect status code (instead of the expected 404), causing your script to think the file exists when it does not. You may need to check for empty values ("") as well.
As a rule, you should avoid using file_get_contents to access files via HTTP. It's not terribly secure, and many hosts don't even allow you to use it that way. Instead, use cURL, which is specifically designed for retrieving data over the web, including via HTTP.

Spontaneous Server Errors During AngularJS $http calls

I'm building an SPA in AngularJS served by a Laravel (5.1) backend. Of late I've been encountering an annoying error, a server 500 or code 0 error which is abit hard to explain how it comes but let me try to may be someone will understand the dental formula of my problem.
When i start my AngularJS controller, I make several server calls (via independent $http calls from services) to retrieve information i might later need in the controller. For example,
Functions.getGrades()
.then(function(response)
{
$scope.grades = response.data;
});
Subjects.offered()
.then(function(response)
{
$scope.subjects = response.data;
});
Later on i pass these variables (grades or subjects) to a service where they are used for processing. However, these functions are randomly returning code 500 server errors after they run, and sometimes returning status code 0 after running. This happens in a random way and it is hard for me to point out the circumstances leading to their popping up. This leaves me with frequent empty Laravel-ised error screens like the ones shown below.
Anyone reading my mind?
Ok, after a suggestion given in a comment above that I check my Laravel log files (located in storage/logs/laravel.log- Laravel 5.1), i found out that the main error most of these times was this one: 'PDOException' with message 'SQLSTATE[HY000] [1044] Access denied for user ''#'localhost' to database 'forge'' in ..., plus another one that paraphrased something like No valid encrypter found. These were the key opener.
On reading another SO thread here, it said in part:
I solved, sometimes laravel not read APP_KEY in .ENV. And returns a value "SomeRandomString" (default is defined in config / app.php), and have the error "key length is invalid", so the solution is to copy the value of APP_KEY, to the value 'key 'in config / app.php, that's all! I solved!
That was exactly the issue! When loading the DB params from the .env to config/database.php, Laravel was sometimes unable to read the environment variables and went for the fallback default fallback options (forge for DB name and username and SomeRandomString for the APP_KEY). So, to solve this i just did as advised: copied the APP_KEY in .env to the config/app.php and edited the default DB parameters to the actual DB name and username/password I'm using. Just that and i was free from pollution. Hope someone finds this helpful.

CakePHP Cakemenu plugin fails after global error due to incorrect string encoding

I am using CakePHP 2.1.2 with PHP 5.3.5 and a plugin called 'Cakemenu' which normally works fine. The plugin stores menus in a db table with the menu link stored as text like
array('plugin'=>null,'controller'=>'assets','action'=>'index');
The helper in the plugin gets those values, then executes this code to convert that text to an array:
//Try to evaluate the link (if starts with array)
if (eregi('^array', $value['Menu']['link'])) {
$code = "\$parse = " . $value['Menu']['link'] . ";";
$result = eval($code);
if (is_array($parse)) {
$value['Menu']['link'] = $parse;
}
}
Everything works fine unless CakePHP is handling an error. For example if I mistype the name of a controller in the browser I should get a menu and then the missing controller message. Instead I get a page full "Parse error: syntax error, unexpected $end in..." messages pointing to the line with the eval statement. If I printout the variable that is getting eval'ed I see that it has been (incorrectly) encoded with Html entities when it normally does not.
Good string to be eval'ed:
$parse = array('plugin'=>null,'controller'=>'assets','action'=>'index');
Bad string to be eval'ed:
$parse = array('plugin'=>null,'controller'=>'Parts','action'=>'add');
To temporarily fix the problem I added two statements to just replace the offending characters
$value['Menu']['link'] = str_replace( ''','\'',$value['Menu']['link']);
$value['Menu']['link'] = str_replace( '>','>',$value['Menu']['link']);
and everything works great again. Some other pieces of information that might be helpful is that the array of data used to generate the menu is read during the beforeFilter of the app and saved in a view variable and then the menu is generated as an element in the view.
I'm thinking that the error causes CakePHP (or PHP) to skip some loading or configuration process and that causes the string to be mishandled. Any help would be appreciated, thanks
Your beforeFilter() method won't be executed on error pages. You'll have to handle your errors yourself and manually call beforeFilter(). I wrote a blog post on how to use custom error pages - pay close attention to the Controller Callbacks section.

How do you specify an HTTP status code in Cakephp?

In my controller, I check a condition to see if the user is allowed to do something. If the check fails, I want to send a 403 back to the browser. How do I do that in Cakephp?
EDIT - This question is quite old and covers different versions of the CakePHP framework. Following is a summary of which version each answer applies to. Don't forget to vote on the solution that helps most.
CakePHP 3.x and 4.x - using response object (Roberto's answer)
CakePHP 2.x - using exceptions (Brad Koch's answer) [preferred solution]
CakePHP 2.x - setting header only (Asa Ayers' answer)
CakePHP 1.x - using error handler (my other answer)
CakePHP 1.x - setting header only (this answer)
EDIT #2 - A more detailed answer for CakePHP 2.x has been added by Mark37.
EDIT #3 - Added solution for CakePHP. (May 2018: CakePHP 3.5 did some function renaming, solution by Roberto is still valid.)
By looking at the relevant API code from the previous comment, it seems you can call Controller::header($status) to output a header without redirection. In your case, the proper usage is most likely:
$this->header('HTTP/1.1 403 Forbidden');
$this->response->statusCode(403);
Will set the status code when Cake is ready to send the response. CakeResponse::send() expects to send the status code and message, so in my tests I think my using header() was getting overwritten. using $this->header('HTTP/1.1 400 Bad Request') doesn't work either because Cake expects any call to $this->header to be split on a colon ex: $this->header('Location: ...')
Notes concerning CakePHP 3.x seem to be missing, so to make this thread complete:
For CakePHP 3.x and 4.x use:
$response = $this->response->withStatus(403);
return $response;
For versions before CakePHP 3.3.x you can use the same style as CakePHP 2.x:
$this->response->statusCode('code');
Note that using the PHP function directly also works (http_response_code(403); die();), though using the response object seems like the intended method.
In CakePHP 2, the preferred method is to throw an exception:
throw new ForbiddenException();
I'm adding in my two cents here because I don't feel like any of these answers covered this topic as thoroughly as I would have liked (at least for Cake 2.x).
If you want to throw an error status, use the Exception classes (as mentioned in other answers):
throw new BadRequestException(); // 400 Bad Request
// Or customize the code...
throw new BadRequestException('Custom error message', 405); // 405 Method Not Allowed
Fun fact: Cake will automatically do some magical error rendering even for RESTful calls via the ExceptionRenderer class. Even more fun of a fact is that it's based on the Status Code, not the fact that an Exception might have been thrown, so if you set the status code to > 400 on your own you will likely get error messages even if you didn't want them.
If you want to return a specific status code for a REST JSON/XML endpoint, take advantage of the new CakeResponse object, but also make sure that you add the special _serialize variable or you'll end up with a 'view not found' error as cake will attempt to find a view to render your JSON/XML. (This is by design - see the JsonView/XmlView class.)
$this->response->setStatus(201); // 201 Created
$this->set('_serialize', array()); // Value must be something other than null
And lastly, if you want to send a non-200 status for a regularly rendered page, you can just use the setStatus() method with nothing else as mentioned in a previous answer:
$this->response->setStatus(201);
UPDATE:
$this->response->setStatus('code');
is no longer available. Use
$this->response->statusCode('code');
Upon revisiting this question, and reading Adriano's comment on my previous answer (regarding redirecting the user to a friendly page), I have come up with a new solution.
Within a controller you can call $this->cakeError('error404') to generate a friendly 404 page. This can can be customised (as with other errors) by creating file at 'app/views/errors/error404.ctp'.
After having a closer look at the code for cakeError, my recommendation is to try extending Cake's ErrorHandler by creating a file at 'app/error.php' or (possibly more preferable) 'app/app_error.php'.
The code for your error403 (mimicking the error404 code) could read as follows:
class AppError extends ErrorHandler {
function error403($params) {
extract($params, EXTR_OVERWRITE);
$this->error(array(
'code' => '403',
'name' => 'Forbidden',
'message' => sprintf(__("Access was forbidden to the requested address %s on this server.", true), $url, $message)));
$this->_stop();
}
}
You should also be able to provide a custom view for this error by creating 'app/views/errors/error403.ctp'. Here is a modified version of the error404 view:
<h2><?php echo $name; ?></h2>
<p class="error">
<strong>Error: </strong>
<?php echo sprintf(__("Access was forbidden to the requested address %s on this server.", true), "<strong>'{$message}'</strong>")?>
</p>
It has changed again since CakePHP 3.6:
Use now
$this->setResponse($this->response->withStatus(403) );
return $this->response; // use this line also
instead of
$response = $this->response->withStatus(403);
https://api.cakephp.org/3.7/class-Cake.Controller.Controller.html#_setResponse
Perhaps something in this section of the cakephp manual can help you.
redirect(string $url, integer $status,
boolean $exit)
The flow control method you’ll use
most often is redirect(). This method
takes its first parameter in the form
of a CakePHP-relative URL. When a user
has successfully placed an order, you
might wish to redirect them to a
receipt screen. The second parameter
of redirect() allows you to define an
HTTP status code to accompany the
redirect. You may want to use 301
(moved permanently) or 303 (see
other), depending on the nature of the
redirect.
The method will issue an exit() after
the redirect unless you set the third
parameter to false.
You can use cakephp response for custom message:
$this->response->header('HTTP/1.0 201', 'custom message');
$this->response->send();
Core PHP link code works in cakePHP.
header('HTTP/1.1 403 Forbidden');

Resources