I'm using varnish in front of a tile server to cache mapbox tiles. To remove old tiles, I intended to use bans to effectively remove a large number of cached tiles. My problem is that varnish still uses the cached objects (at least the age in the response indicates this) and doesn't contact the backend.
I'm first requesting http://varnish/5/3/4.pbf, then adding a ban with curl -X BAN -H 'X-Purge-Regex: 5/3/4.pbf' varnish or alternatively varnishadm and then ban obj.http.url ~ 5/3/4.pbf and afterwards requesting http://varnish/5/3/4.pbf again.
In the beginning my ban list is empty:
Present bans:
1610117471.434488 1 C
The ban is added succesfully with curl -X BAN -H 'X-Purge-Regex: 5/3/4.pbf' varnish
<!DOCTYPE html>
<html>
<head>
<title>200 Ban added</title>
</head>
<body>
<h1>Error 200 Ban added</h1>
<p>Ban added</p>
<h3>Guru Meditation:</h3>
<p>XID: 8</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>
and shows up in the ban list
Present bans:
1610117369.028870 0 - obj.http.url ~ 5/3/4.pbf
1610117307.220739 1 C
After requesting http://varnish/5/3/4.pbf again, the ban list indicates that the ban was used
Present bans:
1610117471.434488 1 - obj.http.url ~ 5/3/4.pbf
but the age of the response is not 0, because it's still the object from the first request.
After a short time the ban is removed:
Present bans:
1610117471.434488 1 C
My vcl_recv looks like this but the error is probably somewhere else as it also doesn't work with varnishadm:
sub vcl_recv {
unset req.http.cookie;
# Allowing PURGE from localhost
if (req.method == "BAN"||req.method == "PURGE") {
if (!client.ip ~ purge) {
return(synth(405,"Not allowed."));
}
if (req.method == "BAN") {
ban("obj.http.url ~ " + req.http.X-Purge-Regex);
# Throw a synthetic page so the
# request won't go to the backend.
return(synth(200, "Ban added"));
}
if (req.method == "PURGE") {
return (purge);
}
}
}
I also tried to use the vcl_purge from https://stackoverflow.com/a/61507014 but this doesn't seem to help for bans (?).
I'm using the X-Purge-Regex header to not worry about having to escape special characters like in https://stackoverflow.com/a/38526921 but just a ban like obj.http.url ~ 0 doesn't work.
I'm using varnish 6.5 with vcl 4.0.
Ban request
* << Request >> 54
- Begin req 53 rxreq
- Timestamp Start: 1610121483.345437 0.000000 0.000000
- Timestamp Req: 1610121483.345437 0.000000 0.000000
- VCL_use boot
- ReqStart 192.168.48.2 50882 http
- ReqMethod BAN
- ReqURL /
- ReqProtocol HTTP/1.1
- ReqHeader Host: varnish-volatile
- ReqHeader User-Agent: curl/7.64.0
- ReqHeader Accept: */*
- ReqHeader X-Purge-Regex: 0
- ReqHeader X-Forwarded-For: 192.168.48.2
- VCL_call RECV
- VCL_acl MATCH purge "importer"
- VCL_return synth
- VCL_call HASH
- VCL_return lookup
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason Ban added
- RespHeader Date: Fri, 08 Jan 2021 15:58:03 GMT
- RespHeader Server: Varnish
- RespHeader X-Varnish: 54
- VCL_call SYNTH
- RespHeader Content-Type: text/html; charset=utf-8
- RespHeader Retry-After: 5
- VCL_return deliver
- Timestamp Process: 1610121483.347281 0.001844 0.001844
- RespHeader Content-Length: 246
- Storage malloc Transient
- Filters
- RespHeader Accept-Ranges: bytes
- RespHeader Connection: keep-alive
- Timestamp Resp: 1610121483.347557 0.002120 0.000276
- ReqAcct 98 0 98 218 246 464
- End
GET after adding ban
* << Request >> 32806
- Begin req 32805 rxreq
- Timestamp Start: 1610121552.733872 0.000000 0.000000
- Timestamp Req: 1610121552.733872 0.000000 0.000000
- VCL_use boot
- ReqStart 192.168.48.1 55176 http
- ReqMethod GET
- ReqURL /public.snow_db/0/0/0.pbf
- ReqProtocol HTTP/1.1
- ReqHeader Host: localhost:8090
- ReqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko/20100101 Firefox/84.0
- ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
- ReqHeader Accept-Language: en-US,en;q=0.5
- ReqHeader Accept-Encoding: gzip, deflate
- ReqHeader DNT: 1
- ReqHeader Connection: keep-alive
- ReqHeader Upgrade-Insecure-Requests: 1
- ReqHeader Pragma: no-cache
- ReqHeader Cache-Control: no-cache
- ReqHeader X-Forwarded-For: 192.168.48.1
- VCL_call RECV
- ReqUnset Host: localhost:8090
- ReqHeader host: localhost:8090
- VCL_return hash
- ReqUnset Accept-Encoding: gzip, deflate
- ReqHeader Accept-Encoding: gzip
- VCL_call HASH
- VCL_return lookup
- Hit 28 601789.331504 10.000000 0.000000
- VCL_call HIT
- VCL_return deliver
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader content-encoding: gzip
- RespHeader content-type: application/x-protobuf
- RespHeader date: Fri, 08 Jan 2021 15:09:02 GMT
- RespHeader Vary: Accept-Encoding
- RespHeader X-Varnish: 32806 28
- RespHeader Age: 3010
- RespHeader Via: 1.1 varnish (Varnish/6.5)
- VCL_call DELIVER
- VCL_return deliver
- Timestamp Process: 1610121552.734070 0.000197 0.000197
- Filters
- RespHeader Accept-Ranges: bytes
- RespHeader Content-Length: 295
- RespHeader Connection: keep-alive
- Timestamp Resp: 1610121552.734217 0.000345 0.000147
- ReqAcct 414 0 414 272 295 567
- End
Reproducing the issue
To reproduce the bug:
git clone https://github.com/Baschdl/varnish-ban-setup.git && cd varnish-ban-setup
docker-compose up
Open http://localhost:8092/5/3/1.pbf
docker-compose exec varnish varnishadm ban obj.http.url ~ pbf
Open http://localhost:8092/5/3/1.pbf again and you will get the old object
The obj.http.url ~ 5/3/4.pbf ban that you are issuing, is matching a url response header.
Remember: the URL is a request header, not a response header. No reason to panic, what you're doing makes perfect sense, and is related to the scope of the so-called ban lurker.
Ban lurker
The ban lurker is a thread that asynchronously processes bans on the ban list and matches objects to the bans in order to remove patterns of objects from the cache.
The ban lurker doesn't operate within a request scope, but only is aware of the object scope.
In order to successfully match request information, request context can be added as a response header. And that's what you're doing via obj.http.url
So why doesn't the ban work?
The reason why your ban isn't working, is because you didn't set obj.http.url in your VCL file. As a result, the ban lurker cannot match any objects to it.
How to fix the issue
The solution is simple: set the missing headers in the backend response context, as illustrated below:
sub vcl_backend_response {
set beresp.http.url = bereq.url;
set beresp.http.host = bereq.http.host;
}
When the backend responds, and right before the object gets stored in cache, we can set the missing headers.
After that, the ban lurker will be able to match the ban expression to the right objects, and remove them from cache.
Don't forget the objects aren't immediately matched: they are only removed when they reach the ban_lurker_age, which is set to 1 minute by default.
Related
I am trying to create a program that logs into a website for me. The problem is, when I follow the redirection the website supplies a unique cookie that I can't figure out how to add to the post request. I have been going through each of the libcurl options on the man page, but I can't find anything that will do this. So far this is the post request function that I have.
void webpost(char* url, char* postdata) {
CURL *handler = curl_easy_init();
CURLcode err;
long size = sizeof(postdata);
if (handler) {
curl_easy_setopt(handler, CURLOPT_URL, url);
curl_easy_setopt(handler, CURLOPT_POSTFIELDSIZE, 50L);
curl_easy_setopt(handler, CURLOPT_POSTFIELDS, postdata);
curl_easy_setopt(handler, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(handler, CURLOPT_VERBOSE, 1L);
err = curl_easy_perform(handler);
if (err != CURLE_OK) {
printf("ERROR POST: %s returned (%s)\n", url, curl_easy_strerror(err));
}
curl_easy_cleanup(handler);
}
}
When this function runs, I get the following result.
* Trying 10.10.10.10...
* TCP_NODELAY set
* Connected to website.com (10.10.10.10) port 2048 (#0)
> POST /login HTTP/1.1
Host: website.com
Accept: */*
Content-Length: 50
Content-Type: application/x-www-form-urlencoded
* upload completely sent off: 50 out of 50 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Moved temporarily
< Date: Wed, 26 Aug 2020 05:59:50 GMT
< Server: EZproxy
< Expires: Mon, 02 Aug 1999 00:00:00 GMT
< Last-Modified: Wed, 26 Aug 2020 05:59:50 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Cache-Control: post-check=0, pre-check=0
< Pragma: no-cache
< Set-Cookie: ezproxy=uRZWAo3IsKyR9O0; Path=/; Domain=.website.com
< Location: http://website.com/connect?session=suRZWAo3IsKyR9O0&url=menu
< Connection: close
<
* Closing connection 0
* Issue another request to this URL: 'http://website.com/connect?session=suRZWAo3IsKyR9O0&url=menu'
* Switch from POST to GET
* Hostname website.com was found in DNS cache
* Trying 10.10.10.10...
* TCP_NODELAY set
* Connected to website.com (10.10.10.10) port 2048 (#1)
> GET /connect?session=suRZWAo3IsKyR9O0&url=menu HTTP/1.1
Host: website.com
Accept: */*
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Moved temporarily
< Date: Wed, 26 Aug 2020 05:59:50 GMT
< Server: EZproxy
< Expires: Mon, 02 Aug 1999 00:00:00 GMT
< Last-Modified: Wed, 26 Aug 2020 05:59:50 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Cache-Control: post-check=0, pre-check=0
< Pragma: no-cache
< Set-Cookie: ezproxy=uRZWAo3IsKyR9O0; Path=/; Domain=.website.com
< Location: http://website.com/connect?session=ruRZWAo3IsKyR9O0&url=menu
< Connection: close
<
* Closing connection 1
* Issue another request to this URL: 'website.com/connect?session=ruRZWAo3IsKyR9O0&url=menu'
* Hostname website.com was found in DNS cache
* Trying 10.10.10.10...
* TCP_NODELAY set
* Connected to website.com (10.10.10.10) port 2048 (#2)
> GET /connect?session=ruRZWAo3IsKyR9O0&url=menu HTTP/1.1
Host: website.com
Accept: */*
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 26 Aug 2020 05:59:50 GMT
< Server: EZproxy
< Content-Type: text/html
< Connection: close
<
<html>
<head>
<title>Cookie Required</title>
</head>
<body>
<p>
Licensing agreements for these databases require that access be extended
only to authorized users. Once you have been validated by this system,
a "cookie" is sent to your browser as an ongoing indication of your authorization to
access these databases. This cookie only needs to be set once during login.
</p>
<p>
If you are using a firewall or network privacy program, you may need
reconfigure it to allow cookies to be set from this server.
</p>
<p>
As you access databases, they may also use cookies. Your ability to use those databases
may depend on whether or not you allow those cookies to be set.
</p>
<p>
To login again, click here.
</p>
</body>
</html>
* Closing connection 2
simply add a header in the next request to a site matching the domain and path parameters of the Set-cookie: header that says:
Cookie: ezproxy=uRZWAo3IsKyR9O0
That will be enough for the server to recognize and locate the session you come from, so it can locate the data belonging to your session.
You can read HTTP for a description of the status management mechanism and read about the Cookie and Set-Cookie headers.
Thank you, I was able to get it working by saving the cookie to a file and loading it in the same request.
curl_easy_setopt(handler, CURLOPT_COOKIEJAR, "cookies.txt");
curl_easy_setopt(handler, CURLOPT_COOKIEFILE, "cookies.txt");
I have some problem with Alfresco (5.0.d), my AngularJS (1.4.3) client and the CORS settings (typical cross-domain / No'Access-Control-Allow-Origin' header is present on the requested resource problem).
I am calling the Alfresco REST API from an AngularJS application running on localhost:3000, to an Alfresco instance running on localhost:8080 (Tomcat, no reverse-proxy in front).
This XHR call works fine:
http://localhost:8080/alfresco/service/slingshot/live-search-docs?t=Project&u=admin&pw=admin&alf_ticket=TICKET_9d9780c83b8b9525c7acb9d3d8da66c5c902fb76
This one
http://localhost:8080/alfresco/service/api/login?u=admin&pw=admin
only works fine in Internet Explorer 11, but in Chrome and Firefox, I am getting status code 200 returned with 0 bytes and the error in the JS console:
XMLHttpRequest cannot load
http://localhost:8080/alfresco/service/api/login?u=admin&pw=admin. No
'Access-Control-Allow-Origin' header is present on the requested
resource. Origin 'http://localhost:3000' is therefore not allowed
access.
In the Alfresco web.xml, I have enabled the CORS filter as follows:
<!-- CORS Filter Mappings Begin -->
<filter-mapping>
<filter-name>CORS</filter-name>
<url-pattern>/api/*</url-pattern>
<url-pattern>/service/*</url-pattern>
<url-pattern>/s/*</url-pattern>
<url-pattern>/cmisbrowser/*</url-pattern>
</filter-mapping>
<!-- CORS Filter Mappings End -->
<!-- CORS Filter Begin -->
<filter>
<filter-name>CORS</filter-name>
<filter-class>com.thetransactioncompany.cors.CORSFilter</filter-class>
<init-param>
<param-name>cors.allowGenericHttpRequests</param-name>
<param-value>true</param-value>
</init-param>
<init-param>
<param-name>cors.allowOrigin</param-name>
<!-- <param-value>http://localhost:3000 http://localhost:8081 http://localhost:8080 https://localhost</param-value> -->
<param-value>*</param-value>
</init-param>
<init-param>
<param-name>cors.allowSubdomains</param-name>
<param-value>true</param-value>
</init-param>
<init-param>
<param-name>cors.supportedMethods</param-name>
<param-value>GET, HEAD, POST, PUT, DELETE, OPTIONS</param-value>
</init-param>
<init-param>
<param-name>cors.supportedHeaders</param-name>
<param-value>origin, authorization, x-file-size, x-file-name, content-type, accept, x-file-type</param-value>
</init-param>
<init-param>
<param-name>cors.supportsCredentials</param-name>
<param-value>true</param-value>
</init-param>
<init-param>
<param-name>cors.maxAge</param-name>
<param-value>3600</param-value>
</init-param>
</filter>-->
<!-- CORS Filter End -->
The calls from within Angular:
The one that works fine:
$http.get('http://localhost:8080/alfresco/service/slingshot/live-search-docs?t=Project&alf_ticket=TICKET_9d9780c83b8b9525c7acb9d3d8da66c5c902fb76').then(function(response) {
console.log('DATA LOADED: ' + response.items);
$scope.contents = response.items;
});
Response headers:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Cache-Control: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Content-Type: application/json;charset=UTF-8
Content-Length: 85
Date: Thu, 13 Aug 2015 06:37:24 GMT
The one that fails:
$http.get('http://localhost:8080/alfresco/service/api/login?u=' + $scope.user.email + '&pw=' + $scope.user.password).
then(function(response) {
console.log('ALF_TICKET: ' + response.data.ticket);
$scope.alfTicket = response.data.ticket;
$state.go('admin-panel.default.introduction');
}, function(response) {
console.log('LOGIN FAILED');
});
Response headers:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: http://localhost:3000
Vary: Origin
Cache-Control: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Date: Thu, 13 Aug 2015 06:38:36 GMT
I have the $httpProvider configured in my AngularJS app, just to be sure:
.config(['$httpProvider', function($httpProvider) {
$httpProvider.defaults.useXDomain = true;
delete $httpProvider.defaults.headers.common['X-Requested-With'];
}
A CURL request simulated from a different host works fine, interestingly though, I do not get a Access-Control-Allow-Origin header back in the response despite the CORS filter enabled in Alfresco:
curl -H "Origin: http://www.microsoft.com" --verbose "http://alfresco.mycompany.ch:8080/alfresco/service/api/login?u=admin&pw=12#echo"
* Hostname was NOT found in DNS cache
* Trying 10.10.0.183...
* Connected to alfresco.mycompany.ch (10.10.0.183) port 8080 (#0)
> GET /alfresco/service/api/login?u=admin&pw=12#echo HTTP/1.1
> User-Agent: curl/7.37.1
> Host: alfresco.mycompany.ch:8080
> Accept: */*
> Origin: http://www.microsoft.com
>
< HTTP/1.1 200 OK
* Server Apache-Coyote/1.1 is not blacklisted
< Server: Apache-Coyote/1.1
< Cache-Control: no-cache
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Pragma: no-cache
< Content-Type: application/json;charset=UTF-8
< Content-Length: 85
< Date: Thu, 13 Aug 2015 08:11:15 GMT
<
{
"data":
{
"ticket":"TICKET_7d07634afbb25a7e823c0348d907e0790eeff97e"
}
}
* Connection #0 to host alfresco.mycompany.ch left intact
therefore I think it's client/AngularJS related.
=== Update 1: ===
I tried to add an interceptor to the $httpProvider as below, but it doesn't help. I don't see the headers I am setting there in any of the responses, although the interceptor gets called.
$httpProvider.interceptors.push(function() {
return {
'request': function(request) {
return request;
},
'response': function(response) {
console.log('Interceptor called.');
response.config.headers['MyTestHeader'] = '12345';
response.config.headers['Access-Control-Allow-Origin'] = '*';
return response;
}
};
});
=== Update 2: ===
Some more findings, but now a bit weird on the Alfresco side. I am doing two almost similar API calls, to the following two urls:
http://localhost:8080/alfresco/service/api/login
http://localhost:8080/alfresco/service/api/whatever
You can see, they only differ at the end. The one with /whatever returns a Access-Control-Allow-Origin response header, the one with /login does not. It must be related to the Alfresco config / webscript, but I have not yet found the origin of it.
curl -H "Origin: http://www.microsoft.com" --verbose "http://localhost:8080/alfresco/service/api/login"
* Trying ::1...
* Connected to localhost (::1) port 8080 (#0)
> GET /alfresco/service/api/login HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.43.0
> Accept: */*
> Origin: http://www.microsoft.com
>
< HTTP/1.1 400 Bad Request
< Server: Apache-Coyote/1.1
< Cache-Control: no-cache
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Pragma: no-cache
< Content-Type: application/json;charset=UTF-8
< Transfer-Encoding: chunked
< Date: Thu, 13 Aug 2015 12:49:46 GMT
< Connection: close
<
But this one:
curl -H "Origin: http://www.microsoft.com" --verbose "http://localhost:8080/alfresco/service/api/whatever"
* Trying ::1...
* Connected to localhost (::1) port 8080 (#0)
> GET /alfresco/service/api/whatever HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.43.0
> Accept: */*
> Origin: http://www.microsoft.com
>
< HTTP/1.1 404 Not Found
< Server: Apache-Coyote/1.1
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: http://www.microsoft.com
< Vary: Origin
< Cache-Control: no-cache
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Pragma: no-cache
< Content-Type: text/html;charset=UTF-8
< Transfer-Encoding: chunked
< Date: Thu, 13 Aug 2015 12:52:15 GMT
<
I also tried the CORS filter of Tomcat instead, same result.
<filter>
<filter-name>CorsFilter</filter-name>
<filter-class>org.apache.catalina.filters.CorsFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>CorsFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
Questions:
(1) Am I missing something on the client/AngularJS $http side? Should I use a angular-http-interceptor? (Sorry, pretty new to AngularJS). I think that this is most probably the reason here, but not sure what's missing.
(2) I don't see why the CORS filter would not add the 'Access-Control-Allow-Origin' header to both responses but only to the one API call, since both API urls start with /alfresco/service, which should be handled by the CORS filter as per <url-pattern>/service/*</url-pattern>
(And why does it work for one API url, but for the other it does not? Only difference I see is that in one case, I am already authenticated and use an alf_ticket, in the other case I am not. But even if I add an (unnecessary) alf_ticket to the login call, it does not make any difference.)
(3) Why does it work in IE then at all? (by the way: with the Chrome extension https://chrome.google.com/webstore/detail/allow-control-allow-origi/nlfbmbojpeacfghkpbjhddihlkkiljbi it also works in Chrome.)
Related SO question: cross domain call using REST Alfresco
I have the same issue, for some reason the login api "GET" call does not return "Access-Control-Allow-Origin" in the headers. WORKAROUND: The "POST" call works fine!
I faced similar issue and got it resolved by using "/alfresco/s/" instead of "/alfresco/service/".
In your case, try hitting using
http://localhost:8080/alfresco/s/api/login?u=admin&pw=admin.
Try if this works.
I am using Jersey 2.17 for SSE - I have the demo working per the docs: https://jersey.java.net/documentation/latest/sse.html.
In my tests, it works fine with simple curl requests (curl --verbose "localhost:9000/api/broadcast"), but if I add -H 'Accept-Encoding: gzip, deflate' (to replicate my javascript code) it doesn't work (eventually times out with 'transfer closed with outstanding read data remaining'). I'm guessing that jersey says its GZIPed the response, but really it hasn't.
My real problem is that the AngularJS javascript code always adds this Accept-Encoding... header.
I'm looking for a solution that will either
Prevent Accept-Encoding: gzip, deflate' from being added as Request Header when initialising the EventSource() object
Setup Jersey to correctly handle/gzip the events sent to the client.
Update - I've noticed that if I close the EventOutput object in Java code that my events are all sent.
This is the console output when using curl:
curl --verbose "localhost:9000/api/broadcast" -H 'Accept-Encoding: gzip, deflate'
* Hostname was NOT found in DNS cache
* Trying ::1...
* connect to ::1 port 9000 failed: Connection refused
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 9000 (#0)
> GET /api/broadcast HTTP/1.1
> User-Agent: curl/7.37.1
> Host: localhost:9000
> Accept: */*
> Accept-Encoding: deflate, gzip
>
< HTTP/1.1 200 OK
* Server Apache-Coyote/1.1 is not blacklisted
< server: Apache-Coyote/1.1
< vary: Accept-Encoding
< content-encoding: deflate
< content-type: text/event-stream
< transfer-encoding: chunked
< date: Tue, 07 Apr 2015 20:35:42 GMT
< connection: close
<
* transfer closed with outstanding read data remaining
* Closing connection 0
curl: (18) transfer closed with outstanding read data remaining
For reference, my js code in my AngularJS app is:
$scope.sseSource = new EventSource('/api/ptt/1/status');
console.log( $scope.sseSource);
$scope.sseSource.onerror = function (event) {
console.log("onerror");
}
$scope.sseSource.onopen = function (event) {
console.log("onopen");
}
$scope.sseSource.onmessage = function (event) {
console.log("onmessage");
}
My object GETs a file over HTTP.
It does so, using the If-Modified-Since header. If the files has not been modified since the time in the header a Not Modified response will be returned and the file should not be fetched&written. Like so:
class YouTube
#...
def parse
uri = URI("http://i.ytimg.com/vi/#{#id}/0.jpg")
req = Net::HTTP::Get.new(uri.request_uri)
if File.exists? thumbname
stat = File.stat thumbname
req['If-Modified-Since'] = stat.mtime.rfc2822
end
res = Net::HTTP.start(uri.hostname, uri.port) {|http|
http.request(req)
}
File.open(thumbname, 'wb') do |f|
f.write res.body
end if res.is_a?(Net::HTTPSuccess)
end
#...
end
I want to test both cases (in both cases, a file on disk exists). To do so, I'd need to stub either something in Net::HTTP, or I need to stub the File.stat to return an mtime for which I a sure the online resource will return a new or Not-modified-since.
Should I stub (or even mock) Net::HTTP? And if so, what?
Or should I stub mtime to return a date far in the past or far in the future to enforce or suppress the Not-modified Header?
Edit: Diving deeper into the matter, I learned that the i.ytimg.com-domain does not support these headers. So i'll need to solve this by inspecing JSON from the YouTube API. However, the problem "What and how to mock when testing if-modified-since-headers" still stands.
Here is how I conclude the domain does not support this:
$curl -I --header 'If-Modified-Since: Sun, 24 Mar 2013 17:33:29 +0100' -L http://i.ytimg.com/vi/D80QdsFWdcQ/0.jpg
HTTP/1.1 200 OK
Content-Type: image/jpeg
Date: Sun, 24 Mar 2013 16:31:10 GMT
Expires: Sun, 24 Mar 2013 22:31:10 GMT
X-Content-Type-Options: nosniff
Server: sffe
Content-Length: 13343
X-XSS-Protection: 1; mode=block
Cache-Control: public, max-age=21600
Age: 822
There is no "Last-Modified" there. Illustrated with another call, to exaple.com, which does support the if-modified-since headers.
$curl -I --header 'If-Modified-Since: Sun, 24 Mar 2013 17:33:29 +0100' -L example.com
HTTP/1.0 302 Found
Location: http://www.iana.org/domains/example/
Server: BigIP
Connection: Keep-Alive
Content-Length: 0
HTTP/1.1 302 FOUND
Date: Sun, 24 Mar 2013 16:47:47 GMT
Server: Apache/2.2.3 (CentOS)
Location: http://www.iana.org/domains/example
Connection: close
Content-Type: text/html; charset=utf-8
HTTP/1.1 304 NOT MODIFIED
Date: Sun, 24 Mar 2013 16:47:47 GMT
Server: Apache/2.2.3 (CentOS)
Connection: close
Quite simply, I'm trying to generate and download a CSV file from a CakePHP controller. No problem generating the CSV, and everything works until the response >= 4096 bytes.
The following controller illustrates the problem:
class TestTypeController extends Controller {
public function csv($size = 100) {
# set the content type
Configure::write('debug', 0);
$this->autoRender = false;
$this->response->type('csv');
# send the response
for ($i = 0; $i < $size; $i++)
echo 'x';
}
}
When I call http://baseurl/test_type/csv/4095, I'm prompted to save the file, and the Content-Type header is text/csv. The response headers are:
HTTP/1.1 200 OK
Date: Tue, 05 Jun 2012 14:28:56 GMT
Server: Apache/2.2.22 (Ubuntu)
X-Powered-By: PHP/5.3.10-1ubuntu3.1
Content-Length: 4095
Keep-Alive: timeout=5, max=98
Connection: Keep-Alive
Content-Type: text/csv; charset=UTF-8
When I call http://baseurl/test_type/csv/4096, the file is printed to the screen, and the response headers are:
HTTP/1.1 200 OK
Date: Tue, 05 Jun 2012 14:28:53 GMT
Server: Apache/2.2.22 (Ubuntu)
X-Powered-By: PHP/5.3.10-1ubuntu3.1
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 38
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html
Obviously, 4kB is the limit where Content-Encoding starts gzipping the response. I'm not familiar with how the Content-Type is meant to react, but I'd obviously prefer it to remain text/csv.
The same problem occurs using the RequestHandlerComponent to manage the type of the response.
I'm using the CakePHP 2.2.0-RC1, but I've verified the problem exists with stable 2.1.3. Any ideas? Pointers in the right direction?
The answer was pretty simple -- the controller should be returning the CSV data instead of echoing it.