CORS enabled but still getting CORS error - angularjs

I'm trying to get a JSON object from an API and the devs for the API said they just enabled CORS but I'm stilling getting the bellow error.
XMLHttpRequest cannot load http://example.com/data/action/getGame/9788578457657. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://dev.our-domain.local' is therefore not allowed access.
I'm using AngularJS to get the JSON in a Service with
app.service("gameService", function ($http, $q)
{
function getGame(GameId) {
var deferred = $q.defer()
var url = 'http://example.com/data/action/getGame/' + gameId;
// var url = 'https://jsonplaceholder.typicode.com/albums/' + gameId; // THIS WORKS
$http({
method: 'GET',
cache: true,
url: url,
headers: {
'Content-Type': 'application/json;charset=UTF-8'
}
}).
then(function(response) {
//your code when success
deferred.resolve(response);
console.log('gameService HTTP CORS SUCCESS!');
}, function(response) {
//your code when fails
console.log('gameService HTTP CORS ERROR!');
// deferred.resolve('');
deferred.reject(response);
});
return deferred.promise;
}
this.getGame = getGame;
})
My AngularJS service works when I test it with jsonplaceholder which has CORS enabled.
Am I missing something?
The API devs said that two CORS-Headers are added to data.service responses but I don't see them. This is what I see on the headers when I curl down the JSON object.
$ curl -X HEAD -i http://example.com/data/action/getGame/9788578457657
HTTP/1.1 200 OK
Date: Wed, 14 Dec 2016 10:39:17 GMT
Server: WildFly/8
Expires: Wed, 14 Dec 2016 10:39:17 GMT
X-Powered-By: Undertow/1
X-dmg-elapsed-time: 20ms
X-dmg-host-address: 1??.??.???.??
Vary: Accept-Encoding,Origin
X-dmg-generated-time: Wed, 14 Dec 2016 10:39:17 GMT
Content-Type: application/json;charset=UTF-8
Content-Language: en-
X-dmg-node-name: defg_node_1
X-Varnish-Bereq-Backend: real_backend_foo_bar_uk
X-Varnish-Bereq-Retries: 0
Last-Modified: Wed, 14 Dec 2016 10:39:17 GMT
Cache-Control: public, max-age=300
X-Varnish: 6876870
Age: 0
Via: 1.1 varnish-v4
X-Varnish-Cache: MISS
X-Varnish-Trimen: www.trimen.com
X-Varnish-Served-By-Host: snarf.foo.uk
X-Varnish-Served-By-IP: 100.100.10.80
X-Varnish-Pool: http_pages
X-Varnish-Req-Backend-Hint: dead
X-Varnish-Req-Restarts: 0
X-Varnish-Hash: /data/action/getGame/9788578457657
X-Varnish-Backend-Ourself: varnish_server_snarf_foo_uk
X-DMG-Version: 6.20.51.2358
Accept-Ranges: none
Connection: keep-alive
Is this what I should be seeing with CORS enabled or is there something more?
Do I need to add more to my AngularJS Service to http get with Cors enabled, as add in more to:
headers: {
'Content-Type': 'application/json;charset=UTF-8'
}
UPDATE
Passing Origin: in the header on my curl request as suggested by #t.niese
$ curl -H "Origin: http://our-production-domain.com/" --verbose \
> http://example.com/data/action/getGame/9788578457657
* Trying 1?.???.??.???...
* Connected to http://example.com/ (1?.???.??.???) port 80 (#0)
> GET /data/action/getGame/9788578457657 HTTP/1.1
> Host: http://example.com/
> User-Agent: curl/7.43.0
> Accept: */*
> Origin: http://our-production-domain.com/
>
< HTTP/1.1 200 OK
< Date: Wed, 14 Dec 2016 11:05:24 GMT
< Server: WildFly/8
< Expires: Wed, 14 Dec 2016 11:05:24 GMT
< X-Powered-By: Undertow/1
< X-dmg-elapsed-time: 27ms
< X-dmg-host-address: 1??.??.???.??
< Vary: Accept-Encoding,Origin
< X-dmg-generated-time: Wed, 14 Dec 2016 11:05:24 GMT
< Content-Type: application/json;charset=UTF-8
< Content-Language: en-
< X-dmg-node-name: defg_node_1
< X-Varnish-Bereq-Backend: real_backend_foo_bar_uk
< X-Varnish-Bereq-Retries: 0
< Last-Modified: Wed, 14 Dec 2016 11:05:24 GMT
< Cache-Control: public, max-age=300
< X-Varnish: 6876870
< Age: 0
< Via: 1.1 varnish-v4
< X-Varnish-Cache: MISS
< X-Varnish-Trimen: www.trimen.com
< X-Varnish-Served-By-Host: snarf.foo.uk
< X-Varnish-Served-By-IP: 100.100.10.80
< X-Varnish-Pool: http_pages
< X-Varnish-Req-Backend-Hint: dead
< X-Varnish-Req-Restarts: 0
< X-Varnish-Hash: /data/action/getGame/9788578457657
< X-Varnish-Backend-Ourself: varnish_server_snarf_foo_uk
< X-DMG-Version: 6.20.51.2358
< Accept-Ranges: none
< Transfer-Encoding: chunked
< Connection: keep-alive
<
{
"errorMessage" : null,
"expiry" : "2016-12-14T11:05:24.379+0000",
"data" : {
// json object data here
}
* Connection #0 to host http://example.com/ left intact
}
and..
$ curl -H "Origin: http://qa.our-qa-domain.com/" --verbose \
> http://example.com/data/action/getGame/9788578457657
* Trying 1?.???.??.???...
* Connected to http://example.com/ (1?.???.??.???) port 80 (#0)
> GET /data/action/getGame/9788578457657 HTTP/1.1
> Host: http://example.com/
> User-Agent: curl/7.43.0
> Accept: */*
> Origin: http://qa.our-qa-domain.com/
>
< HTTP/1.1 200 OK
< Date: Wed, 14 Dec 2016 11:06:11 GMT
< Server: WildFly/8
< Expires: Wed, 14 Dec 2016 11:06:11 GMT
< X-Powered-By: Undertow/1
< X-dmg-elapsed-time: 18ms
< X-dmg-host-address: 1??.??.???.??
< Vary: Accept-Encoding,Origin
< X-dmg-generated-time: Wed, 14 Dec 2016 11:06:11 GMT
< Content-Type: application/json;charset=UTF-8
< Content-Language: en-
< X-dmg-node-name: defg_node_1
< X-Varnish-Bereq-Backend: real_backend_foo_bar_uk
< X-Varnish-Bereq-Retries: 0
< Last-Modified: Wed, 14 Dec 2016 11:06:11 GMT
< Cache-Control: public, max-age=300
< X-Varnish: 1343699
< Age: 0
< Via: 1.1 varnish-v4
< X-Varnish-Cache: MISS
< X-Varnish-Trimen: www.trimen.com
< X-Varnish-Served-By-Host: snarf.foo.uk
< X-Varnish-Served-By-IP: 100.100.10.80
< X-Varnish-Pool: http_pages
< X-Varnish-Req-Backend-Hint: dead
< X-Varnish-Req-Restarts: 0
< X-Varnish-Hash: /data/action/getGame/9788578457657
< X-Varnish-Backend-Ourself: varnish_server_snarf_foo_uk
< X-DMG-Version: 6.20.51.2358
< Accept-Ranges: none
< Content-Length: 2988
< Connection: keep-alive
<
{
"errorMessage" : null,
"expiry" : "2016-12-14T11:06:11.927+0000",
"data" : {
// json data object here
}
* Connection #0 to host http://example.com/ left intact
and..
$ curl -H "Origin: http://dev.my-dev.local/" --verbose \
> http://example.com/data/action/getGame/9788578457657
* Trying 1?.???.??.???...
* Connected to http://example.com/ (1?.???.??.???) port 80 (#0)
> GET /data/action/getGame/9788578457657 HTTP/1.1
> Host: http://example.com/
> User-Agent: curl/7.43.0
> Accept: */*
> Origin: http://dev.my-dev.local/
>
< HTTP/1.1 200 OK
< Date: Wed, 14 Dec 2016 11:07:10 GMT
< Server: WildFly/8
< Expires: Wed, 14 Dec 2016 11:07:10 GMT
< X-Powered-By: Undertow/1
< X-dmg-elapsed-time: 28ms
< X-dmg-host-address: 1??.??.???.??
< Vary: Accept-Encoding,Origin
< X-dmg-generated-time: Wed, 14 Dec 2016 11:07:10 GMT
< Content-Type: application/json;charset=UTF-8
< Content-Language: en-
< X-dmg-node-name: defg_node_1
< X-Varnish-Bereq-Backend: real_backend_foo_bar_uk
< X-Varnish-Bereq-Retries: 0
< Last-Modified: Wed, 14 Dec 2016 11:07:10 GMT
< Cache-Control: public, max-age=300
< X-Varnish: 6619151
< Age: 0
< Via: 1.1 varnish-v4
< X-Varnish-Cache: MISS
< X-Varnish-Trimen: www.trimen.com
< X-Varnish-Served-By-Host: snarf.foo.uk
< X-Varnish-Served-By-IP: 100.100.10.80
< X-Varnish-Pool: http_pages
< X-Varnish-Req-Backend-Hint: dead
< X-Varnish-Req-Restarts: 0
< X-Varnish-Hash: /data/action/getGame/9788578457657
< X-Varnish-Backend-Ourself: varnish_server_snarf_foo_uk
< X-DMG-Version: 6.20.51.2358
< Accept-Ranges: none
< Content-Length: 2988
< Connection: keep-alive
<
{
"errorMessage" : null,
"expiry" : "2016-12-14T11:07:10.764+0000",
"data" : {
// JSON object data here
}
* Connection #0 to host http://example.com/ left intact
}
2nd UPDATE
I disables same origin policy in Chrome and these are the headers to my JSON request from Chrome's network panel.
GET data/action/getGame/9788578457657 HTTP/1.1
Host: example.com
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Accept: application/json, text/plain, */*
Origin: http://dev.my-dev.local/
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36
Referer: http://dev.my-dev.local//game/id-9788578457657
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
HTTP/1.1 200 OK
Date: Wed, 14 Dec 2016 15:38:38 GMT
Server: WildFly/8
Expires: Wed, 14 Dec 2016 15:38:38 GMT
X-Powered-By: Undertow/1
X-dmg-elapsed-time: 25ms
X-dmg-host-address: 172.16.0.70
Vary: Accept-Encoding,Origin
X-dmg-generated-time: Wed, 14 Dec 2016 15:38:38 GMT
Content-Type: application/json;charset=UTF-8
Content-Language: en-
X-dmg-node-name: defg_node_1
Content-Encoding: gzip
Content-Length: 1109
X-Varnish-Bereq-Backend: real_backend_foo_bar_uk
X-Varnish-Bereq-Retries: 0
Last-Modified: Wed, 14 Dec 2016 15:38:38 GMT
Cache-Control: public, max-age=300
X-Varnish: 6619151
Age: 0
Via: 1.1 varnish-v4
X-Varnish-Cache: MISS
X-Varnish-Trimen: www.trimen.com
X-Varnish-Served-By-Host: snarf.foo.uk
X-Varnish-Served-By-IP: 100.100.10.80
X-Varnish-Pool: http_pages
X-Varnish-Req-Backend-Hint: dead
X-Varnish-Req-Restarts: 0
X-Varnish-Hash: /data/action/getGame/9788578457657
X-Varnish-Backend-Ourself: arnish_server_snarf_foo_uk
X-DMG-Version: 6.20.51.2358
Accept-Ranges: none
Connection: keep-alive
3rd UPDATE
So after changing the http method to OPTIONS as in
$http({
method: 'OPTIONS',
...
I ge this error in the chrome consoler
XMLHttpRequest cannot load http://example.com/data/action/getGame/9788578457657. Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://dev.my-dev.local'' is therefore not allowed access. The response had HTTP status code 405.
And these are the headers:
OPTIONS /data/action/getGame/9788578457657 HTTP/1.1
Host: example.com
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Access-Control-Request-Method: OPTIONS
Origin: http://dev.my-dev.local/
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36
Access-Control-Request-Headers:
Accept: */*
Referer: http://dev.my-dev.local//game/id-9788578457657
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
HTTP/1.1 405 Method Not Allowed
Date: Wed, 14 Dec 2016 16:52:03 GMT
Server: Varnish
X-Varnish: 6619151
X-Varnish-Trimen: www.trimen.com
X-Varnish-Served-By-Host: snarf.foo.uk
X-Varnish-Served-By-IP: 100.100.10.80
X-Varnish-Pool:
X-Varnish-Req-Backend-Hint: dead
X-Varnish-Req-Restarts: 0
X-DMG-Version: 6.20.51.2358
Content-Type: text/html; charset=utf-8
Retry-After: 5
Content-Length: 49669
Connection: keep-alive

You need to receive the following headers:
Access-Control-Allow-Origin: * (or whatever host you want to restrict to)
Access-Control-Allow-Methods: * (or whatever methods you want to restrict to)
Access-Control-Allow-Headers: Content-Type
Note the last one which is also important because you are setting Content-Type: application/json;charset=UTF-8. If you have any other custom headers you will need to add those too.
These are all to be done on the server though, your app doesn't need to do anything else.
Alternatively (if possible) you can opt to not use application/json at all and set your Content-Type to application/x-www-form-urlencoded, multipart/form-data, or text/plain and no preflight (OPTIONS) request will be done and it won't matter if CORS is enabled on the server or not.

The answer by ed inspired me on my solution
you need to send the following headers during a preflight request (OPTIONS method for the endpoint)
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET,HEAD,PUT,PATCH,POST,DELETE
Access-Control-Allow-Headers: Content-Type, Access-Control-Allow-Origin, xxx
where xxx is whatever additional headers you are sending when doing the POST/PUT/DELETE/etc request
note that when this solution works, I strongly suggest changing the * to restrict it to your known origins

Related

HTTPS Request Not Working in Google App Engine (Python Requests lib) / Cloud Shell

I have been trying to do some html scraping for one of my projects, wherein the it is working from my local computer, but it is now working in google app engine. It is getting timed out.
I have tried using dig from google cloud shell, which is resolving the correct IP but still curl / trace is failing. Billing is enabled, haven't touched any DNS setting in GAE.
The target site is apparently using chunked reply.
from flask import Flask, render_template, request, send_file, redirect, url_for, Response, send_from_directory
import flask
import requests
import bs4
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
import logging
import os
s = requests.Session()
r = s.get("https://kdseodb.smportkolkata.in/ccuPomsPosWeb/index.jsp")
id1 = r.cookies['JSESSIONID']
r2 = s.get("https://kdseodb.smportkolkata.in/ccuPomsPosWeb/apps/dos/PosCTS.xhtml")
id2 = r2.cookies['oam.Flash.RENDERMAP.TOKEN']
headers = {'Accept': 'application/xml, text/xml, */*; q=0.01', 'Accept-Language': 'en-US,en;q=0.9', 'Connection': 'keep-alive', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Faces-Request': 'partial/ajax', 'Sec-Fetch-Dest': 'empty', 'Sec-Fetch-Mode': 'cors', 'Sec-Fetch-Site': 'same-origin', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36', 'X-Requested-With': 'XMLHttpRequest', 'Cookie': 'JSESSIONID='+id1+'; oam.Flash.RENDERMAP.TOKEN=-'+id2+''}
payload = 'javax.faces.partial.ajax=true&javax.faces.source=form1%3AbtnSearch&javax.faces.partial.execute=form1&javax.faces.partial.render=form1&form1%3AbtnSearch=form1%3AbtnSearch&form1%3AcomMloCd_focus=&form1%3AcomMloCd_input=NONE&form1%3AtxtCntNo=ABCD1234567&form1%3AtblData_scrollState=0%2C0&form1_SUBMIT=1&javax.faces.ViewState=PCsvYkkMUQAmP0zfu2tj7G5AqM%2BXrDH8%2BaG%2FROvEtWIHJyXX'
r3 = s.post("https://kdseodb.smportkolkata.in/ccuPomsPosWeb/apps/dos/PosCTS.xhtml", cookies=r.cookies, headers=headers, data=payload)
soup = bs4.BeautifulSoup(r3.text)
dig output from google cloud shell, NXDOMAIN is coming up when used on the IPs
dig kdseodb.smportkolkata.in
; <<>> DiG 9.16.33-Debian <<>> kdseodb.smportkolkata.in
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 414
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;kdseodb.smportkolkata.in. IN A
;; ANSWER SECTION:
kdseodb.smportkolkata.in. 3600 IN A 117.242.152.158
kdseodb.smportkolkata.in. 3600 IN A 203.153.37.38
;; Query time: 6 msec
;; SERVER: 169.254.169.254#53(169.254.169.254)
;; WHEN: Thu Dec 29 01:44:36 UTC 2022
;; MSG SIZE rcvd: 85
dig 117.242.152.158
; <<>> DiG 9.16.33-Debian <<>> 117.242.152.158
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 29831
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;117.242.152.158. IN A
;; AUTHORITY SECTION:
. 86400 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2022122801 1800 900 604800 86400
;; Query time: 5 msec
;; SERVER: 169.254.169.254#53(169.254.169.254)
;; WHEN: Thu Dec 29 01:45:39 UTC 2022
;; MSG SIZE rcvd: 119
dig 203.153.37.38
; <<>> DiG 9.16.33-Debian <<>> 203.153.37.38
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 32307
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;203.153.37.38. IN A
;; AUTHORITY SECTION:
. 86400 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2022122801 1800 900 604800 86400
;; Query time: 7 msec
;; SERVER: 169.254.169.254#53(169.254.169.254)
;; WHEN: Thu Dec 29 01:46:13 UTC 2022
;; MSG SIZE rcvd: 117

Same HTTP request passes with CURL_CLI but fails with LIBCURL

Am using libcurl to communicate with Amazon S3.
GET calls are success whereas PUT (for uploading files) calls are failing with 403.
PUT call with same headers when ran through CURL-CLI are success.
I have disabled certificate checking. [[ curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0) ]]
Using CURLOPT_DEBUGFUNCTION option as specified in https://curl.haxx.se/libcurl/c/CURLOPT_DEBUGFUNCTION.html, am able to capture the following log.
=> Send header, 0000000578 bytes (0x00000242)
0000: PUT /M2000/activity/8c51f2240f9fc1e7d2329a24210e30c9_200_0603202
0040: 0065517_fota_dlready HTTP/1.1
005f: Host: fota.test.nvtl.s3.amazonaws.com
0086: Accept: */*
0093: Authorization : AWS4-HMAC-SHA256 Credential=AKIAJ2ZI2YKOFDBS4UMQ
00d3: /20200603/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-co
0113: ntent-sha256;x-amz-date, Signature=61613a8357e45ac6c14067856f1e1
0153: 56449adcba20d3b8d295ed1c67a126dda0c
0178: x-amz-content-sha256:e2b646fdb491ec4be82661a9ea86ce3b08fe5fabfda
01b8: 2b9a9f8c3d9a783135837
01cf: x-amz-date:20200603T065517Z
01ec: content-length: 3394
0202: content-type: application/octet-stream
022a: Expect: 100-continue
0240:
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: ....P
<= Recv header, 0000000024 bytes (0x00000018)
0000: HTTP/1.1 403 Forbidden
<= Recv header, 0000000036 bytes (0x00000024)
0000: x-amz-request-id: BE05183350F1A797
<= Recv header, 0000000090 bytes (0x0000005a)
0000: x-amz-id-2: b6rrLoMPXn0Umvv8YzFiVL8CX27oszT0mQqWnjxaBtD49DoqsI6C
0040: NedmUoOebVAf3R96Q7c59tg=
<= Recv header, 0000000031 bytes (0x0000001f)
0000: Content-Type: application/xml
<= Recv header, 0000000028 bytes (0x0000001c)
0000: Transfer-Encoding: chunked
<= Recv header, 0000000037 bytes (0x00000025)
0000: Date: Wed, 03 Jun 2020 06:55:19 GMT
<= Recv header, 0000000019 bytes (0x00000013)
0000: Connection: close
<= Recv header, 0000000018 bytes (0x00000012)
0000: Server: AmazonS3
<= Recv header, 0000000002 bytes (0x00000002)
0000:
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: ....0
<= Recv data, 0000000254 bytes (0x000000fe)
0000: f3
0004: <?xml version="1.0" encoding="UTF-8"?>.<Error><Code>AccessDenied
0044: </Code><Message>Access Denied</Message><RequestId>BE05183350F1A7
0084: 97</RequestId><HostId>b6rrLoMPXn0Umvv8YzFiVL8CX27oszT0mQqWnjxaBt
00c4: D49DoqsI6CNedmUoOebVAf3R96Q7c59tg=</HostId></Error>
00f9: 0
00fc:
=> Send SSL data, 0000000005 bytes (0x00000005)
0000: ....0
== Info: TLSv1.2 (OUT), TLS alert, Client hello (1):
=> Send SSL data, 0000000002 bytes (0x00000002)
0000: ..
== Info: TLSv1.2 (OUT), TLS header, Certificate Status (22):
=> Send SSL data, 0000000005 bytes (0x00000005)
0000: .....
== Info: TLSv1.2 (OUT), TLS handshake, Client hello (1):
=> Send SSL data, 0000000512 bytes (0x00000200)
0000: .........;r..f..g.]..50.=.+.xh..M..Ia.....0.,.(.$.............k.
0040: j.i.h.9.8.7.6.........2...*.&.......=.5.../.+.'.#.............g.
0080: #.?.>.3.2.1.0.........E.D.C.B.1.-.).%.......<./...A.............
00c0: ............3...$."...fota.test.nvtl.s3.amazonaws.com...........
0100: ................................ ...............................
0140: ......3t.........http/1.1.......................................
0180: ................................................................
01c0: ................................................................
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: ....W
== Info: TLSv1.2 (IN), TLS handshake, Server hello (2):
<= Recv SSL data, 0000000087 bytes (0x00000057)
0000: ...S...Z..J....v...6.F........i3..j:2. ..ptS.oT..h...8..J_..:o..
0040: )..a...................
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: ....T
== Info: TLSv1.2 (IN), TLS handshake, Certificate (11):
<= Recv SSL data, 0000002900 bytes (0x00000b54)
0000: ...P..M...0...0...........-........r..8..0...*.H........0d1.0...
0040: U....US1.0...U....DigiCert Inc1.0...U....www.digicert.com1#0!..U
0080: ....DigiCert Baltimore CA-2 G20...191109000000Z..210312120000Z0l
00c0: 1.0...U....US1.0...U....Washington1.0...U....Seattle1.0...U....A
0100: mazon.com, Inc.1.0...U....*.s3.amazonaws.com0.."0...*.H.........
0140: ....0.........[...)X.+\.x....<1j.....p.y.c..hx..c=.#..e......t,.
0180: g.^c3......z...}~>.2.eC.;...Y.;h..e....A..y....$#.K.#.&...2O..fD
01c0: ..7k..1.R..3....+~...{}.XT.gp4.l.0$.V.)Z.o.6..?..^Oz,%s....~...~
0200: ........z.c..P...~7G..-....!6K.....\....}....F.7..H.T.z..#p;..F
0240: #Jr...)L..xa..........0..|0...U.#..0......(thFg.p%t..E[.}\D0...U
0280: ........&.d...\.......'...0/..U...(0&..*.s3.amazonaws.com..s3.am
02c0: azonaws.com0...U...........0...U.%..0...+.........+.......0....U
0300: ...z0x0:.8.6.4http://crl3.digicert.com/DigiCertBaltimoreCA-2G2.c
0340: rl0:.8.6.4http://crl4.digicert.com/DigiCertBaltimoreCA-2G2.crl0L
0380: ..U. .E0C07..`.H...l..0*0(..+.........https://www.digicert.com/C
03c0: PS0...g.....0y..+........m0k0$..+.....0...http://ocsp.digicert.c
0400: om0C..+.....0..7http://cacerts.digicert.com/DigiCertBaltimoreCA-
0440: 2G2.crt0...U.......0.0..}..+.....y......m...i.g.v.......X......g
0480: p.<5.......w.........nMv;0.....G0E.!....)Y.!i...a.......4.w....w
04c0: .e..y. 3..cv.....R]....A.......a_.......v.D.e......#....(.......
0500: 1.?.3........nMv;(.....G0E.!...B).+.S..[..^..mD..&2.c8.*....7. B
0540: ..#.....f.s..B.....#mA&.u..v.C..u.......q...#...{G8W...R....d6..
0580: .....nMv;5.....F0D. ..._A/^..Y..V.....oJ.#.k.8...fDj. A..^..o...
05c0: .......09O.....dEgX.K.0...*.H.................x".V...6....2#....
0600: ..*.6q........CS.....-...=..ep...V......M.C......#.$.y.k.F?.v.&(
0640: .<N....a.o......i../..Q8K+^.;k86.F.....rW..._..>...3.q.Tc...l..
0680: $...fx0.;...)[.......f_....0T*....g.....<u{........8/6....."...J
06c0: .38...# %.K.nt.5M...'m5Pk..wH.0...Z....-.c..g0..c0..K...........
0700: ...&..;'....0...*.H........0Z1.0...U....IE1.0...U....Baltimore1.
0740: 0...U....CyberTrust1"0 ..U....Baltimore CyberTrust Root0...15120
0780: 8120507Z..250510120000Z0d1.0...U....US1.0...U....DigiCert Inc1.0
07c0: ...U....www.digicert.com1#0!..U....DigiCert Baltimore CA-2 G20..
0800: "0...*.H.............0..............s....\.u.\ps...z...#....?.!.
0840: .M..-...1..k..].Q..^r..f.....A..&....i.S..O>..:C+.Y.....Y#rZg...
0880: U..+............%.E......t....eP...E../..6......R....6q+.......[
08c0: [9...................Z../r....g....J. ...r....OB.M'BM.....u.6`.&
0900: T...7....)Y ..%........................0...0...U.........(thFg.p
0940: %t..E[.}\D0...U.#..0.....Y0.GX....T6.{:..M.0...U.......0.......0
0980: ...U...........04..+........(0&0$..+.....0...http://ocsp.digicer
09c0: t.com0:..U...3010/.-.+.)http://crl3.digicert.com/Omniroot2025.cr
0a00: l0=..U. .60402..U. .0*0(..+.........https://www.digicert.com/CPS
0a40: 0...*.H............./.7f....U.)..P.....(.t..i;D0=..I.h6..0....IB
0a80: cFR.i..I...W.....u....3.b.CT.c...S.........ex.3...>.p....x.3...,
0ac0: X..#.mA...../...pk.:.{....K..o..)....T.T.... .i,./?P...W...s..$.
0b00: ...E....).f...l.O.+.LG....A....FwB......%K.PW...N.?.%.A...bmo...
0b40: ......).......T.KI..
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: ....M
== Info: TLSv1.2 (IN), TLS handshake, Server key exchange (12):
<= Recv SSL data, 0000000333 bytes (0x0000014d)
0000: ...I...A.>...,R..3R.ry.../WTDH..b...lcB`.RB<dPQJ..2..$M.`6.A....
0040: ..2e.C.......$W+.8.g.].....A[q...,G.......u;.....q....8..:......
0080: G ....F.-.o.b54...P-.o..:|H..!......5.....O.7..d..El7.;....\....
00c0: t......'.v&...#..Y*.y..yoN....3 >...?...vH...i.J....K.....ar.:..
0100: ..Ll.............(Z..mx.;.....BQ...7.b.[...|..z.;..d....*. ..]M/
0140: ./.........=.
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: .....
== Info: TLSv1.2 (IN), TLS handshake, Server finished (14):
<= Recv SSL data, 0000000004 bytes (0x00000004)
0000: ....
=> Send SSL data, 0000000005 bytes (0x00000005)
0000: ....F
== Info: TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
=> Send SSL data, 0000000070 bytes (0x00000046)
0000: ...BA.wm.Jg'... .......,y8%B.....1..........}X.5.~....oC...T.].6
0040: .Q\..8
=> Send SSL data, 0000000005 bytes (0x00000005)
0000: .....
== Info: TLSv1.2 (OUT), TLS change cipher, Client hello (1):
=> Send SSL data, 0000000001 bytes (0x00000001)
0000: .
=> Send SSL data, 0000000005 bytes (0x00000005)
0000: ....#
== Info: TLSv1.2 (OUT), TLS handshake, Finished (20):
=> Send SSL data, 0000000016 bytes (0x00000010)
0000: ......1.e.].....
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: .....
== Info: TLSv1.2 (IN), TLS change cipher, Client hello (1):
<= Recv SSL data, 0000000001 bytes (0x00000001)
0000: .
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: ....#
== Info: TLSv1.2 (IN), TLS handshake, Finished (20):
<= Recv SSL data, 0000000016 bytes (0x00000010)
0000: ....t.....%. 1..
=> Send SSL data, 0000000005 bytes (0x00000005)
0000: ....P
When i ran the same from CLI it is success.
# curl --insecure -v -X PUT https://fota.test.nvtl.s3.amazonaws.c
om/M2000/activity/8c51f2240f9fc1e7d2329a24210e30c9_200_06032020065517_fota_dlrea
dy -H 'Authorization: AWS4-HMAC-SHA256 Credential=AKIAJ2ZI2YKOFDBS4UMQ/20200603/
us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, S
ignature=61613a8357e45ac6c14067856f1e156449adcba20d3b8d295ed1c67a126dda0c' -H x-
amz-content-sha256:e2b646fdb491ec4be82661a9ea86ce3b08fe5fabfda2b9a9f8c3d9a783135
837 -H x-amz-date:20200603T065517Z -H 'content-length: 3394' -H 'content-type: a
pplication/octet-stream' -T /opt/nvtl/data/8c51f2240f9fc1e7d2329a24210e30c9_
200_06032020065517_fota_dlready
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
> PUT /M2000/activity/8c51f2240f9fc1e7d2329a24210e30c9_200_06032020065517_fota_dlready HTTP/1.1
> Host: fota.test.nvtl.s3.amazonaws.com
> User-Agent: curl/7.52.1
> Accept: */*
> Authorization: AWS4-HMAC-SHA256 Credential=AKIAJ2ZI2YKOFDBS4UMQ/20200603/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=61613a8357e45ac6c14067856f1e156449adcba20d3b8d295ed1c67a126dda0c
> x-amz-content-sha256:e2b646fdb491ec4be82661a9ea86ce3b08fe5fabfda2b9a9f8c3d9a783135837
> x-amz-date:20200603T065517Z
> content-length: 3394
> content-type: application/octet-stream
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< x-amz-id-2: xnaNU1mmmuynkWQQDA0OJsgKKAYBOCRbx6lpTkj54HGEicXWOMtnd3qb4ZbY6vmogULiq7vOFcA=
< x-amz-request-id: BAE5F92ED54058EC
< Date: Wed, 03 Jun 2020 06:57:53 GMT
< ETag: "595f51bb7d2cc4c5c3f30b3bc3d350c3"
< Content-Length: 0
< Server: AmazonS3
<
Could any give hints what might actually be wrong here.
Is there any other method/way where we can have more verbose ouput regards the error.
Note:
HEADERS that are passed to both the requests (libcurl and curl_cli) are same.
Certificate check is disabled in both.
Response after adding user-agent.
0000: .....
=> Send header, 0000000609 bytes (0x00000261)
0000: PUT /M2000/activity/8c51f2240f9fc1e7d2329a24210e30c9_200_0603202
0040: 0142553_fota_dlready HTTP/1.1
005f: Host: fota.test.nvtl.s3.amazonaws.com
0086: User-Agent: libcurl-agent/1.0
00a5: Accept: */*
00b2: Authorization : AWS4-HMAC-SHA256 Credential=AKIAJ2ZI2YKOFDBS4UMQ
00f2: /20200603/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-co
0132: ntent-sha256;x-amz-date, Signature=9a72f22ce185e48c023a8d3314004
0172: ac2a76b91ccd56b208d769e27bafe7b1446
0197: x-amz-content-sha256:83a7ba5f8f0c829537965116840aad257f6b71e7899
01d7: bafc4c93542b681c9454a
01ee: x-amz-date:20200603T142554Z
020b: content-length: 3394
0221: content-type: application/octet-stream
0249: Expect: 100-continue
025f:
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: ....P
<= Recv header, 0000000024 bytes (0x00000018)
0000: HTTP/1.1 403 Forbidden
<= Recv header, 0000000036 bytes (0x00000024)
0000: x-amz-request-id: AD607FF3FA529447
<= Recv header, 0000000090 bytes (0x0000005a)
0000: x-amz-id-2: w0ZHrVJ9R8uslrB6kf9KLLRtshjksK9cLxeovH53GXL4uRoK17U6
0040: MJaWuhiGhZMKdIphBWkY+mE=
<= Recv header, 0000000031 bytes (0x0000001f)
0000: Content-Type: application/xml
<= Recv header, 0000000028 bytes (0x0000001c)
0000: Transfer-Encoding: chunked
<= Recv header, 0000000037 bytes (0x00000025)
0000: Date: Wed, 03 Jun 2020 14:25:57 GMT
<= Recv header, 0000000019 bytes (0x00000013)
0000: Connection: close
<= Recv header, 0000000018 bytes (0x00000012)
0000: Server: AmazonS3
<= Recv header, 0000000002 bytes (0x00000002)
0000:
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: ....0
<= Recv data, 0000000254 bytes (0x000000fe)
0000: f3
0004: <?xml version="1.0" encoding="UTF-8"?>.<Error><Code>AccessDenied
0044: </Code><Message>Access Denied</Message><RequestId>AD607FF3FA5294
0084: 47</RequestId><HostId>w0ZHrVJ9R8uslrB6kf9KLLRtshjksK9cLxeovH53GX
00c4: L4uRoK17U6MJaWuhiGhZMKdIphBWkY+mE=</HostId></Error>
UPDATED
Figured out the issue.
Issue seems to be with spacing.
"Authorization : AWS4-XXX" gives 403 error
whereas
"Authorization: AWS4-XXX" is success.
Thanks,
Trinadh
best guess: fota.test.nvtl.s3.amazonaws.com blocks requests without any User-Agent. several websites do this, for example Wikipedia.org does the same thing (not sure why),
curl the cli program has a default user-agent, it looks like User-Agent: curl/7.52.1, but libcurl doesn't have any default user-agent, you can set one with the CURLOPT_USERAGENT option.

Operation timed out after 10000 milliseconds with 11584 out of 1526223 bytes received

I am trying to get about 1.4 to 2 MB of JSON data from a server using the curl library in C.
I have increased the Buffer size, connection timeout
curl_easy_setopt(conn->easy, CURLOPT_CONNECTTIMEOUT, 10);
curl_easy_setopt(conn->easy, CURLOPT_BUFFERSIZE, 524288);
#ifdef CURL_MAX_WRITE_SIZE
#undef CURL_MAX_WRITE_SIZE
#define CURL_MAX_WRITE_SIZE 524288
#endif
I did take a capture on the client I am running and the packets are not received completely.
I also tried the url from a browser and the browser is able to get the data and display.
Any help on this is appreciated.
Thanks
So basically my code is based on the example https://curl.haxx.se/libcurl/c/crawler.html
The difference here is that the timer_cb is called frequently and the easy_handle is released only if there is something to read.
static void timer_cb(int fd, short kind, void * userp)
{
GlobalInfo * g = (GlobalInfo *)userp;
CURLMcode rc;
(void)fd;
(void)kind;
rc = curl_multi_socket_action(g->multi,
CURL_SOCKET_TIMEOUT, 0, &g->still_running);
mcode_or_die("timer_cb: curl_multi_socket_action", rc);
check_multi_info(g);
}
I read the response here in this api and new_conn_init will create/malloc a new easy handle and call the curl_multi_add_handle.
static void check_multi_info(GlobalInfo * g)
{
char * eff_url;
CURLMsg * msg;
int msgs_left;
ConnInfo * conn = NULL;
CURL * easy;
.....
while ((msg = curl_multi_info_read(g->multi, &msgs_left)))
{
if (msg->msg == CURLMSG_DONE)
{
easy = msg->easy_handle;
res = msg->data.result;
do
{
rc = ERROR;
conn = NULL;
contentType = NULL;
iValidContent = 1;
..........
}
while (0);
u8 ret = 0;
curl_multi_remove_handle(g->multi, easy);
curl_easy_cleanup(easy);
if (conn)
new_conn_init(&(conn->reqInfo), g, 1);
}
new_conn_init ()
{
conn->easy = curl_easy_init();
curl_easy_setopt(conn->easy, CURLOPT_URL, conn->url);
curl_easy_setopt(conn->easy, CURLOPT_BUFFERSIZE, 524288);
curl_easy_setopt(conn->easy, CURLOPT_WRITEFUNCTION, write_cb);
curl_easy_setopt(conn->easy, CURLOPT_WRITEDATA, &(conn->respData));
// curl_easy_setopt(conn->easy, CURLOPT_HEADERDATA, &(conn->headerData));
// curl_easy_setopt(conn->easy, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(conn->easy, CURLOPT_VERBOSE, (long)shmHttpClientConfig->debugLibCurl);
curl_easy_setopt(conn->easy, CURLOPT_FORBID_REUSE, 1L);
curl_easy_setopt(conn->easy, CURLOPT_ERRORBUFFER, conn->error);
curl_easy_setopt(conn->easy, CURLOPT_PRIVATE, conn);
curl_easy_setopt(conn->easy, CURLOPT_PROTOCOLS, CURLPROTO_HTTP | CURLPROTO_HTTPS);
curl_easy_setopt(conn->easy, CURLOPT_DNS_CACHE_TIMEOUT, -1);
curl_easy_setopt(conn->easy, CURLOPT_NOPROGRESS, 1);
curl_easy_setopt(conn->easy, CURLOPT_CONNECTTIMEOUT, 10);
rc = curl_multi_add_handle(g->multi, conn->easy);
}
I did some debugging and looks like the connections are getting closed quite late in if you can see the op of the curl debug. I am sending url queries every 20 seconds.
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#1)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:37:53 GMT
<
* Found bundle for host 12.0.0.25: 0x9f8d20
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#2)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:38:13 GMT
<
* Found bundle for host 12.0.0.25: 0x9f8d20
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#3)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:38:33 GMT
<
* Found bundle for host 12.0.0.25: 0x9f8d20
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#4)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
* Operation timed out after 10000 milliseconds with 160728 out of 1526223 bytes received
* Closing connection 1
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:38:53 GMT
<
* Closing connection 4
* Found bundle for host 12.0.0.25: 0x9f8d20
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Operation timed out after 10001 milliseconds with 11584 out of 1526223 bytes received
* Closing connection 2
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#5)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:39:13 GMT
<
* Closing connection 5
* Operation timed out after 10001 milliseconds with 845632 out of 1526223 bytes received
* Closing connection 3
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#6)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:43:33 GMT
<
* Found bundle for host 12.0.0.25: 0x9f31f0
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#7)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< C
[root#BENU TEMP/MEG-1] ~# cat /opt/benu-data/admin/benu_libcurl_op.txt | more
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#1)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:37:53 GMT
<
* Found bundle for host 12.0.0.25: 0x9f8d20
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#2)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:38:13 GMT
<
* Found bundle for host 12.0.0.25: 0x9f8d20
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#3)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:38:33 GMT
<
* Found bundle for host 12.0.0.25: 0x9f8d20
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#4)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
* Operation timed out after 10000 milliseconds with 160728 out of 1526223 bytes received
* Closing connection 1
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:38:53 GMT
<
* Closing connection 4
* Found bundle for host 12.0.0.25: 0x9f8d20
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Operation timed out after 10001 milliseconds with 11584 out of 1526223 bytes received
* Closing connection 2
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#5)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*

S3 uploads thru angular

Using this plugin
Sooooo ive got a problem
So, i am generating a signed S3 url on my server, and attempting to upload using this plugin. I can generate the url and upload a file using
How its worked before
shell
curl -T file.jpg http://bla.s3.amazon.com/blahbahbla
How im doing it
When I attempt to upload the file from the browser like this
```javascript
$upload.http({
url: decodeURIComponent(data.signed_url),
method: 'PUT',
data: file,
headers: {
'Content-Type':file.type,
}
}).progress(function(evt) {
console.log('percent: ' + parseInt(100.0 * evt.loaded / evt.total));
}).success(function(data, status, headers, config) {
console.log(data);
}).error(function(data, status, headers, config) {
console.warn(data, status);
});
```
Response
I get a response like this
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><StringToSignBytes>50 55 54 0a 0a 69 6d 61 67 65 2f 6a 70 65 67 0a 31 34 35 32 38 37 39 36 38 36 0a 78 2d 61 6d 7a 2d 61 63 6c 3a 70 75 62 6c 69 63 2d 72 65 61 64 0a 2f 62 67 70 72 6f 6f 66 2f 66 61 63 65 62 6f 6f 6b 3a 31 30 30 30 30 33 34 37 31 35 37 34 31 30 35 2f 33 65 31 62 36 64 32 39 2d 62 66 34 31 2d 34 37 62 31 2d 61 35 31 33 2d 30 38 38 61 39 63 30 32 61 39 62 63 2e 6a 70 67</StringToSignBytes><RequestId>CE628CB59F561A3D</RequestId><HostId>u5iLkvYLE0hRj30IU+qMS+XiwlwY/VXt8ZACxyot2KDwnLw5S2HL/pGgk4F/nHfr</HostId><SignatureProvided>hJg fYN2h6qz9jJATt4Zp A C94=</SignatureProvided><StringToSign>PUT
image/jpeg
1452879686
x-amz-acl:public-read
/kirkstrobeck/facebook:100003471574105/3e1b6d29-bf41-47b1-a513-088a9c02a9bc.jpg</StringToSign><AWSAccessKeyId>MY_ACCESS_KEY</AWSAccessKeyId></Error>
Here are the headers for both requests
Req Headers
PUT /facebook:100003471574105/3e1b6d29-bf41-47b1-a513-088a9c02a9bc.jpg?AWSAccessKeyId=MY_ACCESS_KEY&Expires=1452879686&Signature=hJg+fYN2h6qz9jJATt4Zp+A+C94=&x-amz-acl=public-read HTTP/1.1
Host: kirkstrobeck.s3.amazonaws.com
Connection: keep-alive
Content-Length: 362867
Cache-Control: no-cache
Pragma: no-cache
Accept: application/json, text/plain, */*
Origin: http://localhost:5005
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
Content-Type: image/jpeg
Referer: http://localhost:5005/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Res Headers
HTTP/1.1 403 Forbidden
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, PUT, DELETE
Access-Control-Max-Age: 10000
Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
x-amz-request-id: CE628CB59F561A3D
x-amz-id-2: u5iLkvYLE0hRj30IU+qMS+XiwlwY/VXt8ZACxyot2KDwnLw5S2HL/pGgk4F/nHfr
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Wed, 15 Jan 2014 05:41:26 GMT
Connection: close
Server: AmazonS3
This is how we are generating the signed putObject url in NodeJS
var uuid = require('node-uuid');
var AWS = require('aws-sdk');
var s3 = new AWS.S3({
apiVersion: '2014-01-10',
accessKeyId: 'AWS_ACCESS_KEY',
secretAccessKey: 'AWS_SEC_ACCESS_KEY'
});
module.exports = function (req, res, next) {
var exten = req.param('ext');
var expire = req.param('expire');
var key = uuid.v4() + exten;
var params = {
Bucket: 'BUCKET_NAME',
Key: key,
Expires: 900,
ACL: 'public-read'
};
s3.getSignedUrl('putObject', params, function(err, url){
// .. handle callback
});
};
This example can maybe help:
https://github.com/bookingbricks/file-upload-example
Using: Node, aws-sdk-js, jQuery-file-upluad (blueimp)
Server:
var AWS = require('aws-sdk');
AWS.config.update({accessKeyId: AWS_ACCESS_KEY, secretAccessKey: AWS_SECRET_KEY});
AWS.config.region = 'eu-west-1';
app.post('/s', function (req, res) {
var s3 = new AWS.S3();
var params = {Bucket: 'BUCKETNAME', Key: req.body.name, ContentType: req.body.type};
s3.getSignedUrl('putObject', params, function(err, url) {
if(err) console.log(err);
res.json({url: url});
});
});
Client:
$.ajax({
url: '/s',
type: 'POST',
data: {name: file.name, size: file.size, type:file.type},
}).success(function(res){
$.ajax({
url: res.url,
type: 'PUT',
data: file,
processData: false,
contentType: file.type,
}).success(function(res){
console.log('Done');
});
Looking through http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html, it looks like your's using the query string method of authentication, so your request must be of the form
GET /my-bucket/foo
?Signature=<urlencode(base64(hmac-sha1(VERB + "\n" + CONTENT-MD5 + "\n" + CONTENT-TYPE + "\n" + Expires + "\n" + CanonicalizedAmzHeaders + "\n" + CanonicalizedResource)))>
&Expires=<seconds since epoch>
&AWSAccessKeyId=<aws-id>
Addressing each of these, comparing how you generate the signed request with the request itself
VERB: According to http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/frames.html using 'putObject' is the correct way to request signing a PUT request. Your request headers seem to use a PUT request.
CONTENT-MD5: You haven't specified a ContentMD5 option when signing your request on the server. According to the docs this is optional, but an empty string + new line must be included in the string for which the signature is calculated. So you might need to add an empty ContentMD5 parameter to the params object passed to getSignedUrl
CONTENT-TYPE: ContentType is also optional. However, I note you are sending the ContentType header in the PUT request, but not specifying it when generating the signature. So the S3 may use this to generate the signature to check against, and fail the request. I suggest either setting the ContentType when generating the signature to what it will be in the browser, or make it empty in both cases (probably passing an empty string to the call to getSignedUrl
Expires: From the docs:
The Expires field is given as the number of seconds since epoch time
I'm not sure how the Node API uses the Expires option, but setting it as 900 might mean it has already expired. The url you've written has it output as 1452879686, which is on January 20th 2014. Given your question was written on January 15th/17th, I'm not sure how this matches up. I would check exactly what you need to pass as Expires, and try passing a full date to the call to 'getSignedUrl'.
CanonicalizedAmzHeaders: From what I can see, the call to getSignedUrl added x-amz-acl=public-read, so I can only assume it used it when generating the url.
CanonicalizedResource: Again I can only assume that the call to getSignedUrl is using this correctly.
AWSAccessKeyId: It's in the URL generated, so I assume getSignedUrl used it correctly.
In summary:
ContentMD5: Set it to the empty string in the params object
Make sure ContentType is specified when generating the signature, and make it the same as for the PUT request (or empty or image/jpeg)
Set a full date as the Expires option to the params object getSignedUrl, and check the generated URL that this is reasonable.
S3 has very specific requirements for uploading files via POST. There are specific input fields that require values that appear to be missing from your request.
Check out the docs for the required fields: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingHTTPPOST.html

Why actual bytes read from server's response less than the Content-Length?

I'm writing a proxy in C in Linux 3.0 kernel. The proxy forwards the request from browser to the target server and read response from server.
This is one request:
GET http://www.yongchuan.org/ HTTP/1.1
Host: www.yongchuan.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Proxy-Connection: keep-alive
This is one response the proxy read:
HTTP/1.1 200 OK
Content-Length: 3170
Content-Type: text/html
Content-Location: http://www.yongchuan.org/index.html
Last-Modified: Fri, 22 Jul 2011 01:28:50 GMT
Accept-Ranges: bytes
ETag: "72d9d7b5e48cc1:1a73"
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Date: Sun, 04 Mar 2012 16:26:05 GMT
<html>
<head>
..............
<tr>
<td width="43%" ></td>
<td width="46%"></td>
</tr>
<tr>
<td width="43%"><img src="images/triangle.gif"> ��ѧ�ڿγ̣�Ӧ��ͼ��</td>
<td width="46%" ><img src=
The problem is, the actual size of bytes the proxy read is 2880 and the data ends up suddenly, much less than Context length: 3170.
My code is as below:
241 int readlen;
242 char buffer[128 * 4096];
243 do {
244 readlen = read(servfd, buffer, 128 * 4096);
245 printf("readlen:%d\n", readlen);
246 if( readlen < 0 ) {
247 perror("read() from server failed");
248 }
249 printf("read content:\n%s", buffer2);
250 write(cliefd, buffer, readlen);
251 } while( readlen == 128 * 4096 );
Here servfd is the socket to server, cliefd is the socket to client.
What's the possible reason? Thanks!
while( readlen == 128 * 4096 );
The while condition is wrong. You should be checking readlen > 0 instead.
There's no requirement that read only hands you completely filled buffers. In the worst case (highly unlikely with current TCP stacks), it could hand you the bytes one by one.
At any rate, if the server doesn't close the connection after delivering the output you must keep reading until you get the Content-Length you were promised.

Resources