Passing JSON request starting with the character [ gives the error: invalid request format with GET operation [duplicate] - request

This question already has an answer here:
Karate DSL: Getting connection timeout error
(1 answer)
Closed 2 years ago.
When I pass a JSON request starting with [, I get the error which says: invalid request format with GET operation.
Here is my request. The same works fine in PostMan.
[
{
"equipmentNumber": "76576",
"systemIdentifier": "00045F063547",
"serialNumber": "00098",
"materialNumber": "786786"
}
]

Your question is incomplete. Anyway I'll give it a shot. Here is a test that works perfectly fine in Karate, cut and paste it into a new Scenario and see for yourself.
* url 'https://httpbin.org/post'
* request
"""
[
{
"equipmentNumber": "76576",
"systemIdentifier": "00045F063547",
"serialNumber": "00098",
"materialNumber": "786786"
}
]
"""
* method post
Which results in this request:
1 > POST https://httpbin.org/post
1 > Accept-Encoding: gzip,deflate
1 > Connection: Keep-Alive
1 > Content-Length: 112
1 > Content-Type: application/json; charset=UTF-8
1 > Host: httpbin.org
1 > User-Agent: Apache-HttpClient/4.5.5 (Java/1.8.0_231)
[{"equipmentNumber":"76576","systemIdentifier":"00045F063547","serialNumber":"00098","materialNumber":"786786"}]
So if you are still stuck, follow this process - else no one can help you with the lack of information in your question: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue

Related

CURL and my little http client give different results

I have this little C program implementing a HTTP client. Basically I used this reference code with slight modifications, mainly for debugging.
My problem is, that the content I get back using this is not quite the same as I get when I use curl for example.
Basically what the code does is:
Construct a HTTP headers for a GET request
Establishes TCP connection with the server
Sends the HTTP headers
Receives back a response
The read part looks like this:
while((recived_len = recv(sock, BUF, BUFSIZ-1, 0)) > 0)
{
BUF[recived_len] = '\0';
response = (char*)realloc(response, strlen(response) + strlen(BUF) + 1);
sprintf(response, "%s%s", response, BUF);
}
In particular, I always get in the begining of the body part 4 bytes which I don't understand where they come from:
HTTP/1.1 200 OK
Date: Tue, 20 Apr 2021 09:17:54 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Server: gws
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Set-Cookie: 1P_JAR=2021-04-20-09; expires=Thu, 20-May-2021 09:17:54 GMT; path=/; domain=.google.com; Secure
Set-Cookie: NID=213=ts4T4alAR2ODEf4vlWrmoZj-cjJcAXACbbxf64Zte4lEbuvUgik6TUgKkdY5OVHDQuTWM59DekV3ayNXDl08TcETU-WwztPVmMFz9BXegk93QFyno5WCS9fJDGq3sSrbFsFjxPOPCLTOx-b8H3a4Ed_HbI9lXBocjGu07ULo8PY; expires=Wed, 20-Oct-2021 09:17:54 GMT; path=/; domain=.google.com; HttpOnly
Accept-Ranges: none
Vary: Accept-Encoding
Connection: close
Transfer-Encoding: chunked
5225 // what is this?
<!doctype html><html dir="rtl" itemscope="" itemtype="http://schema.org/WebPage" lang="iw"><head><meta content="text/html; charset=UTF-8" http-equiv="Content-Type"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop="image"><title>Google</title><script nonce="ks4dIW3TspucDhFX4XRWZA==">(function(){window.google={kEI:'Qpx-YKSmGIeSaPf3mJgP',
The headers part looks fine, so I don't think it's a problem with the resquet's headers, and also the beggining of the content (except from these 4 bytes) looks fine. However, down the stream things get messier and it gets quite different from the curl response, which seems much more reasonable.
Can someone tell me why this happening?
If I try to send a GET request to
EDIT
OK thanks I understand this 4 bytes are the length of the Chunk, since the Transfer-Encoding is of type chunk. From what I've read here about it, chunks should be sent one after another, with each one preceded by (or followed by, depending from where you look) \r\n<chunk_length>\r\n
Basically what I'm trying to do is implement a MP3 player, so I want to be able to read a stream of MP3 from a streaming server and play it.
When I use curl to get the stream, I can play it. I can even see that the data is sent in chunks as described above.
However, when I use my program, the data I'm getting is for some reason not structured properly..
My final goal in all this is to use ESP8266 module as a Radio player. I try to understand all the moving parts that should be involved in order to make this work. First I want to implement this on my PC before I move to working on the ESP8266
EDIT #2
Thanks to #Emanuel P I noticed that I get back from the server content type of text/html; charset=ISO-8859-1, so I added a Content-Type: */* just as curl does, and I do get back a Content-Type: audio/mpeg, as expected, but it still doesn't work for some reason - the MP3 player doesn't read the file well:
$ ./mp3player.out stream.mp3
Note: Illegal Audio-MPEG-Header 0x04e248be at offset 2258.
Note: Trying to resync...
Note: Skipped 87 bytes in input.
Warning: Big change from first (MPEG version, layer, rate). Frankenstein stream?

Varnish ban is added but old object is returned

I'm using varnish in front of a tile server to cache mapbox tiles. To remove old tiles, I intended to use bans to effectively remove a large number of cached tiles. My problem is that varnish still uses the cached objects (at least the age in the response indicates this) and doesn't contact the backend.
I'm first requesting http://varnish/5/3/4.pbf, then adding a ban with curl -X BAN -H 'X-Purge-Regex: 5/3/4.pbf' varnish or alternatively varnishadm and then ban obj.http.url ~ 5/3/4.pbf and afterwards requesting http://varnish/5/3/4.pbf again.
In the beginning my ban list is empty:
Present bans:
1610117471.434488 1 C
The ban is added succesfully with curl -X BAN -H 'X-Purge-Regex: 5/3/4.pbf' varnish
<!DOCTYPE html>
<html>
<head>
<title>200 Ban added</title>
</head>
<body>
<h1>Error 200 Ban added</h1>
<p>Ban added</p>
<h3>Guru Meditation:</h3>
<p>XID: 8</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>
and shows up in the ban list
Present bans:
1610117369.028870 0 - obj.http.url ~ 5/3/4.pbf
1610117307.220739 1 C
After requesting http://varnish/5/3/4.pbf again, the ban list indicates that the ban was used
Present bans:
1610117471.434488 1 - obj.http.url ~ 5/3/4.pbf
but the age of the response is not 0, because it's still the object from the first request.
After a short time the ban is removed:
Present bans:
1610117471.434488 1 C
My vcl_recv looks like this but the error is probably somewhere else as it also doesn't work with varnishadm:
sub vcl_recv {
unset req.http.cookie;
# Allowing PURGE from localhost
if (req.method == "BAN"||req.method == "PURGE") {
if (!client.ip ~ purge) {
return(synth(405,"Not allowed."));
}
if (req.method == "BAN") {
ban("obj.http.url ~ " + req.http.X-Purge-Regex);
# Throw a synthetic page so the
# request won't go to the backend.
return(synth(200, "Ban added"));
}
if (req.method == "PURGE") {
return (purge);
}
}
}
I also tried to use the vcl_purge from https://stackoverflow.com/a/61507014 but this doesn't seem to help for bans (?).
I'm using the X-Purge-Regex header to not worry about having to escape special characters like in https://stackoverflow.com/a/38526921 but just a ban like obj.http.url ~ 0 doesn't work.
I'm using varnish 6.5 with vcl 4.0.
Ban request
* << Request >> 54
- Begin req 53 rxreq
- Timestamp Start: 1610121483.345437 0.000000 0.000000
- Timestamp Req: 1610121483.345437 0.000000 0.000000
- VCL_use boot
- ReqStart 192.168.48.2 50882 http
- ReqMethod BAN
- ReqURL /
- ReqProtocol HTTP/1.1
- ReqHeader Host: varnish-volatile
- ReqHeader User-Agent: curl/7.64.0
- ReqHeader Accept: */*
- ReqHeader X-Purge-Regex: 0
- ReqHeader X-Forwarded-For: 192.168.48.2
- VCL_call RECV
- VCL_acl MATCH purge "importer"
- VCL_return synth
- VCL_call HASH
- VCL_return lookup
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason Ban added
- RespHeader Date: Fri, 08 Jan 2021 15:58:03 GMT
- RespHeader Server: Varnish
- RespHeader X-Varnish: 54
- VCL_call SYNTH
- RespHeader Content-Type: text/html; charset=utf-8
- RespHeader Retry-After: 5
- VCL_return deliver
- Timestamp Process: 1610121483.347281 0.001844 0.001844
- RespHeader Content-Length: 246
- Storage malloc Transient
- Filters
- RespHeader Accept-Ranges: bytes
- RespHeader Connection: keep-alive
- Timestamp Resp: 1610121483.347557 0.002120 0.000276
- ReqAcct 98 0 98 218 246 464
- End
GET after adding ban
* << Request >> 32806
- Begin req 32805 rxreq
- Timestamp Start: 1610121552.733872 0.000000 0.000000
- Timestamp Req: 1610121552.733872 0.000000 0.000000
- VCL_use boot
- ReqStart 192.168.48.1 55176 http
- ReqMethod GET
- ReqURL /public.snow_db/0/0/0.pbf
- ReqProtocol HTTP/1.1
- ReqHeader Host: localhost:8090
- ReqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko/20100101 Firefox/84.0
- ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
- ReqHeader Accept-Language: en-US,en;q=0.5
- ReqHeader Accept-Encoding: gzip, deflate
- ReqHeader DNT: 1
- ReqHeader Connection: keep-alive
- ReqHeader Upgrade-Insecure-Requests: 1
- ReqHeader Pragma: no-cache
- ReqHeader Cache-Control: no-cache
- ReqHeader X-Forwarded-For: 192.168.48.1
- VCL_call RECV
- ReqUnset Host: localhost:8090
- ReqHeader host: localhost:8090
- VCL_return hash
- ReqUnset Accept-Encoding: gzip, deflate
- ReqHeader Accept-Encoding: gzip
- VCL_call HASH
- VCL_return lookup
- Hit 28 601789.331504 10.000000 0.000000
- VCL_call HIT
- VCL_return deliver
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader content-encoding: gzip
- RespHeader content-type: application/x-protobuf
- RespHeader date: Fri, 08 Jan 2021 15:09:02 GMT
- RespHeader Vary: Accept-Encoding
- RespHeader X-Varnish: 32806 28
- RespHeader Age: 3010
- RespHeader Via: 1.1 varnish (Varnish/6.5)
- VCL_call DELIVER
- VCL_return deliver
- Timestamp Process: 1610121552.734070 0.000197 0.000197
- Filters
- RespHeader Accept-Ranges: bytes
- RespHeader Content-Length: 295
- RespHeader Connection: keep-alive
- Timestamp Resp: 1610121552.734217 0.000345 0.000147
- ReqAcct 414 0 414 272 295 567
- End
Reproducing the issue
To reproduce the bug:
git clone https://github.com/Baschdl/varnish-ban-setup.git && cd varnish-ban-setup
docker-compose up
Open http://localhost:8092/5/3/1.pbf
docker-compose exec varnish varnishadm ban obj.http.url ~ pbf
Open http://localhost:8092/5/3/1.pbf again and you will get the old object
The obj.http.url ~ 5/3/4.pbf ban that you are issuing, is matching a url response header.
Remember: the URL is a request header, not a response header. No reason to panic, what you're doing makes perfect sense, and is related to the scope of the so-called ban lurker.
Ban lurker
The ban lurker is a thread that asynchronously processes bans on the ban list and matches objects to the bans in order to remove patterns of objects from the cache.
The ban lurker doesn't operate within a request scope, but only is aware of the object scope.
In order to successfully match request information, request context can be added as a response header. And that's what you're doing via obj.http.url
So why doesn't the ban work?
The reason why your ban isn't working, is because you didn't set obj.http.url in your VCL file. As a result, the ban lurker cannot match any objects to it.
How to fix the issue
The solution is simple: set the missing headers in the backend response context, as illustrated below:
sub vcl_backend_response {
set beresp.http.url = bereq.url;
set beresp.http.host = bereq.http.host;
}
When the backend responds, and right before the object gets stored in cache, we can set the missing headers.
After that, the ban lurker will be able to match the ban expression to the right objects, and remove them from cache.
Don't forget the objects aren't immediately matched: they are only removed when they reach the ban_lurker_age, which is set to 1 minute by default.

AngularJs $http.post() request is not working properly

AngularJs $http.post() request is not working properly
I want to store one task to my db. If amount of data in assignedMember is more than 175 it will not send got 404 error but If amount of data in assignedMember is less than 175 it will send success and store my db. Any idea about this. I dont what wrong with me. Please help me thank you
This is my json data
$scope.task=
{
"title": "My Title",
"description": "My Description",
"assignedMember": [
{
"userId": "51b701dae4b0dd92df2c32d1",
"status": "ASSIGNED"
},
{
"userId": "52de0811e4b04615ce7ed6bd",
"status": "ASSIGNED"
},
{
"userId": "559f8e97e4b0a5cdcd66bb76",
"status": "ASSIGNED"
},
.
.
.
.
.
.etc upto 500 data
]
}
This is my post request api
var responsePromise = $http.post("api/tasks",$scope.task);
responsePromise.success(function(data, status, headers, config) {
alert("Data created successfully");
});
responsePromise.error(function(data, status, headers, config) {
alert("Error")
});
If assigned member size is more than 175 or Content Length in browser is greater than 24580 when i send this json i got 404 error
If assigned member size is less than 175 or Content Length in browser is lesser than 10080 when i send this json it will success
If i getting 404 error my browser console is like this
Request header
-------------
Host: localhost
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:45.0) Gecko/20100101 Firefox/45.0
Accept: application/json, text/plain, /
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/json;charset=utf-8
Referer: http://localhost/login.do
Content-Length: 24580
Response header
--------------
Connection: close
Content-Encoding: gzip
Content-Type: text/html
Date: Thu, 15 Dec 2016 14:21:56 GMT
Server: nginx/1.10.1
Transfer-Encoding: chunked
Is it any restriction in my nginx server? Please help me
Post request have no restriction rit? and get request is limited to 2048KB
Actually I am sending via post so what problem i am facing?
Connection: close, I think your server have not accept large count of data.
nginx "fails fast" when the client informs it that it's going to send a body larger than the client_max_body_size by sending a 413 response and closing the connection.
Most clients don't read responses until the entire request body is sent. Because nginx closes the connection, the client sends data to the closed socket, causing a TCP RST.
If your HTTP client supports it, the best way to handle this is to send an Expect: 100-Continue header. Nginx supports this correctly as of 1.2.7, and will reply with a 413 Request Entity Too Large response rather than 100 Continue if Content-Length exceeds the maximum body size.
Referred from

Jersey Server Sent Events not working with Accept-Encoding gzip

I am using Jersey 2.17 for SSE - I have the demo working per the docs: https://jersey.java.net/documentation/latest/sse.html.
In my tests, it works fine with simple curl requests (curl --verbose "localhost:9000/api/broadcast"), but if I add -H 'Accept-Encoding: gzip, deflate' (to replicate my javascript code) it doesn't work (eventually times out with 'transfer closed with outstanding read data remaining'). I'm guessing that jersey says its GZIPed the response, but really it hasn't.
My real problem is that the AngularJS javascript code always adds this Accept-Encoding... header.
I'm looking for a solution that will either
Prevent Accept-Encoding: gzip, deflate' from being added as Request Header when initialising the EventSource() object
Setup Jersey to correctly handle/gzip the events sent to the client.
Update - I've noticed that if I close the EventOutput object in Java code that my events are all sent.
This is the console output when using curl:
curl --verbose "localhost:9000/api/broadcast" -H 'Accept-Encoding: gzip, deflate'
* Hostname was NOT found in DNS cache
* Trying ::1...
* connect to ::1 port 9000 failed: Connection refused
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 9000 (#0)
> GET /api/broadcast HTTP/1.1
> User-Agent: curl/7.37.1
> Host: localhost:9000
> Accept: */*
> Accept-Encoding: deflate, gzip
>
< HTTP/1.1 200 OK
* Server Apache-Coyote/1.1 is not blacklisted
< server: Apache-Coyote/1.1
< vary: Accept-Encoding
< content-encoding: deflate
< content-type: text/event-stream
< transfer-encoding: chunked
< date: Tue, 07 Apr 2015 20:35:42 GMT
< connection: close
<
* transfer closed with outstanding read data remaining
* Closing connection 0
curl: (18) transfer closed with outstanding read data remaining
For reference, my js code in my AngularJS app is:
$scope.sseSource = new EventSource('/api/ptt/1/status');
console.log( $scope.sseSource);
$scope.sseSource.onerror = function (event) {
console.log("onerror");
}
$scope.sseSource.onopen = function (event) {
console.log("onopen");
}
$scope.sseSource.onmessage = function (event) {
console.log("onmessage");
}

Arduino Ethercard - return content of website

I am trying to access a website, and then return whatever it outputs in the body -> eg. "Success" or "Failed".
When I try with my code, I am getting the following back.
<<< REQ >>>
HTTP/1.1 200 OK
Date: Sat, 30 Aug 2014 17:36:31 GMT
Content-Type: text/html
Connection: close
Set-Cookie: __cfduid=d8a4fc3c84849b6786c6ca890b92e2cc01409420191023; expires=Mon, 23-Dec-2019 23:50:00 GMT; path=/; domain=.japseyz.com; HttpOnly
Vary: Accept-Encoding
X-Powered-By: PHP/5.3.28
Server.
My code is: http://pastebin.com/WwWbnLNn
If all you want to know is whether the HTTP transaction succeeded or failed, then you need to examine the HTTP Response code... which is in the first line of the response. In your example it is "200"... the human readable interpretation of it is "OK".
Here is a link to most of the HTTP 1.1 response codes: w3.org-rfc2616 RespCodes
Your question indicated you wanted to extract this information from the "body"...
... but that information is not located in the "body", it is in the first response
header, as described above.
have you tried ethercard samples? there is a webclient sample, in which you can find procedure called CALLBACK - in that procedure you can process data stored in buf variable.
in your case you need to look for first empty line, which tells you that headers has been sent and page content(what php writes to the page i.e.) follows.
how familiar are you at pointers? how deep you do need to process the page output? i.e. OK or ERROR is enough, or you do need to pass same parameters back to duino?

Resources