Using Azure.AI.FormRecognizer 1.0.0-preview.2 and trying to upload a file and then run forms recognition on it. Code is
var fileName = Path.Combine(#"c:\temp\", sourceFile.FileName);
var fileUri = new Uri(fileName);
sourceFile.SaveAs(fileName);
var forms = await recogClient.StartRecognizeCustomFormsFromUri(modelId, fileUri).WaitForCompletionAsync();
The file URI becomes, for example, file:///c:/temp/DC002.pdf which I believe is a valid URI. However, when running StartRecognizeCustomFormsFromUri, I get the error:
Service request failed. Status: 400 (Bad Request) Content: {"error":{"code":"1003","message":"Parameter 'Source' is not a valid Uri."}} Headers: Transfer-Encoding: chunked x-envoy-upstream-service-time: REDACTED apim-request-id: REDACTED Strict-Transport-Security: REDACTED x-content-type-options: REDACTED Date: Wed, 27 May 2020 12:30:01 GMT Content-Type: application/json; charset=utf-8
FileUri needs to be a public accessible URL, you can not point to your local filesystem. If you would like to send a local file, you should send the file as a file stream.
The Uri is a great way to speed up the processing when your files are already on a blob storage, or any other public accessible cloud storage, saving time of not streaming the file to the Form Recognizer service.
Recognize forms from
File
Recognize forms from Uri
Related
Angular version is 1.3.13
I'm making a url request to a public url to get some data. For one url it functions but for the other it does not. I have no control over the server side.
The URL returns with 200 but AngularJS passes it to the error function. In my browser console the data is there, as JSON, but I do not have access to it in Angular. I was thinking it might the the http.get calling something before passing the data on.
I've tried with plain $http and a custom transformResponse but it still falls to the error of the custom response.
This is the URL:
https://uatmerchant.sixdots.be/oidc/.well-known/openid-configuration
These are the response headers from the URL:
Connection
close
Content-Security-Policy
reflected-xss block
Content-Type
application/json
Date
Mon, 15 Jan 2018 12:33:30 GMT
Set-Cookie
BIGipServer~DMZ~pool_uat_5000=…omain=.uatmerchant.sixdots.be
Strict-Transport-Security
max-age=15552000; includeSubDomains
Transfer-Encoding
chunked
X_CORRELATION_ID
UAT-MER-F5-20180115133330632
X-Content-Security-Policy
reflected-xss block
X-Content-Type-Options
nosniff
X-Frame-Options
SAMEORIGIN
X-Xss-Protection
1; mode=block
This is part of the data that is in the response in the browser console
request_parameter_supported true
claims_parameter_supported false
scopes_supported […]
0 openid
1 profile
2 email
3 address
4 phone
issuer https://uatmerchant.sixdots.be/oidc
acr_values_supported […]
0 tag:sixdots.be,2016-06:acr_basic
Here is the code making the call with the URL that is detailed at the start of this post above
var oohahhel = $sce.trustAsResourceUrl(urlVar);
$http.get(oohahhel)
.then(
function success(response){
var jsonResponse = angular.fromJson(response);
//process response
}
,function error(reason){
//process error
});
I currently have a fully functional email notification system. It is all working correctly, but have noticed that one of the inline images, which is attached to the email via a rest service, is being requested three times.
It appears to be happening once transport.send is invoked. I'm assuming that when javamail goes to send email, it then calls for all inline images referenced and generates the base64 image and places it in the outgoing email.
My question is...
Is the above assumptions correct and why would it be calling the service multiple times when it has been verified in the raw email that it contains the image only once.
Below is a copy of the raw email with addresses changed, base64 images, text/plain and text/html versions removed.
Raw email
Date: Thu, 25 Feb 2016 20:40:04 -0800 (PST)
From: something.news.noreply#something.org
To: user.email#something.org
Message-ID: <56557433.31456461613977.JavaMail.something.news.noreply#something.orgh>
Subject: Business News for 02/26/2016
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_Part_6_670038123.1456461604268"
X-Priority: 3
------=_Part_6_670038123.1456461604268
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
TEXT VERSION OF EMAIL
------=_Part_6_670038123.1456461604268
Content-Type: multipart/related;
boundary="----=_Part_7_2073972040.1456461604268"
------=_Part_7_2073972040.1456461604268
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
HTML VERSION OF EMAIL
------=_Part_7_2073972040.1456461604268
Content-Type: image/png;
name="840da574-a395-4fb8-8f33-bcbb1837220e?t=1452039853427"
Content-Transfer-Encoding: base64
Content-ID: <#9dc060f2-84ec-4344-8871-68d4e4885b70_840da574-a395-4fb8-8f33-bcbb1837220e?t=1452039853427>
Content-Disposition: inline;
filename="840da574-a395-4fb8-8f33-bcbb1837220e?t=1452039853427"
BASE 64 IMAGE
------=_Part_7_2073972040.1456461604268
Content-Type: image/png;
name="21ae1010-9675-4daa-8ac8-659052b943e8?t=1450724912940"
Content-Transfer-Encoding: base64
Content-ID: <#17ab1018-2bc5-44f5-9934-fb7b0f1a860b_21ae1010-9675-4daa-8ac8-659052b943e8?t=1450724912940>
Content-Disposition: inline;
filename="21ae1010-9675-4daa-8ac8-659052b943e8?t=1450724912940"
BASE 64 IMAGE
------=_Part_7_2073972040.1456461604268
Content-Type: image/png;
name="a96af9b6-9093-4086-bf91-e5677bef533f?t=1450966405365"
Content-Transfer-Encoding: base64
Content-ID: <#15a682c9-95e8-42ee-880b-f52a47396341_a96af9b6-9093-4086-bf91-e5677bef533f?t=1450966405365>
Content-Disposition: inline;
filename="a96af9b6-9093-4086-bf91-e5677bef533f?t=1450966405365"
BASE 64 IMAGE
------=_Part_7_2073972040.1456461604268
Content-Type: image/png;
name="25caf361-80a9-44c2-9ae6-97c19709becf?t=1450966427231"
Content-Transfer-Encoding: base64
Content-ID: <#2e9839ae-9c10-4c8f-9c4a-bc9e1a934fc3_25caf361-80a9-44c2-9ae6-97c19709becf?t=1450966427231>
Content-Disposition: inline;
filename="25caf361-80a9-44c2-9ae6-97c19709becf?t=1450966427231"
BASE 64 IMAGE
------=_Part_7_2073972040.1456461604268
Content-Type: image/png; name=upChart
Content-Transfer-Encoding: base64
Content-ID: <#6ca90a68-1d24-4414-ac8a-8c84b50bb663_upChart>
Content-Disposition: inline; filename=upChart
BASE 64 IMAGE
THIS IS THE IMAGE THAT IS RETRIEVED VIA A WEB SERVICE CALL. IT CREATES A CHART
IMAGE THAT GET ATTACHED TO EMAIL.
------=_Part_7_2073972040.1456461604268
Content-Type: image/png;
name="10739ce0-c979-4e47-8cee-666304b9a92e?t=1451426510519"
Content-Transfer-Encoding: base64
Content-ID: <#14227419-3db7-444b-a704-7abaccd1e380_10739ce0-c979-4e47-8cee-666304b9a92e?t=1451426510519>
Content-Disposition: inline;
filename="10739ce0-c979-4e47-8cee-666304b9a92e?t=1451426510519"
BASE 64 IMAGE
------=_Part_7_2073972040.1456461604268
Content-Type: image/png;
name="995344fa-765f-4a27-848c-d36d26c593f3?t=1451426553685"
Content-Transfer-Encoding: base64
Content-ID: <#f17f0535-a2a3-47a8-aca4-9ed883542de8_995344fa-765f-4a27-848c-d36d26c593f3?t=1451426553685>
Content-Disposition: inline;
filename="995344fa-765f-4a27-848c-d36d26c593f3?t=1451426553685"
BASE 64 IMAGE
------=_Part_7_2073972040.1456461604268--
------=_Part_6_670038123.1456461604268--
I have added I believe the relevant code. The code starts with what it started out like and then two changes I made based on the suggestion.
Started as:
try {
// Get a REST client so we can call the Highcharts service.
hcClient = new RestClient(hcServer, hcResource);
// Create a client response object.
ClientResponse cr = null;
// Initialize tries counter and loop a maximum of 3 tries
// if highcharts export server doesn't respond with a 200.
int tries = 0;
do {
// Increase tries count.
tries++;
// Call the service and place response data into our data object.
cr = hcClient.getWebResource().post(ClientResponse.class, params);
// Pull the response status from client.
upChartRespStatus = cr.getStatus();
// Determine if we received a 200 code or not.
if(upChartRespStatus == 200) {
// Obtain entity and build response.
response = Response
.ok(new ByteArrayInputStream(cr.getEntity(cbChart.getClass())))
.build();
} else {
// Log warning that chart was not received, what the response
// status was and what try we are on.
logger.warn("Unable to get requested chart, response code was "+upChartRespStatus+".");
logger.warn("Reattempting to get chart. The current count of retries is: "+tries);
}
} while (tries <= 3 && upChartRespStatus != 200);
} catch(Exception e) {
logger.error(e.getMessage());
} finally {
hcClient.destroy();
}
I thought maybe I would try sending a base64 encoded image. I didn't even get the image in the email doing this.
// Determine if we received a 200 code or not.
if(upChartRespStatus == 200) {
// Get entity from client request.
ByteArrayInputStream bais = new ByteArrayInputStream(cr.getEntity(cbChart.getClass()));
String b64Image = new String(Base64.encode(read(bais)));
// Obtain entity and build response.
response = Response
.ok(b64Image)
.header("Content-Type", "image/png")
.header("Content-Length", b64Image.length())
.header("Content-Transfer-Encoding", "BASE64")
.build();
} else {
// Log warning that chart was not received, what the response
// status was and what try we are on.
logger.warn("Unable to get requested chart, response code was "+upChartRespStatus+".");
logger.warn("Reattempting to get chart. The current count of retries is: "+tries);
}
My last attempt was going back to binary, but making sure that the headers were all in place. This last attempt did yield the chart image in the email, but still in all cases the transport.send is still calling the service three times before it actually sends.
// Determine if we received a 200 code or not.
if(upChartRespStatus == 200) {
// Get entity from client request.
ByteArrayInputStream bais = new ByteArrayInputStream(cr.getEntity(cbChart.getClass()));
// Obtain entity and build response.
response = Response
.ok(bais)
.header("Content-Type", "image/png")
.header("Content-Length", bais.available())
.header("Content-Transfer-Encoding", "BINARY")
.build();
} else {
// Log warning that chart was not received, what the response
// status was and what try we are on.
logger.warn("Unable to get requested chart, response code was "+upChartRespStatus+".");
logger.warn("Reattempting to get chart. The current count of retries is: "+tries);
}
Okay. So I got some success. Instead of setting headers at the service, I think that was to late. I took a look at the code that creates the email and specifically when the image gets added to its part. I've added what the code was and what I changed. By doing this, it is now only calling the service twice instead of three times.
private void addImagesInline(Multipart parent, List<URL> embeded, HashMap<String,String> cids) throws MessagingException {
if (embeded != null) {
for (URL img : embeded) {
final MimeBodyPart htmlPartImg = new MimeBodyPart();
DataSource htmlPartImgDs = new URLDataSource(img);
htmlPartImg.setDataHandler(new DataHandler(htmlPartImgDs));
String fileName = img.getFile();
fileName = getFileName(fileName);
String newFileName = cids.get(fileName);
boolean imageNotReferencedInHtml = newFileName == null;
if (imageNotReferencedInHtml) continue;
htmlPartImg.setHeader("Content-ID", "<"+newFileName+">");
htmlPartImg.setDisposition(BodyPart.INLINE);
htmlPartImg.setFileName(fileName);
parent.addBodyPart(htmlPartImg);
}
}
}
I added these to lines of code.
htmlPartImg.setHeader("Content-Type", "image/png");
htmlPartImg.setHeader("Content-Transfer-Encoding", "BASE64");
Thanks in advance for any replies!
I would expect the image to be requested twice. JavaMail will read the image once to determine what encoding is appropriate, based on the content of the image data. It will then read it again to encode it and include it in the message.
You can read the image yourself and save it locally to avoid multiple requests, or you can tell JavaMail what encoding to use by setting the Content-Transfer-Encoding header for the part; that way it won't have to guess what to use.
I've run a pen test tool (Burp) against my node(express)/angular application and it identified a reflected XSS vulnerability specifically when attempting a GET request for static assets (noticeably vulnerabilities were not found for any of the requests being made when a user interacts with the application).
The issue detail is:
The name of an arbitrarily supplied URL parameter is copied into a
JavaScript expression which is not encapsulated in any quotation
marks. The payload 41b68(a)184a9=1 was submitted in the name of an
arbitrarily supplied URL parameter. This input was echoed unmodified
in the application's response.
This behavior demonstrates that it is possible to inject JavaScript
commands into the returned document. An attempt was made to identify a
full proof-of-concept attack for injecting arbitrary JavaScript but
this was not successful. You should manually examine the application's
behavior and attempt to identify any unusual input validation or other
obstacles that may be in place.
The vulnerability was tested by passing an arbitrary url parameter to the request like so:
GET /images/?41b68(a)184a9=1
The response was:
HTTP/1.1 404 Not Found
X-Content-Security-Policy: connect-src 'self'; default-src 'self'; font-src 'self'; frame-src; img-src 'self' *.google-analytics.com; media-src; object-src; script-src 'self' 'unsafe-eval' *.google-analytics.com; style-src 'self' 'unsafe-inline'
X-XSS-Protection: 1; mode=block
X-Frame-Options: DENY
Strict-Transport-Security: max-age=10886400; includeSubDomains; preload
X-Download-Options: noopen
X-Content-Type-Options: nosniff
Content-Type: text/html; charset=utf-8
Content-Length: 52
Date: Wed, 08 Oct 2015 10:46:43 GMT
Connection: close
Cannot GET /images/?41b68(a)184a9=1
You can see that I have CSP in place (using Helmet to implement) and other protections against exploits. The app is served over https, but no user auth is required. CSP restricts request to the app's domain only plus google analytics.
The pen test report advises validating input (I am, but surely that would make requests including data sent by a user unsafe if I wasn't?), and encoding html which angular does by default.
I'm really struggling to find a solution to preventing or mitigating this for those requests for static assets:
Should I whitelist all requests for my application under csp?
Can I even do this, or will it only whitelist domains?
Can/should all responses from node/express to requests for static assets be encoded in some way?
The report states that "The name of an arbitrarily supplied URL parameter is copied into a JavaScript expression which is not encapsulated in any quotation marks". Could this expression be somewhere in the express code that handles returning static assets?
Or that GET request param can somehow be evaluated in my application code?
Update
Having done some investigation into this it seems that at least part of the mitigation is to escape data in url param values and sanitize the input in the url.
Escaping of the url is already in place so:
curl 'http://mydomain/images/?<script>alert('hello')</script>'
returns
Cannot GET /images/?<script>alert(hello)</script>
I've also put express-sanitized in place on top of this.
However, if I curl the original test the request param is still reflected back.
curl 'http://mydomain/images/?41b68(a)184a9=1'
Cannot GET /images/?41b68(a)184a9=1
Which you would expect because html is not being inserted into the url.
The responses to GET requests for static assets are all handled by app.use(express.static('static-dir')) so the query is passed into this. express.static is based on serve-static which depends on parseurl.
The cause of the issue is that for invalid GET requests express will return something like:
Cannot GET /pathname/?yourQueryString
Which in many cases is a valid response, even for serving static assets. However, in my case and I'm sure for others the only valid requests for static assets will be something like:
GET /pathname/your-file.jpg
I have a custom 404 handler that returns a data object:
var data = {
status: 404,
message: 'Not Found',
description: description,
url: req.url
};
This is only handled for invalid template requests in app.js with:
app.use('/template-path/*', function(req, res, next) {
custom404.send404(req, res);
});
I've now added explicit handlers for requests to static folders:
app.use('/static-path/*', function(req, res, next) {
custom404.send404(req, res);
});
Optionally I could also strip out request query params before the 404 is returned:
var data = {
status: 404,
message: 'Not Found',
description: description,
url: url.parse(req.url).pathname // needs a var url = require('url')
};
I am trying to access a website, and then return whatever it outputs in the body -> eg. "Success" or "Failed".
When I try with my code, I am getting the following back.
<<< REQ >>>
HTTP/1.1 200 OK
Date: Sat, 30 Aug 2014 17:36:31 GMT
Content-Type: text/html
Connection: close
Set-Cookie: __cfduid=d8a4fc3c84849b6786c6ca890b92e2cc01409420191023; expires=Mon, 23-Dec-2019 23:50:00 GMT; path=/; domain=.japseyz.com; HttpOnly
Vary: Accept-Encoding
X-Powered-By: PHP/5.3.28
Server.
My code is: http://pastebin.com/WwWbnLNn
If all you want to know is whether the HTTP transaction succeeded or failed, then you need to examine the HTTP Response code... which is in the first line of the response. In your example it is "200"... the human readable interpretation of it is "OK".
Here is a link to most of the HTTP 1.1 response codes: w3.org-rfc2616 RespCodes
Your question indicated you wanted to extract this information from the "body"...
... but that information is not located in the "body", it is in the first response
header, as described above.
have you tried ethercard samples? there is a webclient sample, in which you can find procedure called CALLBACK - in that procedure you can process data stored in buf variable.
in your case you need to look for first empty line, which tells you that headers has been sent and page content(what php writes to the page i.e.) follows.
how familiar are you at pointers? how deep you do need to process the page output? i.e. OK or ERROR is enough, or you do need to pass same parameters back to duino?
fetch data from server returns me json data as a string datatype rather than as application/json datatype, as a result the collection does not get refreshed.
I have tried giving the jquery.ajax option contentType:"application/json" to the fetch options, but still does not work.
how can i make it work? do i send a mimetype from the server? if so, how?
i am using json_encode on the data sent.
preloader.fetch({
contentType:'application/json'
});
preloader is an instance of my collection.
edit:
my template for a subview was not getting detected as i had kept it out of the masterview's $el element, corrected it, and now i am getting underscore.js error, that
str is null in
str.replace(/\\/g, '\\\\') //at line 913
is this because the backbone app is not taking it as a json object?
Request headers
Connection close
Content-Type text/html
Date Thu, 12 Apr 2012 13:00:58 GMT
Server Apache
Transfer-Encoding chunked
Vary Accept-Encoding
Response headers
has the line
Accept application/json, text/javascript, */*; q=0.01
means it is a json, then what is the problem?
I think the contentType option is for the request (your request).
Try dataType:"json".