What and where exactly are privileged code, chrome code, Gecko? - firefox-addon-webextensions

In reading about the File API and wanting to write data directly from an indexedDB database to the client disk instead of first building and holding a large blob in RAM to download to disk, there are a few basic items I'm not understanding.
In the MDN documents these two statements are found:
In Gecko, privileged code can create File objects representing any local file without user interaction.
If you want to use the DOM File API in chrome code, you can do so without restriction. In fact, you get one bonus feature: you can create File objects specifying the path of the file on the user's computer. This only works from privileged code, so web content can't do it.
Where exactly does one write chrome code and/or Gecko priveleged code? Is this beyond a web extension? I've read and experimented with extensions; so, I'm not asking specifically about how to access them.
I'm not concerned about a 'normal' web page and server accessing the client disk. I know that it's not permitted inorder to protect the individual.
I'm interested in what can be done offline through the browser--with the aid of web extensions and/or a separate profile granting special permissions but without node.js, electron, etc.-- by an individual who knowingly wants to use the browser to do maybe what they should have built in the OS rather than the browser.
Put another way, if I want to use the browser just to run my javascript code to perform tasks all offline on my own machine, where is the privileged code written that gives access to these types of APIs that aren't subject to the security issues of a normal web page?
Is it still javascript or C++ in these areas?
Thank you.
This old question provides a link to their extension which includes the File API that writes to disk in a way that appears to provide a means to bypass the creation of a large blob of data. It's six years old but appears to contain what is needed, at least to get started.
I'm not referring to their trying to get around using indexedDB, but just that using this type of extension could allow for writing each object from the database directly to the client disk without first having to generate a large blob to download.
Attempt at employing Andrew Swan's suggestion
I'm trying to put the pieces together but have reached a point where am not sure how to continue. I wrote the code below in the background script of an extension. In attempting to employ Andrew Swan's suggestion, the plan is to intitiate a GET request for a text/csv file, which is intercepted and replaced by data extracted from database and written to the GET request by the stream filter.
First, make a GET request to a bogus url and listen for response, as follows:
let request = new XMLHttpRequest();
request.open("GET", url );
request.setRequestHeader( "Content-Type", "text/csv" );
request.send(null);
request.onreadystatechange = () =>
{
portFromCS.postMessage( { 'func' : 'disp_result', 'args' : { 'msg' : "request.status :", 'value' : request.status + ' : ' + request.statusText} } );
};
Second, intercept the request and write to the GET, as follows:
browser.webRequest.onBeforeRequest.addListener(
listener,
{ urls : ["<all_urls>"] },
["blocking"] );
function listener( details )
{
let filter = browser.webRequest.filterResponseData(details.requestId);
let decoder = new TextDecoder("utf-8");
let encoder = new TextEncoder();
filter.onstart = event =>
{
let str = decoder.decode(event.data, {stream: true});
str = '' +
'HTTP/1.1 200 OK \r\n' +
'Content-Length: 17 \r\n' +
'Content-Type: text/csv \r\n\r\n\r\n' +
'This is a string.';
filter.write(encoder.encode(str));
filter.disconnect();
};
}
The message sent from the background script in the request.onreadystatechange function is received in the content script, and the request.status is '0'.
The filter.onstart is used because the ondata event will never fire since the url is bogus. Also, that means there will be no converting of data from the url, but only the writing of new data through the filter.
The str data is written and received by the request, but only as responseText and not as a response header. The request.status remains '0' instead of '200'.
It seems that can't change the response header unless in onHeadersReceived which will never take place, it appears, for a bogus url. However, I tried this on a real url and, even though the event fired, an error of webRequest.HttpHeaders is not a function was thrown. I had "responseHeaders" in the webRequest extraInfoSpec at that time.
My questions are:
Can a response header be written to set the request.status to '200' and then start writing the database data through an async function in small blocks as retrieved?
Can the Content Disposition section of the header response be set such that it automatically starts the download of the response.text and allows the user to select the file name and save location, and stay "open" as keep writing to the file as the data is extracted from the database and passed to the GET request through the filter.write()?
Thank you.
Conclusion
It was a good idea but I don't think it is possible for at least two reasons.
One is that webRequest doesn't appear to intercept a downloads.download() function at all, or any download event; so, you can't intercept a download, and an event with a Content Disposition of 'attachment' is needed to even try to write to it with a stream. I could intercept a forced click to an anchor tag href but no other events fired beyond onBeforeRequest.
The other is that a response header can't be modified until an onHeadersReceived event, which means the fake URL has to return something. You can't just cancel it in onBeforeRequest. So, this wouldn't work offline. But, even if you let it process online to an existing URL that returns a reponse header, it won't accept a modification. I tried repeatedly to modify the response header and it just won't work. I tried an XMLHttpRequest GET and can intercept the events that fire but can't modify the response header; so, can't set Content Disposition to 'attachment', with or without file, to start a download. I can write to it but it's no good unless it is going to download what is written. It would be ok if the written content was going to a web page.
Also, if you redirect the URL along the way to anything other than a webRequest acceptable URL, the other events won't be interceptable. So, if redirect to an object URL in onBeforeRequest, you won't intercept the response headers stage in webRequest but can view threm in the onreadystatechange event of the XMLHttpRequest.
So, the upshot is that it appears the response headers cannot be modified even though the MDN Web Docs say it is possible. And, this idea of using awebRequest stream filter to stream data generated on the client or extracted from an indexedDB database, as opposed to building one large blob for download, won't work because can't intercept a download or change the response headers to trigger a download into which to write via the stream filter.
It was an interesting idea though. I still wonder whether or not the download would remain 'open', so to speak, while the data was being written on the client and passed in blocks or chunks. Perhaps, if that part of the response headers that states how data is to be passed and received was modified also it would work.
For now, I am no longer pursuing this approach. One of the Web Docs or a bug records stated that it is planned to allow a data URL to be intercepted. Perhaps, for an offline download to the client, that would be preferrable to a fake URL.
If anyone gets this to work, please let us know. Thank you.

A couple of terms:
"Gecko" is the rendering engine on which Firefox (and a few other applications like Thunderbird) is built
"Chrome" in this context means the browser user interface and features, as opposed to the contents of a web page being displayed by the browser.
In Firefox, much of the browser chrome is implemented in Javascript. The code that implements the user interface needs to be able to do things that normal web pages cannot do (such as reading and writing the local filesystem). Therefore, this code runs with different privileges than Javascript that runs as part of a web page. The terms "privileged code", "chrome privileged code", "Gecko privileged code" are all different ways to describe the same thing: Javascript code that is built in to the browser and has access to capabilities that web pages do not have.
Prior to the Firefox Quantum (version 57) release, Firefox extensions were allowed to run privileged Javascript code. As you might imagine, this was fraught with problems for security, performance, and stability, among other things. With WebExtensions, extensions now run with the same level of privilege as regular web content (ie, they do not execute with elevated privileges). Some browser features are exposed to extensions through extension APIs.
So, if you're interested in what you can do from an extension, any documents on MDN that reference privileged code, are effectively irrelevant. There are not currently any APIs available to WebExtensions that would allow you to directly access the filesystem, but there is an open bug to add some this capability. (that bug has existed for quite some time, but I suspect there will be progress relatively soon...)

Related

JMeter - embedded requests has Bundle.js which in turn triggers more secondary requests - Recursive retrieving of embedded resource requests

My JMeter version is apache-jmeter-5.4.1.
I am trying to set up a HTTP Request something like this on a react based website:
HTTP Request - GET http://YYY.YYY.YYYY/141719 (With Retrieve Embedded Resources checked)
When I run this, I can see JMeter captures embedded resource requests (Secondary requests) which are like *.css, *.js
Second set of embedded resource requests :
However one of these secondary requests called - bundle.xxxxxxx.js creates another set of embedded resource requests to the server which retrieve further *.js files as part of the Request Initiator chain.
While the name of this file itself is randomly generated like, ex., bundle.0787f963ab0ac67dd7d4.js
The browser of course parses this bundle.xxxxxxx.js immediately and gets all the embedded resources/requests(including chunk.*.js)
My problem is how do I replicate this behaviour using JMeter, for the second set of embedded resource requests also to be triggered. At the moment, I can only achieve capturing the first set of embedded resource requests. This does not give me true load test results as the second set has more traffic to the server. Is there a way to recursively Retrieve all embedded resources.
Our application under test is based on React JS.
As per JMeter project main page:
JMeter is not a browser, it works at protocol level. As far as web-services and remote services are concerned, JMeter looks like a browser (or rather, multiple browsers); however JMeter does not perform all the actions supported by browsers. In particular, JMeter does not execute the Javascript found in HTML pages. Nor does it render the HTML pages as a browser does (it's possible to view the response as HTML etc., but the timings are not included in any samples, and only one sample in one thread is ever displayed at a time).
There are several initiator types:
parser (index, styles, fonts, etc.) - these guys will be captured by embedded resources downloading functionality. All below will need to be handled somehow else
redirect
script
other
So if you need to mimic a number of HTTP requests which originate from JavaScript it will be required to replicate the logic from this JavaScript code in JMeter.
There is a minimal chance that you will be able to copy and paste the JavaScript into JSR223 PostProcessor however most likely it relies on certain objects like navigator, window or XMLHttpRequest so it's highly likely that you will have to re-write it in Groovy. Once you have enough data to properly build HTTP Request samplers most probably you will need to put them under the Parallel Controller
Another option is going for WebDriver Sampler and use real browsers for your tests but be aware that browsers are very resources intensive comparing to HTTP Request samplers

Status 500 Internal Server Error in IE-11 with Angular Js Application

I am implementing single page application(SPA) using of Angular Js, MongoDb. And I am using rest call with promises. Rest call working fine in Chrome, Mozila browser which is using for development. But rest call is not working in IE-11. It is giving me 500 Internal Server Error.
I am not able to find out line of rest call. Because it is not showing line number. But I can share sample code of rest call.
Rh.all('apicall').get('dbname/_aggrs/'+ ar_dep +'?avars=' + query).then(function (d) {
console.log("response data");
});
Above call is not printing console. Because It is breaking in IE-11, But these rest call working fine in other browser.
If I putt direct path not with variable then it is working in IE-11.
Working Rest Call below
Rh.all('apicall').get('dbname').then(function (d) {
console.log("response data");
});
NETWORK in Console(IE-11)
IN CHROME
I am updating my question. Because I found some difference parsing url, Because of restheart.
IN CHROME:
Rh.all('apicall').get('dbname/_aggrs/'+ ar_dep +'?avars=' + query)
After parsing
localhost:8080/apicall/dbname/_aggrs/rout?avars={%22routes%22:%22US%22}
In query object I have routes:us. So in chrome it parsing %22--%22 place of " ".
IN IE-11
Rh.all('apicall').get('dbname/_aggrs/'+ ar_dep +'?avars=' + query)
After parsing
localhost:8080/apicall/dbname/_aggrs/rout?avars={"routes":"US"}
In IE-11, It is not parsing double qoutes to %22 %22. It is parsing same as string.
A 500 error is always related to the server. The symptoms may only occur with a specific browser, but it is the server that is failing; the request that is being sent to the server is causing the server-side code to fail in some way.
Error 500 on its own is too generic; without knowing more details about the error, it is always very hard to diagnose, and frankly I won't be able to give you a definitive answer here.
At your end, you should rule out the obvious, and check your browser settings in IE. Specifically, any settings that might cause it to fail to communicate properly with the server. For example, make sure that cookies are enabled and are working properly.
But the first thing you should do is discuss with the vendor or developers of the API because they will have access to the server error logs, and they will want to know about it if their code is throwing a 500 error.
However, if you do want to investigate at your end, the fact that it is specific to one browser is a clue. If the other browsers are working, then what this tells us is that this one browser (IE11) is sending the request with something about it that is different to the other browsers, and it is that something that is triggering the server-side code to fail. This gives us something to work with in the investigation.
So the first thing to do is to examine the request in all browsers. Use the F12 dev tools in Chrome, Firefox and IE, and get to the point where you've made the same call in all three of them, and it works in FF and Chrome but not in IE11.
In the dev tools, you should now be able to examine the request details for all three. Compare them.
Start by looking at the request data -- ie the actual query string that was sent. If there are differences, consider whether any of these differences may be responsible for the error. Something may stand out obviously; eg if IE has truncated a variable or something like that. If this solves the problem, then great.
If it doesn't help, then you need to look in more detail. Maybe there are some differences but they don't look like they should break anything? Modern browser dev tools allow you to edit and re-send a request, so try editing the request in Chrome or Firefox's dev tools, and make the parameters the same as the ones from IE that failed. Now try re-sending that request. If you're lucky, this will cause the request to fail in the other browser, which will allow you to show that a specific set of data is the problem (rather than a specific browser). You mentioned that it's a third party API, so you'll then need to discuss with the API vendor to find out why that query breaks their API.
If you still haven't found the problem at this point, and you're sending identical queries in both browsers, and you're logged in as the same user, then the next step is to look at the request headers.
There is one request header that will definitely be different: the User Agent string. But there may be others too. Again, try re-sending request that works in Chrome, but with headers from the failing request in IE (including the UA string). Does the request now fail in Chrome? If so, narrow down which headers are different that make it fail.
Again, if this allows you to find a specific set of request data and headers that causes the problem, then you will need to discuss with the API vendor.
If all of this doesn't help, then try looking at the cookies. You already checked that cookies are working, so this seems like a long shot now, but again compare the cookies between browsers, and see if there's anything obviously different about them.
I hope the above is enough to help you diagnose the issue.

How to log react console errors to a file?

I am new in front-end/client-side app development for website. I am setting up a new react project. I used create-react-app.
How should I handle the console errors(other than network call errors)? Is there a way to log them in any file?
What are the best logging practices?
Even though there are means and ways to store data on the browser (not to actual client files) that's not really the general strategy for logging browser errors (which I believe is what you're referring to).
In my experience we'd have a dedicated logging server with a simple API with adequate security that filters traffic and applies rate limiting. That would finally enhance and write logs into a document database that can later be analysed.
A naive JavaScript solution would be using the following to capture errors and send them out to a logging server.
window.onerror = function(message, url, lineNumber) {
// make ajax call to api to save error on server
return true;
};
I should also mention Sentry.io - they provide a service that does this and even though there are some limitations they're usually enough for a small to mid app.
https://github.com/getsentry

API to Database?

Please presume that I do not know anything about any of the things I will be mentioning because I really do not.
Most OpenData sites have the possibility of exporting the presented file either in for example .csv or .json formats (Example). They also always have an API tab (Example API).
I presume using the API would mean that if the data is updated you would receive the change whereas exporting it as .csv would mean the content will not be changed anymore.
My questions is: how does one use this API code to display the same table one would get when exporting a .csv file.
Would you use a database to extract this information? What kind of database and how do you link the API to the database?
I presume using the API would mean that if the data is updated you
would receive the change whereas exporting it as .csv would mean the
content will not be changed anymore.
You are correct in the sense that, if you download the csv to your computer, that csv file won't be updated any more.
An API is something you would call - in this case, you can call the API, saying "Hey, do you have the latest data on xxx?", and you will be given back the latest information about what you have asked. This does not mean though, that this site will notify you when there's a new update - you will have to keep calling the API (every hour, every day etc) to see if there are any changes.
My questions is: how does one use this API code to display the same
table one would get when exporting a .csv file.
You would:
Call the API from a server code, or a cloud service
Let the server code or cloud service decipher (or "Parse") the response
Use the deciphered response to create a table made out of HTML, or to place it into a database
Would you use a database to extract this information? What kind of
database and how do you link the API to the database?
You wouldn't necessarily need a database to extract information, although a database would be nice to place the final data inside.
You would first need some sort of way to "call the REST API". There are many ways to do this - using Shell Script, using Python, using Excel VBA etc.
I understand this is hard to visualize, so here is an example of step 1, where you can retrieve information.
Try placing in the below URL (taken from the site you showed us) in your address bar of your Chrome browser, and hit enter
http://opendata.brussels.be/api/records/1.0/search/?dataset=associations-clubs-sportifs
See how it gives back a lot of text with many brackets and commas? You've basically asked the site to give you some data, and this is the response they gave back (different browsers work differently - IE asks you to download the response as a .json file). You've basically called an API.
To see this data more cleanly, open your developer tools of your Chrome browser, and enter the following JavaScript code
var url = 'http://opendata.brussels.be/api/records/1.0/search/?dataset=associations-clubs-sportifs';
var xhr = new XMLHttpRequest();
xhr.open('GET', url);
xhr.setRequestHeader('X-Requested-With', 'XMLHttpRequest');
xhr.onload = function() {
if (xhr.status === 200) {
// success
console.log(JSON.parse(xhr.responseText));
} else {
// error
console.log(JSON.parse(xhr.responseText));
}
};
xhr.send();
When you hit enter, a response will come back, stating "Object". If you click through the arrows, you can see this is a cleaner version of the data we just saw - more human readable.
In this case, I used JavaScript to retrieve the data, but you can use whatever code you want. You could proceed to use JavaScript to decipher the data, manipulate it, and push it into a database.
kintone is an online cloud database where you can customize it to run JavaScript codes, and have it store the data in their database, so you'll have the data stored online like in the below image. This is just one example of a database you can use.
There are other cloud services which allow you to connect API end points of different services with each other, like IFTTT and Zapier, but I'm not sure if they connect with open data.
The page you linked to shows that the API returns values as a JSON object. To access the data you can just send an appropriate http request and the response will be the requested data as a JSON. You can send requests like that over your browser if you want to.
Most languages allow JSON objects to be manipulated pro grammatically if you need to do work on the data.
Restful APIs publish model is "request and publish". Wen you request data via an API endpoint, you would receive response strings in JSON objects, CSV tables or XML.
The publisher, in this case Opendata.brussel.be would update their database on regular basis and publish the results via an API endpoint.
If you want to download the table as a relational data table in a CSV file, you'd need to parse the JSON objects into relational tables. This can be tricky since each JSON response string can vary in their paths.
There're several ways to do it. You can either write scripts to flatten the JSON objects or use a tool to parse and flatten the objects for you.
I use a tool called Acho to turn API endpoints into CSV files. It would parse almost all API endpoints through the parameters and even configure for multiple requests, such as iterative and recursive requests.
Acho API parser

How to secure file download?

I have an application written in angularjs and a dropwizard backend. All API calls are ajax, with the exception of file downloads, which is done by performing a redirect to a standard GET request.
All API calls are secured through a token which is passed as a Token header. We use SSL for all APIs.
The download GET request works but I'm having a hard time figuring out how to secure it. I have no way of setting a custom header, which is required to pass the token. So theoretically, I'm left with two options, clearly none of them acceptable: 1. Pass the token as one the GET parameters 2. Leave the download unsecured.
Any ideas how to secure file download?
Putting a secret token in a URL query parameter isn't great because URL tend to be leakable, for example through history/logging/referrers. There are ways to mitigate this: for example you could have the server side issue a download token that is only good for one use or for a limited amount of time. Or the client could pass a time-limited token created using a signature over the secret token that the server side could verify.
Alternatively you could, just for this one interface (eg path-limited, quitckly-expiring) put the token in a cookie.
Another approach is to download the whole file through AJAX, thus allowing you to set the header as normal. Then you have to present the content as a downloadable local resource, which requires a cocktail of browser-specific hacks (eg using data: or filesystem: URLs, and potentially links with the download attribute). Given the complication this isn't usually worth bothering with, especially if the file is very large which may present further storage constraints.

Resources