So far i was able to set
'disableCaching: false'
in Ext.Loader.config(in app.js) and debug extjs applications on chrome browser.
But now, on inspecting the source, i see that the files have filename.js?dc=1123123 and every time the files are fetched from remote and not cached. So i am unable to set breakpoints and debug run-time on browser.
Please note that this scenario is when i do a browser refresh.
Plz let me know how i can resolve this issue.
Set disableCacheing to false in app.js before Ext.application({...
Ext.Loader.setConfig({
disableCaching: false
});
This will remove _dc cache param from requests that are getting files.
For disabling _dc on XHR Ext.Ajax requests use
Ext.Ajax.disableCaching = false;
And for proxy communication with server use noCache property on Ext.data.proxy.Server class.
noCache: true
You can also set cache config in app.json file.
"loader": {
// This property controls how the loader manages caching for requests:
//
// - true: allows requests to receive cached responses
// - false: disable cached responses by adding a random "cache buster"
// - other: a string (such as the build.timestamp shown here) to allow
// requests to be cached for this build.
//
"cache": "${build.timestamp}",
// When "cache" is not true, this value is the request parameter used
// to control caching.
//
"cacheParam": "_dc"
}
Also if using Chrome Dev Tools for debugging take a look at disableCache on Networks tab and if using FF use CTRL + F5 insted F5 to reload page
Add ?cache=false to the end of your URL to temporarily disable caching on a per-request basis.
Setting Ext.Loader configuration is one thing. There is also something called Ext.Boot which is used before loader is up and running. Boot has his own disableCaching setting. It defaults to something like this:
disableCaching: (/[?&](?:cache|disableCacheBuster)\b/i.test(location.search) ||
!(/http[s]?\:/i.test(location.href)) ||
/(^|[ ;])ext-cache=1/.test(doc.cookie)) ? false :
true,
Probably files with dc appended to url are loaded by Boot. If you want to disable it permanently just replace this code and set it to false.
app.json:
"loader": {"cache": true},
then:
sencha app refresh
it may seem confusing, but the _dc parameter then won't be present anymore - at all.
Related
Background story: I need to obtain the handles of the tagged Twitter users from an attached Twitter media. There's no current API method to do that unfortunately (see https://twittercommunity.com/t/how-to-get-tags-of-a-media-in-a-tweet/185614 and https://github.com/twitterdev/open-evolution/issues/34).
I have no other choice but to scrape, this is an example URL: https://twitter.com/justinwood_/status/1626275168157851650/media_tags. This is the page which pops up when you click on the tags link under the media of the parent Tweet: https://twitter.com/justinwood_/status/1626275168157851650/
The React generated DOM is deep and ugly, but would be scrapeable, however I do not want to log in with any account to get banned. Unfortunately when you visit https://twitter.com/justinwood_/status/1626275168157851650/media_tags in an Incognito window the popup shows up dead empty. However when I dig into the network requests the /TweetDetail GraphQL endpoint is full of messages about the anonymous page visit, fortunately it still contains the list of handles I need despite of all of this.
So what I need to have is a scraper which is able to process JavaScript, and capture the response for that specific GraphQL call. Selenium uses a headless Chrome under the hood, so it is able to process JavaScript, and Selenium-Wire offers the ability to capture the response.
Unfortunately my crafted Selenium-Wire script only has the TweetResultByRestId and UsersByRestId GraphQL requests but is missing the TweetDetail. I don't know what to tweak to make all the requests to happen. I iterated over a ton of Chrome options. Here is a variation of my script:
from seleniumwire import webdriver
from selenium.webdriver.chrome.service import Service
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--headless") # for Jenkins
chrome_options.add_argument("--disable-dev-shm-usage") # Jenkins
chrome_options.add_argument('--start-maximized')
chrome_options.add_argument('--window-size=1900,1080')
chrome_options.add_argument('--ignore-certificate-errors-spki-list')
chrome_options.add_argument('--ignore-ssl-errors')
selenium_options = {
'request_storage_base_dir': '/tmp', # Use /tmp to store captured data
'exclude_hosts': ''
}
ser = Service('/usr/bin/chromedriver')
ser.service_args=["--verbose", "--log-path=test.log"]
driver = webdriver.Chrome(service=ser, options=chrome_options, seleniumwire_options=selenium_options)
tweet_id = "1626275168157851650"
twitter_media_url = f"https://twitter.com/justinwood_/status/{tweet_id}/media_tags"
driver.get(twitter_media_url)
driver.wait_for_request("/TweetDetail", timeout=10)
Any ideas?
Apparently it looks like I'd rather need to scrape the parent Tweet URL https://twitter.com/justinwood_/status/1626275168157851650/ and right now it seems my craved GraphQL call happens. Probably I got confused while trying 100 combinations.
I am loading an <iframe> in my HTML page and trying to access the elements within it using JavaScript, but when I try to execute my code, I get the following error:
SecurityError: Blocked a frame with origin "http://www.example.com" from accessing a cross-origin frame.
How can I access the elements in the frame?
I am using this code for testing, but in vain:
$(document).ready(function() {
var iframeWindow = document.getElementById("my-iframe-id").contentWindow;
iframeWindow.addEventListener("load", function() {
var doc = iframe.contentDocument || iframe.contentWindow.document;
var target = doc.getElementById("my-target-id");
target.innerHTML = "Found it!";
});
});
Same-origin policy
You can't access an <iframe> with different origin using JavaScript, it would be a huge security flaw if you could do it. For the same-origin policy browsers block scripts trying to access a frame with a different origin.
Origin is considered different if at least one of the following parts of the address isn't maintained:
protocol://hostname:port/...
Protocol, hostname and port must be the same of your domain if you want to access a frame.
NOTE: Internet Explorer is known to not strictly follow this rule, see here for details.
Examples
Here's what would happen trying to access the following URLs from http://www.example.com/home/index.html
URL RESULT
http://www.example.com/home/other.html -> Success
http://www.example.com/dir/inner/another.php -> Success
http://www.example.com:80 -> Success (default port for HTTP)
http://www.example.com:2251 -> Failure: different port
http://data.example.com/dir/other.html -> Failure: different hostname
https://www.example.com/home/index.html:80 -> Failure: different protocol
ftp://www.example.com:21 -> Failure: different protocol & port
https://google.com/search?q=james+bond -> Failure: different protocol, port & hostname
Workaround
Even though same-origin policy blocks scripts from accessing the content of sites with a different origin, if you own both the pages, you can work around this problem using window.postMessage and its relative message event to send messages between the two pages, like this:
In your main page:
const frame = document.getElementById('your-frame-id');
frame.contentWindow.postMessage(/*any variable or object here*/, 'https://your-second-site.example');
The second argument to postMessage() can be '*' to indicate no preference about the origin of the destination. A target origin should always be provided when possible, to avoid disclosing the data you send to any other site.
In your <iframe> (contained in the main page):
window.addEventListener('message', event => {
// IMPORTANT: check the origin of the data!
if (event.origin === 'https://your-first-site.example') {
// The data was sent from your site.
// Data sent with postMessage is stored in event.data:
console.log(event.data);
} else {
// The data was NOT sent from your site!
// Be careful! Do not use it. This else branch is
// here just for clarity, you usually shouldn't need it.
return;
}
});
This method can be applied in both directions, creating a listener in the main page too, and receiving responses from the frame. The same logic can also be implemented in pop-ups and basically any new window generated by the main page (e.g. using window.open()) as well, without any difference.
Disabling same-origin policy in your browser
There already are some good answers about this topic (I just found them googling), so, for the browsers where this is possible, I'll link the relative answer. However, please remember that disabling the same-origin policy will only affect your browser. Also, running a browser with same-origin security settings disabled grants any website access to cross-origin resources, so it's very unsafe and should NEVER be done if you do not know exactly what you are doing (e.g. development purposes).
Google Chrome
Mozilla Firefox
Safari
Opera: same as Chrome
Microsoft Edge: same as Chrome
Brave: same as Chrome
Microsoft Edge (old non-Chromium version): not possible
Microsoft Internet Explorer
Complementing Marco Bonelli's answer: the best current way of interacting between frames/iframes is using window.postMessage, supported by all browsers
Check the domain's web server for http://www.example.com configuration for X-Frame-Options
It is a security feature designed to prevent clickJacking attacks,
How Does clickJacking work?
The evil page looks exactly like the victim page.
Then it tricked users to enter their username and password.
Technically the evil has an iframe with the source to the victim page.
<html>
<iframe src='victim-domain.example'/>
<input id="username" type="text" style="display: none;"/>
<input id="password" type="text" style="display: none;"/>
<script>
//some JS code that click jacking the user username and input from inside the iframe...
<script/>
<html>
How the security feature work
If you want to prevent web server request to be rendered within an iframe add the x-frame-options
X-Frame-Options DENY
The options are:
SAMEORIGIN: allow only to my own domain render my HTML inside an iframe.
DENY: do not allow my HTML to be rendered inside any iframe
ALLOW-FROM https://example.com/: allow specific domain to render my HTML inside an iframe
This is IIS config example:
<httpProtocol>
<customHeaders>
<add name="X-Frame-Options" value="SAMEORIGIN" />
</customHeaders>
</httpProtocol>
The solution to the question
If the web server activated the security feature it may cause a client-side SecurityError as it should.
For me i wanted to implement a 2-way handshake, meaning:
- the parent window will load faster then the iframe
- the iframe should talk to the parent window as soon as its ready
- the parent is ready to receive the iframe message and replay
this code is used to set white label in the iframe using [CSS custom property]
code:
iframe
$(function() {
window.onload = function() {
// create listener
function receiveMessage(e) {
document.documentElement.style.setProperty('--header_bg', e.data.wl.header_bg);
document.documentElement.style.setProperty('--header_text', e.data.wl.header_text);
document.documentElement.style.setProperty('--button_bg', e.data.wl.button_bg);
//alert(e.data.data.header_bg);
}
window.addEventListener('message', receiveMessage);
// call parent
parent.postMessage("GetWhiteLabel","*");
}
});
parent
$(function() {
// create listener
var eventMethod = window.addEventListener ? "addEventListener" : "attachEvent";
var eventer = window[eventMethod];
var messageEvent = eventMethod == "attachEvent" ? "onmessage" : "message";
eventer(messageEvent, function (e) {
// replay to child (iframe)
document.getElementById('wrapper-iframe').contentWindow.postMessage(
{
event_id: 'white_label_message',
wl: {
header_bg: $('#Header').css('background-color'),
header_text: $('#Header .HoverMenu a').css('color'),
button_bg: $('#Header .HoverMenu a').css('background-color')
}
},
'*'
);
}, false);
});
naturally you can limit the origins and the text, this is easy-to-work-with code
i found this examlpe to be helpful:
[Cross-Domain Messaging With postMessage]
There is a workaround, actually, for specific scenarios.
If you have two processes running on the same domain but different ports, the two Windows can interact without limitations. (i.e. localhost:3000 & localhost:2000). To make this work, each window needs to change their domain to the shared origin:
document.domain = 'localhost'
This also works in the scenario that you are working with different subdomains on the same second-level domain, i.e. you are on john.site.example trying to access peter.site.example or just site.example
document.domain = 'site.example'
By explicitily setting document.domain; the browser will ignore the hostname difference and the Windows can be treated as coming from the 'same-origin'. Now, in a parent window, you can reach into the iframe: frame.contentWindow.document.body.classList.add('happyDev')
If you have control over the content of the iframe - that is, if it is merely loaded in a cross-origin setup such as on Amazon Mechanical Turk - you can circumvent this problem with the <body onload='my_func(my_arg)'> attribute for the inner html.
For example, for the inner html, use the this html parameter (yes - this is defined and it refers to the parent window of the inner body element):
<body onload='changeForm(this)'>
In the inner html :
function changeForm(window) {
console.log('inner window loaded: do whatever you want with the inner html');
window.document.getElementById('mturk_form').style.display = 'none';
</script>
I experienced this error when trying to embed an iframe and then opening the site with Brave. The error went away when I changed to "Shields Down" for the site in question. Obviously, this is not a full solution, since anyone else visiting the site with Brave will run into the same issue. To actually resolve it I would need to do one of the other things listed on this page. But at least I now know where the problem lies.
I would like to add Java Spring specific configuration that can effect on this.
In Web site or Gateway application there is a contentSecurityPolicy setting
in Spring you can find implementation of WebSecurityConfigurerAdapter sub class
contentSecurityPolicy("
script-src 'self' [URLDomain]/scripts ;
style-src 'self' [URLDomain]/styles;
frame-src 'self' [URLDomain]/frameUrl...
...
.referrerPolicy(ReferrerPolicyHeaderWriter.ReferrerPolicy.STRICT_ORIGIN_WHEN_CROSS_ORIGIN)
Browser will be blocked if you have not define safe external contenet here.
Open the start menu
Type windows+R or open "Run
Execute the following command.
chrome.exe --user-data-dir="C://Chrome dev session" --disable-web-security
I'm using Webdriver.io to download a file continuously
I tried the following code:
var webdriverio = require('webdriverio');
var options = {
desiredCapabilities: {
browserName: 'chrome'
// waitforTimeout: 1000000
}
};
webdriverio
.remote(options)
.init()
.url('https://xxx')
.setValue('#username', ‘xxx#gmail.com’)
.click('#login-submit')
.pause(1000)
.setValue('#password’,’12345’)
.click('#login-submit')
.getTitle().then(function(title){
console.log('Title was: ' + title);
})
.pause(20000)
.getUrl().then(function(url){
console.log('URL: ' + url);
})
.getTitle().then(function(title){
console.log('Title was: ' + title);
})
.click("a[href='/wiki/admin'] button.iwdh")
.getUrl().then(function (url) {
console.log('URL after settings ' + url);
})
.pause(3000)
.scroll('div.jsAtfH',0,1000)
.click("a[href='/wiki/plugins/servlet/ondemandbackup/admin']")
.pause(10000)
.click('//*[#id="backup"]/a')
//.pause(400000)
.end();
Note: The file size is 7GB and how long it will take to download is depend upon the network so instead of using pause() and timeout() is there any way to do it using webdriver.io or node.js ?
To begin with, your current task (waiting for a HUUUUGE file to download) is not a common use-case when it comes to Webdriver-based automation frameworks, WebdriverIO included. Such frameworks aren't meant to download massive files.
First off, you're confusing the waitforTimeout value with WebdriverIO test timeout. Your test is timing out before the .pause() ends.
Currently you're running your tests via the WebdriverIO test-runner. If you want to increase the test timeout, you have to use a different test framework (Mocha, Jasmine, or Cucumber) and set its timeout value to w/e you find appropriate. Going on, I recommend you use Mocha (coming from an ex-Cucumber guy).
You will have to install Mocha: npm install --save-dev wdio-mocha-framework and run your tests with it. Your test should look like this afterwards:
describe("Your Testsuite", function() {
it("\nYour Testcase\n", function() {
return browser
.url('https://xxx')
.setValue('#username', ‘xxx#gmail.com’)
.click('#login-submit')
// rest of the steps
.scroll('div.jsAtfH',0,1000)
.click("a[href='/wiki/plugins/servlet/ondemandbackup/admin']")
.pause(10000)
.click('//*[#id="backup"]/a')
)}
)}
Your config (wdio.conf.js) should contain the following:
framework: 'mocha',
mochaOpts: {
ui: 'bdd',
timeout: 99999999
}
As a side-note, I tried waiting a very long time (> 30 mins) using the above config and had no issues what-so-ever.
Let me know if this helps. Cheers!
If you click on a download button in your browser and you close your browser then your download will be also closed. If you are owning the website where you click on the download button then try to rewrite your code that you have a download able url. Then you can search for a module or way to download files from http url. If you are not the owner and you cant find a url in the href then you can maybe get the generated download url from the network section at your inspector.
Also I never heard that a browser gets closed after timeout? Maybe it comes from webdriver.io I never let my chrome so long open with webdriver.io
You can try to make a workaround use Intervall each 1 Minute as example and then use a webdriver.io command to don´t timeout.
I know it's very old question but I wanted to answer question from comment (and have no such possibility yet). But I will answer main question too.
When i am giving timeout in "wdio.conf.js" file it's not able to
downlaod file it's closing the session but by giving .pause(2000000)
in webdriver.io code it's able to download file of 7GB. What is the
use of timeout in "wdio.conf.js" file if it's kicking out the session
without downlaod?
So this timeout is related to elements state during the test run. So it "determines how long the instance should wait for that element to reach the state".
https://webdriver.io/docs/timeouts.html - this can help. But to answer the question too:
There are more many timeouts such test deals with. Like iamdanchiv wrote for this you should try using one of automatically supported frameworks like Mocha or Jasmine.
IMO right now the easiest way would to do the quick fresh setup using CLI provided by WDIO:
https://webdriver.io/docs/gettingstarted.html
Where you can just simply pick the additional framework you want to use. I would suggest using Jasmine and Chromedriver for this. Than in your wdio.conf.js you can change this part:
waitforTimeout: 10000,
jasmineNodeOpts: {
// Jasmine default timeout
defaultTimeoutInterval: 60000,
//
},
To something that works for you. Or you can use boilerplate projects from wdio page like this one:
https://webdriver.io/docs/boilerplate.html
But that's not all! Still you will have to create some method or function that checks for the file. So check where do you download the file or make it download where you want to and then create a method that uses some kind of wait:
https://webdriver.io/docs/api/browser/waitUntil.html
browser.waitUntil(condition, { timeout, timeoutMsg, interval })
So you can set the timeout either here or in wdio.conf in 'waitforTimeout'. Inside this method condition you can use node filesystem (https://nodejs.org/api/fs.html) to check the state of the file.
This can be helpful to get through waiting for file condition:
https://blog.kevinlamping.com/downloading-files-using-webdriverio/
It seems like after initial config POST request I can't change any properties (webhook and features), what is the way to change those? is this a temporary bug? :)
{
"webhook": "http://example.com/api/kik",
"features": {}
}
what I get
{
"webhook": "http://example.com/incoming",
"features": {
"receiveReadReceipts": false,
"receiveIsTyping": false,
"manuallySendReadReceipts": false,
"receiveDeliveryReceipts": false
}
}
The python library has a set_configuration function, that if called with a Config object with just a webhook set, will set the webhook. See http://kik.readthedocs.org/en/latest/user.html#configuration, though there’s a bug in those docs right now, the seconds function is set_configuration
If you aren't using that library can you show me what code you are using. Blank out your bot-name and api-key
Using Ext, default Ext.Ajax add to GET-request _dc parameter. For example
GET /ConnViewProcessing/?_dc=1263286227619
How to remove this parameter?
PS: it's necessary to manually cache response to ETag and If-None-Match.
Set disableCaching option to false:
Ext.Ajax.disableCaching = false;
Using Ext JS 4.1, and after adding the following code to app.js, the _dc parameter disappears:
// Disable _dc parameter
Ext.Loader.setConfig({
disableCaching: false
});
// My App
Ext.application({
The proper way to accomplish that with Sencha Cmd 6.x is to set a (global) switch in app.json (because all of those hacks and overrides might interfere unnecessarily with the functionality):
"loader": {
"cache": true
},
Then run sencha app refresh, in order to update the application's bootstrap.json.
Alternatively, one can configure Ext.Loader (at run-time):
Ext.Loader.setConfig({disableCaching: false});
When scrolling upwards and reading the actual question, concerning Ext.Ajax (per request):
Ext.Ajax.request({url: '/ConnViewProcessing', disableCaching: false});
The result: no more _dc parameters on scripted requests.
#see Sencha Cmd 6.x - The Microloader.
Note that the use of Ext.Loader has changed in ExtJS 5.
In ExtJS 5, caching can be disabled:
temporarily by adding "?cache" to the end of the URL
by setting a cookie called 'ext-cache' with the value of 1
or by editing the file .sencha/app/Boot.js and setting the '_config.disableCaching' property to be true (overwriting the dynamic lookup).
I am using ExtJS 4.2, but this should work for Ext JS 4.1 and on. In the proxy there is a property called noCache you should set this to false.
Ext4.define('Server',{
extend: 'Ext4.data.Model',
fields: [
{name: 'id'},
{name: 'key'},
{name: 'value'}
],
proxy: {
type: 'rest',
url : 'yaddayaddayadda',
noCache: false,
reader : {
type: 'json'
}
}
});
The reason my code says Ext4. is because I am using the sandbox mode as I move old Ext JS 3x code into 4.2
This should work with extjs 4.0.7:
Ext.Loader.config.disableCaching = false;
Setting the flag disableCaching to false (double negation - yay!) on the Ext.data.Connection should do the trick.
For more, look at the disableCaching-documentation.
(Please note that quite a few classes in Ext seem to have the option available, so you might have to muck around a bit.)
For those that want to set "disableCaching: false" in Sencha Architect 3+, here is how..:
In the project inspector window, select the top node,
"Application"
Then in the "Config" window below that where you
set the object properties, etc, select "Loader Config".. in my case
I had to click the "+" to the right of this as I hadn't set any
items yet. This will create a new "LoaderXX" object in the "Project
Inspector" window above; Loader25 in my case.
Now either select the new object in the
"Project Inspector" window, or click on the right arrow beside the
new "LoaderXX" (Loader25 in my case). This will take you to the
properties for the object.
Untick the "disableCaching" item.
Save the project and refresh the browser window, and enjoy persistent breakpoints, etc, etc in Chrome.
The only way I was able to disable _dc in ExtJS 4.2.x globally on my project:
Ext.define('Ext.data.Connection', {override:'Ext.data.Connection', disableCaching:false });
Ext.define('Ext.data.proxy.Server', {override:'Ext.data.proxy.Server', noCache:false });
Ext.define('Ext.data.JsonP', {override:'Ext.data.JsonP', disableCaching:false });
This is ugly, but any other ideas?
This is how I did this:
Ext.Ajax.request({
url: url,
disableCaching:false
});
I decided that I wanted the cache to be destroyed client side, but server side I was using my own caching mechanism (PHP's APC).
I left the _dc in the Ext ajax request, but then removed it from the REQUEST_URI, and then use the REQUEST_URI as the basis for the cache key
I found this useful: Regular expression to remove one parameter from query string
If you develop under Sencha CMD you can do like this
http://localhost:1841/?disableCacheBuster
or just
http://localhost:1841/?cache
For all who are looking for a way to disable it in a newer version:
proxy: {
type: 'ajax',
noCace: false
}
I use Ext.NET on top of Ext.JS. It adds some more voodoo to Ext.js...
I tried to get rid of the dc= parameter, but all mentioned configurations did not work. So, this is my uber-effective, uber-dirty solution:
Ext.Date.now = function () { return ""; }
As far as I can see, Ext.Date.now() is only used for the caching logic. So it should be relativity save.