Is it possible to use the Admin API to update the minimum number of total instances for a GAE Flex environment?
I've tried both using the client library as well as the web API explorer and I keep getting the 400 response "Frontend automatic scaling should NOT have the following parameter(s): [min_total_instances]"
My update mask is: automaticScaling.min_total_instances
My request body is:
{
"automaticScaling": {
"minTotalInstances": 4
}
I've tried different variants of the update mask and I still get the same error. According to the documentation, this operation should be possible.
This is actually not correctly documented but you need to add the "env": "flex" parameter, since it defaults to standard in the Version's instance in the request body:
{
"automaticScaling": {
"minTotalInstances":
}
"env":"flex"
}
I've raised a documentation update request to make it clearer.
Related
Background story: I need to obtain the handles of the tagged Twitter users from an attached Twitter media. There's no current API method to do that unfortunately (see https://twittercommunity.com/t/how-to-get-tags-of-a-media-in-a-tweet/185614 and https://github.com/twitterdev/open-evolution/issues/34).
I have no other choice but to scrape, this is an example URL: https://twitter.com/justinwood_/status/1626275168157851650/media_tags. This is the page which pops up when you click on the tags link under the media of the parent Tweet: https://twitter.com/justinwood_/status/1626275168157851650/
The React generated DOM is deep and ugly, but would be scrapeable, however I do not want to log in with any account to get banned. Unfortunately when you visit https://twitter.com/justinwood_/status/1626275168157851650/media_tags in an Incognito window the popup shows up dead empty. However when I dig into the network requests the /TweetDetail GraphQL endpoint is full of messages about the anonymous page visit, fortunately it still contains the list of handles I need despite of all of this.
So what I need to have is a scraper which is able to process JavaScript, and capture the response for that specific GraphQL call. Selenium uses a headless Chrome under the hood, so it is able to process JavaScript, and Selenium-Wire offers the ability to capture the response.
Unfortunately my crafted Selenium-Wire script only has the TweetResultByRestId and UsersByRestId GraphQL requests but is missing the TweetDetail. I don't know what to tweak to make all the requests to happen. I iterated over a ton of Chrome options. Here is a variation of my script:
from seleniumwire import webdriver
from selenium.webdriver.chrome.service import Service
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--headless") # for Jenkins
chrome_options.add_argument("--disable-dev-shm-usage") # Jenkins
chrome_options.add_argument('--start-maximized')
chrome_options.add_argument('--window-size=1900,1080')
chrome_options.add_argument('--ignore-certificate-errors-spki-list')
chrome_options.add_argument('--ignore-ssl-errors')
selenium_options = {
'request_storage_base_dir': '/tmp', # Use /tmp to store captured data
'exclude_hosts': ''
}
ser = Service('/usr/bin/chromedriver')
ser.service_args=["--verbose", "--log-path=test.log"]
driver = webdriver.Chrome(service=ser, options=chrome_options, seleniumwire_options=selenium_options)
tweet_id = "1626275168157851650"
twitter_media_url = f"https://twitter.com/justinwood_/status/{tweet_id}/media_tags"
driver.get(twitter_media_url)
driver.wait_for_request("/TweetDetail", timeout=10)
Any ideas?
Apparently it looks like I'd rather need to scrape the parent Tweet URL https://twitter.com/justinwood_/status/1626275168157851650/ and right now it seems my craved GraphQL call happens. Probably I got confused while trying 100 combinations.
I am loading an <iframe> in my HTML page and trying to access the elements within it using JavaScript, but when I try to execute my code, I get the following error:
SecurityError: Blocked a frame with origin "http://www.example.com" from accessing a cross-origin frame.
How can I access the elements in the frame?
I am using this code for testing, but in vain:
$(document).ready(function() {
var iframeWindow = document.getElementById("my-iframe-id").contentWindow;
iframeWindow.addEventListener("load", function() {
var doc = iframe.contentDocument || iframe.contentWindow.document;
var target = doc.getElementById("my-target-id");
target.innerHTML = "Found it!";
});
});
Same-origin policy
You can't access an <iframe> with different origin using JavaScript, it would be a huge security flaw if you could do it. For the same-origin policy browsers block scripts trying to access a frame with a different origin.
Origin is considered different if at least one of the following parts of the address isn't maintained:
protocol://hostname:port/...
Protocol, hostname and port must be the same of your domain if you want to access a frame.
NOTE: Internet Explorer is known to not strictly follow this rule, see here for details.
Examples
Here's what would happen trying to access the following URLs from http://www.example.com/home/index.html
URL RESULT
http://www.example.com/home/other.html -> Success
http://www.example.com/dir/inner/another.php -> Success
http://www.example.com:80 -> Success (default port for HTTP)
http://www.example.com:2251 -> Failure: different port
http://data.example.com/dir/other.html -> Failure: different hostname
https://www.example.com/home/index.html:80 -> Failure: different protocol
ftp://www.example.com:21 -> Failure: different protocol & port
https://google.com/search?q=james+bond -> Failure: different protocol, port & hostname
Workaround
Even though same-origin policy blocks scripts from accessing the content of sites with a different origin, if you own both the pages, you can work around this problem using window.postMessage and its relative message event to send messages between the two pages, like this:
In your main page:
const frame = document.getElementById('your-frame-id');
frame.contentWindow.postMessage(/*any variable or object here*/, 'https://your-second-site.example');
The second argument to postMessage() can be '*' to indicate no preference about the origin of the destination. A target origin should always be provided when possible, to avoid disclosing the data you send to any other site.
In your <iframe> (contained in the main page):
window.addEventListener('message', event => {
// IMPORTANT: check the origin of the data!
if (event.origin === 'https://your-first-site.example') {
// The data was sent from your site.
// Data sent with postMessage is stored in event.data:
console.log(event.data);
} else {
// The data was NOT sent from your site!
// Be careful! Do not use it. This else branch is
// here just for clarity, you usually shouldn't need it.
return;
}
});
This method can be applied in both directions, creating a listener in the main page too, and receiving responses from the frame. The same logic can also be implemented in pop-ups and basically any new window generated by the main page (e.g. using window.open()) as well, without any difference.
Disabling same-origin policy in your browser
There already are some good answers about this topic (I just found them googling), so, for the browsers where this is possible, I'll link the relative answer. However, please remember that disabling the same-origin policy will only affect your browser. Also, running a browser with same-origin security settings disabled grants any website access to cross-origin resources, so it's very unsafe and should NEVER be done if you do not know exactly what you are doing (e.g. development purposes).
Google Chrome
Mozilla Firefox
Safari
Opera: same as Chrome
Microsoft Edge: same as Chrome
Brave: same as Chrome
Microsoft Edge (old non-Chromium version): not possible
Microsoft Internet Explorer
Complementing Marco Bonelli's answer: the best current way of interacting between frames/iframes is using window.postMessage, supported by all browsers
Check the domain's web server for http://www.example.com configuration for X-Frame-Options
It is a security feature designed to prevent clickJacking attacks,
How Does clickJacking work?
The evil page looks exactly like the victim page.
Then it tricked users to enter their username and password.
Technically the evil has an iframe with the source to the victim page.
<html>
<iframe src='victim-domain.example'/>
<input id="username" type="text" style="display: none;"/>
<input id="password" type="text" style="display: none;"/>
<script>
//some JS code that click jacking the user username and input from inside the iframe...
<script/>
<html>
How the security feature work
If you want to prevent web server request to be rendered within an iframe add the x-frame-options
X-Frame-Options DENY
The options are:
SAMEORIGIN: allow only to my own domain render my HTML inside an iframe.
DENY: do not allow my HTML to be rendered inside any iframe
ALLOW-FROM https://example.com/: allow specific domain to render my HTML inside an iframe
This is IIS config example:
<httpProtocol>
<customHeaders>
<add name="X-Frame-Options" value="SAMEORIGIN" />
</customHeaders>
</httpProtocol>
The solution to the question
If the web server activated the security feature it may cause a client-side SecurityError as it should.
For me i wanted to implement a 2-way handshake, meaning:
- the parent window will load faster then the iframe
- the iframe should talk to the parent window as soon as its ready
- the parent is ready to receive the iframe message and replay
this code is used to set white label in the iframe using [CSS custom property]
code:
iframe
$(function() {
window.onload = function() {
// create listener
function receiveMessage(e) {
document.documentElement.style.setProperty('--header_bg', e.data.wl.header_bg);
document.documentElement.style.setProperty('--header_text', e.data.wl.header_text);
document.documentElement.style.setProperty('--button_bg', e.data.wl.button_bg);
//alert(e.data.data.header_bg);
}
window.addEventListener('message', receiveMessage);
// call parent
parent.postMessage("GetWhiteLabel","*");
}
});
parent
$(function() {
// create listener
var eventMethod = window.addEventListener ? "addEventListener" : "attachEvent";
var eventer = window[eventMethod];
var messageEvent = eventMethod == "attachEvent" ? "onmessage" : "message";
eventer(messageEvent, function (e) {
// replay to child (iframe)
document.getElementById('wrapper-iframe').contentWindow.postMessage(
{
event_id: 'white_label_message',
wl: {
header_bg: $('#Header').css('background-color'),
header_text: $('#Header .HoverMenu a').css('color'),
button_bg: $('#Header .HoverMenu a').css('background-color')
}
},
'*'
);
}, false);
});
naturally you can limit the origins and the text, this is easy-to-work-with code
i found this examlpe to be helpful:
[Cross-Domain Messaging With postMessage]
There is a workaround, actually, for specific scenarios.
If you have two processes running on the same domain but different ports, the two Windows can interact without limitations. (i.e. localhost:3000 & localhost:2000). To make this work, each window needs to change their domain to the shared origin:
document.domain = 'localhost'
This also works in the scenario that you are working with different subdomains on the same second-level domain, i.e. you are on john.site.example trying to access peter.site.example or just site.example
document.domain = 'site.example'
By explicitily setting document.domain; the browser will ignore the hostname difference and the Windows can be treated as coming from the 'same-origin'. Now, in a parent window, you can reach into the iframe: frame.contentWindow.document.body.classList.add('happyDev')
If you have control over the content of the iframe - that is, if it is merely loaded in a cross-origin setup such as on Amazon Mechanical Turk - you can circumvent this problem with the <body onload='my_func(my_arg)'> attribute for the inner html.
For example, for the inner html, use the this html parameter (yes - this is defined and it refers to the parent window of the inner body element):
<body onload='changeForm(this)'>
In the inner html :
function changeForm(window) {
console.log('inner window loaded: do whatever you want with the inner html');
window.document.getElementById('mturk_form').style.display = 'none';
</script>
I experienced this error when trying to embed an iframe and then opening the site with Brave. The error went away when I changed to "Shields Down" for the site in question. Obviously, this is not a full solution, since anyone else visiting the site with Brave will run into the same issue. To actually resolve it I would need to do one of the other things listed on this page. But at least I now know where the problem lies.
I would like to add Java Spring specific configuration that can effect on this.
In Web site or Gateway application there is a contentSecurityPolicy setting
in Spring you can find implementation of WebSecurityConfigurerAdapter sub class
contentSecurityPolicy("
script-src 'self' [URLDomain]/scripts ;
style-src 'self' [URLDomain]/styles;
frame-src 'self' [URLDomain]/frameUrl...
...
.referrerPolicy(ReferrerPolicyHeaderWriter.ReferrerPolicy.STRICT_ORIGIN_WHEN_CROSS_ORIGIN)
Browser will be blocked if you have not define safe external contenet here.
Open the start menu
Type windows+R or open "Run
Execute the following command.
chrome.exe --user-data-dir="C://Chrome dev session" --disable-web-security
Our front-end side uploads document to S3 using pre-signed urls and seems to be failing randomly. This part of functionality if very critical to us.
Our pre-signed urls are generated by back-end using boto3.
[...]
#classmethod
def get_presigned_url(cls, filename, user, content_type, size=None):
client = cls.get_s3_client()
import logging
logging.info(cls.generate_keyname(filename, user))
key = cls.generate_keyname(filename, user)
params ={'Bucket': cls.s3_staging_bucket, 'Key': key,
"ContentType": content_type}
if size:
params['ContentLength'] = size
# It's private as default
if cls.is_private:
params['ACL'] = 'private'
else:
params['ACL'] = 'public-read'
return client.generate_presigned_url(
'put_object',
Params=params,
ExpiresIn=600
), cls.get_url(key, cls.s3_staging_bucket)
[...]
So the front-end sends following information to request upload link:
[...]
// Request Presigned url
Restangular.all('upload').all('get_presigned_url').post(
{
'resource_type': 'candidate-cv',
'filename': vm.file.name,
'size': vm.file.size || null,
'content_type': vm.file.type || 'application/octet-stream'
}
).then(
[...]
Things to note in above example: the size and type are not available in all browsers so I have to fallback to defaults.
Once link is retrieved front-end attempts to upload directly to s3 bucket:
[...]
$http.put(
data['presigned_url'],
vm.file,
{
headers: {
'Content-Type': vm.file.type || 'application/octet-stream',
'Authorization': undefined // Needed to remove default ApiKey
}
}
).then(
[...]
The above code gives sometimes -1 response. Sometimes is a problem because it happens a way to often. Probably something around 3% of cases.
We have checked inserted debug logger that sends debug information on every bad response but everything really seems to be alright there.
Our facts so far:
It seemed to me in the beginning that's connectivity issue but should the response status be 0 instead -1?
It happens way too often for connectivity issue (~3%)
It happens on whole range or user agents Windows/Mac Chrome/Edge Mobile/Desk old and new.
It happens with whole range of document formats docx/doc/pdf.
Same users tried several times in a row during 1 hour period all failed with -1.
Same users with same user-agents seem to be able to do upload successfully day before or day after.
We are unable replicate it.
What do we do wrong? What direction should we take to investigate this problem? What next steps should we follow to solve the issue?
Thanks for your input.
EDIT:
As #tcrite suggested that -1 means client side timeout. That seem to be correct to replicate the problem in my local env. We updated production server adding long client timeouts: 250 seconds.
But just recently we have got several -1 responses. The user tried to submit file 6 times in 2 minutes with all resulting with -1 response code and timeout config was present:
Response:
{
"data":null,
"status":-1,
"config":{
"method":"PUT",
"transformRequest":[
null
],
"transformResponse":[
null
],
"jsonpCallbackParam":"callback",
"headers":{
"Content-Type":"application/msword",
"Accept":"application/json, text/plain, */*"
},
"timeout":250000,
"url":"https://stackoverflow-question.s3.amazonaws.com/uploads/files/a-b-a36b9b2f216..."
}
}
It can't be S3 timeout as I tried in my local env to upload file on slow connection for ~5 minutes and it was uploaded sucessfully.
I think you should make a server side web application to upload files ( rather than browse based angular ).Because browser are sometime restricted by company policy.
Check this python django application.I believe youa re already using python.
[https://testdriven.io/blog/storing-django-static-and-media-files-on-amazon-s3/][1]
I try to create a project in gitlab via their API,
with a request (in angular) like this :
$http.post(
"https://gitlab.com/api/v3/projects",
{private_token: <token>}
)
But then I get, as a returned data, a project document with a default_branch : null ... and then it is impossible to update the project by for example post files with the API "https://gitlab.com/api/v3/projects//repository/files" because gitlab will return me an error that I need to be in a specified branch.
Unfortunately a post of a branch with
$http.post(
"https://gitlab.com/api/v3/projects/<projectId>/repository/branches",
{
private_token: <token>
branch_name: "master"
}
)
returns me also an error... because I need to specify also a ref parameter, but it would make any sense if I don't have yet an origin master branch !
If your project is empty (contains no git objects), you should be able to post to "https://gitlab.com/api/v3/projects/repository/files", and specify a branch name, and that will become your default branch. If that isn't working, open a bug on: https://gitlab.com/gitlab-org/gitlab-ce/issues
In my GAE app (Python) I have implemented multitenancy and multi-site support based on the host part of the request object.
For example, www.foo.com/index.html and www.bar.com/index.html are both handled by the same app (e.g. myapp.appspot.com). The app reads the host-value and then decides which namespace and site-configuration to use. This works great as long as the app receives the request directly from the user-agent.
However, I would like to use the Channel API, but there is an issue because requests to /_ah/channel/connected/ and /_ah/channel/disconnected/ are not originating from the original user-agent. Instead, the request has Host: myapp.appspot.com and the parameter to=myapp.appspot.com. (The from parameter is the token I expect. Also www.foo.com/_ah/channel/jsapi is redirected to a talkgadget server which is not documented but seems to be as expected.)
I assume, the problem is caused by the code in channel.js which doesn't call my app using the original host, e.g. www.foo.com/_ah/channel/connected. Instead it uses a talkgadget.google.com host which (as far as I can tell) will then call my app, but using myapp.appspot.com, ignoring the original host and so I cannot use the request's host value for my purpose.
As a workaround, I can figure out a way of including the host information into the channel token, so when my connected and disconnected handlers receive the token, they can use the token instead.
However, I would like to know if there is a better approach, where I could still get the original host name (e.g. www.foo.com) requests to /_ah/channel/connected/ and /_ah/channel/disconnected/. Any ideas?
This is what I have tried so far (with-out any success):
Adding the custom domain host name to the JS src attribute:
<script type="text/javascript" src="//www.foo.com/_ah/channel/jsapi"></script>
I also tried to manually override the base-url of the channel socket, suggested here: https://stackoverflow.com/questions/16558776/google-app-engine-channel-api-same-origin-policy
<script type="text/javascript">
onOpened = function() {
// TODO
};
onMessage = function() {
// TODO
};
onError = function() {
// TODO
};
onClose = function() {
// TODO
};
goog.appengine.Socket.BASE_URL = "https://www.foo.com/_ah/channel/";
channel = new goog.appengine.Channel('{{channelToken}}');
socket = channel.open();
socket.onopen = onOpened;
socket.onmessage = onMessage;
socket.onerror = onError;
socket.onclose = onClose;
</script>
I couldn't find any official documentation for channel.js and I don't want to implement something that is going to break easily with the next update by Google.
Short of a proxy, I don't see a better way than including the information in-band. The problem is that the library/infrastructure (can't be certain without looking deeper) is stripping the HTTP-layer information (the Host header), and indeed you don't have any control of the HTTP layer to pass custom headers, etc. So, you either need to have the info at a lower layer (TCP doesn't even provide a means to do this, and since the entrypoint of your code is through the browser running channel.js, rather than a system-level process running on the bare network interface, this is out of the picture decisively), or at a higher layer, ie. within the channel.