HTTPS Request Not Working in Google App Engine (Python Requests lib) / Cloud Shell - google-app-engine

I have been trying to do some html scraping for one of my projects, wherein the it is working from my local computer, but it is now working in google app engine. It is getting timed out.
I have tried using dig from google cloud shell, which is resolving the correct IP but still curl / trace is failing. Billing is enabled, haven't touched any DNS setting in GAE.
The target site is apparently using chunked reply.
from flask import Flask, render_template, request, send_file, redirect, url_for, Response, send_from_directory
import flask
import requests
import bs4
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
import logging
import os
s = requests.Session()
r = s.get("https://kdseodb.smportkolkata.in/ccuPomsPosWeb/index.jsp")
id1 = r.cookies['JSESSIONID']
r2 = s.get("https://kdseodb.smportkolkata.in/ccuPomsPosWeb/apps/dos/PosCTS.xhtml")
id2 = r2.cookies['oam.Flash.RENDERMAP.TOKEN']
headers = {'Accept': 'application/xml, text/xml, */*; q=0.01', 'Accept-Language': 'en-US,en;q=0.9', 'Connection': 'keep-alive', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Faces-Request': 'partial/ajax', 'Sec-Fetch-Dest': 'empty', 'Sec-Fetch-Mode': 'cors', 'Sec-Fetch-Site': 'same-origin', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36', 'X-Requested-With': 'XMLHttpRequest', 'Cookie': 'JSESSIONID='+id1+'; oam.Flash.RENDERMAP.TOKEN=-'+id2+''}
payload = 'javax.faces.partial.ajax=true&javax.faces.source=form1%3AbtnSearch&javax.faces.partial.execute=form1&javax.faces.partial.render=form1&form1%3AbtnSearch=form1%3AbtnSearch&form1%3AcomMloCd_focus=&form1%3AcomMloCd_input=NONE&form1%3AtxtCntNo=ABCD1234567&form1%3AtblData_scrollState=0%2C0&form1_SUBMIT=1&javax.faces.ViewState=PCsvYkkMUQAmP0zfu2tj7G5AqM%2BXrDH8%2BaG%2FROvEtWIHJyXX'
r3 = s.post("https://kdseodb.smportkolkata.in/ccuPomsPosWeb/apps/dos/PosCTS.xhtml", cookies=r.cookies, headers=headers, data=payload)
soup = bs4.BeautifulSoup(r3.text)
dig output from google cloud shell, NXDOMAIN is coming up when used on the IPs
dig kdseodb.smportkolkata.in
; <<>> DiG 9.16.33-Debian <<>> kdseodb.smportkolkata.in
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 414
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;kdseodb.smportkolkata.in. IN A
;; ANSWER SECTION:
kdseodb.smportkolkata.in. 3600 IN A 117.242.152.158
kdseodb.smportkolkata.in. 3600 IN A 203.153.37.38
;; Query time: 6 msec
;; SERVER: 169.254.169.254#53(169.254.169.254)
;; WHEN: Thu Dec 29 01:44:36 UTC 2022
;; MSG SIZE rcvd: 85
dig 117.242.152.158
; <<>> DiG 9.16.33-Debian <<>> 117.242.152.158
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 29831
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;117.242.152.158. IN A
;; AUTHORITY SECTION:
. 86400 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2022122801 1800 900 604800 86400
;; Query time: 5 msec
;; SERVER: 169.254.169.254#53(169.254.169.254)
;; WHEN: Thu Dec 29 01:45:39 UTC 2022
;; MSG SIZE rcvd: 119
dig 203.153.37.38
; <<>> DiG 9.16.33-Debian <<>> 203.153.37.38
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 32307
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;203.153.37.38. IN A
;; AUTHORITY SECTION:
. 86400 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2022122801 1800 900 604800 86400
;; Query time: 7 msec
;; SERVER: 169.254.169.254#53(169.254.169.254)
;; WHEN: Thu Dec 29 01:46:13 UTC 2022
;; MSG SIZE rcvd: 117

Related

JMeter & the JDBC Driver Using Docker - "DBCP DataSource configured without a 'password'" mssql

This is based on the answer in JMeter & Gitlab: How to Install & SQLServer JDBC Driver?.
In short, the debug output below shows that the appropriate JAR and DLL are in the Java-path on the Docker image, yet I am seeing the following error in the JMeter output:
"DBCP DataSource configured without a 'password'" mssql
"DBCP DataSource configured without a 'user'" mssql
The image uses JMeter 5.4.2 ("latest").
The JDBC driver files I am using are:
mssql-jdbc_auth-9.4.1.x86.dll
mssql-jdbc-9.4.1.jre8.jar
So is it a compatibility issue?
I have JMeter 5.4.1 installed on my computer, and the test plan works fine using the local application.
Here is the output from the Gitlab job:
[0KRunning with gitlab-runner 13.5.0 (ece86343)
[0;m[0K on qaildk03 VFwBJtXZ
[0;msection_start:1641760700:prepare_executor
[0K[0K[36;1mPreparing the "docker" executor[0;m
[0;m[0KUsing Docker executor with image guythedocker/jmeter_mssql:latest ...
[0;m[0KPulling docker image guythedocker/jmeter_mssql:latest ...
[0;m[0KUsing docker image sha256:7ebf2689ca4b15d4053c8148055524bfaf8173b76d2d66f45d0a13ce6992b901 for guythedocker/jmeter_mssql:latest with digest guythedocker/jmeter_mssql#sha256:316e3ce3f37c174147b10ce541e66121a4d5017f597f862b64cb081fae4035b8 ...
[0;msection_end:1641760705:prepare_executor
[0Ksection_start:1641760705:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mRunning on runner-vfwbjtxz-project-516-concurrent-0 via qaildk03...
section_end:1641760707:prepare_script
[0Ksection_start:1641760707:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Reinitialized existing Git repository in /builds/guyl/api-performance-test/.git/
[32;1mChecking out 89b301b5 as develop...[0;m
Removing public/
Removing testresults/
[32;1mSkipping Git submodules setup[0;m
section_end:1641760709:get_sources
[0Ksection_start:1641760709:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m
[0;m[32;1m$ ls /opt/apache-jmeter-*/lib/[0;m
etc
mssql-jdbc-9.4.1.jre8.jar <------------------
mssql-jdbc_auth-9.4.1.x86.dll <------------------
etc
[32;1m$ env[0;m
CI_PROJECT_NAMESPACE=guyl
GITLAB_USER_ID=128
CI_RUNNER_VERSION=13.5.0
FF_SKIP_NOOP_BUILD_STAGES=true
LYNX_OP_USERS=50
CI_SERVER_NAME=GitLab
CI_RUNNER_DESCRIPTION=qaildk03
GITLAB_USER_EMAIL=guyl#xxxx.com
CI_SERVER_REVISION=4469fca16f1
CI_RUNNER_EXECUTABLE_ARCH=linux/amd64
CI_REGISTRY_USER=gitlab-ci-token
CI_API_V4_URL=https://gitlab.xxxx.net/api/v4
CI_REGISTRY_PASSWORD=[MASKED]
CI_RUNNER_SHORT_TOKEN=VFwBJtXZ
CI_JOB_NAME=api test
HOSTNAME=runner-vfwbjtxz-project-516-concurrent-0
GITLAB_USER_LOGIN=guyl
OCTO_KEY=[MASKED]
CI_PROJECT_NAME=api-performance-test
CI_PIPELINE_SOURCE=schedule
CI_BUILD_REF_NAME=develop
CI_JOB_STATUS=running
MONYX_LOOPS=100
FF_SHELL_EXECUTOR_USE_LEGACY_PROCESS_KILL=false
CI_PIPELINE_ID=74075
CI_BUILD_REF_SLUG=develop
CI_COMMIT_REF_SLUG=develop
CI_SERVER=yes
CI_COMMIT_SHORT_SHA=89b301b5
FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION=false
CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX=gitlab.xxxx.net:443/guyl/dependency_proxy/containers
PWD=/builds/guyl/api-performance-test
CI_RUNNER_TAGS=
CI_PROJECT_PATH=guyl/api-performance-test
RAMPUP=
CI_SERVER_TLS_CA_FILE=/builds/guyl/api-performance-test.tmp/CI_SERVER_TLS_CA_FILE
MOMA2_LOOPS=0
TZ=Europe/Amsterdam
FF_RESET_HELPER_IMAGE_ENTRYPOINT=true
CI_COMMIT_REF_PROTECTED=true
CI_SERVER_VERSION_MINOR=8
CI_COMMIT_SHA=89b301b5c6370b45bbd7770e1d827fa334a5300d
HOME=/root
FF_NETWORK_PER_BUILD=false
CI_DEPENDENCY_PROXY_PASSWORD=[MASKED]
CI_PROJECT_VISIBILITY=internal
CI_CONCURRENT_PROJECT_ID=0
JMETER_HOME=/opt/apache-jmeter-5.4.2 <------------------------
CI_COMMIT_MESSAGE=Update .gitlab-ci.yml file
USERS=
CI_BUILD_NAME=api test
FF_USE_DIRECT_DOWNLOAD=true
CI_PAGES_DOMAIN=gitlab-pages.xxxx.net
CI_SERVER_VERSION=13.8.6
CI_SERVER_PORT=443
CURRENT_VERSION=10.2
MOMA2_USERS=2
JMETER_DOWNLOAD_URL=https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-5.4.2.tgz
CI_PAGES_URL=https://guyl.gitlab-pages.xxxx.net/api-performance-test
CI_PIPELINE_IID=238
CI_REPOSITORY_URL=https://gitlab-ci-token:[MASKED]#gitlab.xxxx.net/guyl/api-performance-test.git
CI_SERVER_URL=https://gitlab.xxxx.net
GITLAB_FEATURES=
ENGINE_LOOPS=100
CI_COMMIT_DESCRIPTION=
MOMA2_RAMPUP=6
CI_JOB_STAGE=test
CI_PIPELINE_URL=https://gitlab.xxxx.net/guyl/api-performance-test/-/pipelines/74075
ENGINE_RAMPUP=9
CI_DEFAULT_BRANCH=develop
ENGINE_USERS=0
CI_BUILD_REF=89b301b5c6370b45bbd7770e1d827fa334a5300d
CI_SERVER_VERSION_PATCH=6
CI_COMMIT_TITLE=Update .gitlab-ci.yml file
CI_PROJECT_ROOT_NAMESPACE=guyl
MONYX_USERS=100
GITLAB_USER_NAME=Guy Levin
CI_PROJECT_DIR=/builds/guyl/api-performance-test
LYNX_OP_LOOPS=0
SHLVL=2
CI_RUNNER_ID=24
CI_COMMIT_TIMESTAMP=2022-01-09T20:36:44+00:00
CI_DISPOSABLE_ENVIRONMENT=true
CI_JOB_JWT=[MASKED]
CI_BUILD_ID=169407
CI_SERVER_PROTOCOL=https
LYNX_OP_RAMPUP=25
JMETER_BIN=/opt/apache-jmeter-5.4.2/bin <---------------------------------
CI_COMMIT_REF_NAME=develop
CI_SERVER_HOST=gitlab.xxxx.net
CI_JOB_URL=https://gitlab.xxxx.net/guyl/api-performance-test/-/jobs/169407
CI_JOB_TOKEN=[MASKED]
CI_CONCURRENT_ID=0
CI_BUILD_TOKEN=[MASKED]
CI_COMMIT_BRANCH=develop
FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=true
CI_RUNNER_REVISION=ece86343
CI_BUILD_BEFORE_SHA=0000000000000000000000000000000000000000
CI_DEPENDENCY_PROXY_USER=gitlab-ci-token
MONYX_RAMPUP=0
CI_PROJECT_PATH_SLUG=guyl-api-performance-test
CI_PROJECT_CONFIG_PATH=.gitlab-ci.yml
CI_NODE_TOTAL=1
CI_BUILDS_DIR=/builds
CI_JOB_ID=169407
CI_PROJECT_REPOSITORY_LANGUAGES=javascript,batchfile
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/apache-jmeter-5.4.2/bin <-------------
CI_PROJECT_ID=516
CI=true
GITLAB_CI=true
CI_JOB_IMAGE=guythedocker/jmeter_mssql:latest
CI_COMMIT_BEFORE_SHA=0000000000000000000000000000000000000000
CI_PROJECT_TITLE=api-performance-test
CI_SERVER_VERSION_MAJOR=13
CI_CONFIG_PATH=.gitlab-ci.yml
CI_BUILD_STAGE=test
CI_DEPENDENCY_PROXY_SERVER=gitlab.xxxx.net:443
CI_PROJECT_URL=https://gitlab.xxxx.net/guyl/api-performance-test
OLDPWD=/opt/apache-jmeter-5.4.2
_=/usr/bin/env
[32;1m$ echo $PATH[0;m
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/apache-jmeter-5.4.2/bin <-----------
[32;1m$ /entrypoint.sh --version[0;m
START Running Jmeter on Sun Jan 9 21:38:30 CET 2022
JVM_ARGS=-Xmn502m -Xms2008m -Xmx2008m
jmeter args=--version
jmeter ALL ARGS=-Dlog4j2.formatMsgNoLookups=true --version
Jan 09, 2022 9:38:32 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
_ ____ _ ____ _ _ _____ _ __ __ _____ _____ _____ ____
/ \ | _ \ / \ / ___| | | | ____| | | \/ | ____|_ _| ____| _ \
/ _ \ | |_) / _ \| | | |_| | _| _ | | |\/| | _| | | | _| | |_) |
/ ___ \| __/ ___ \ |___| _ | |___ | |_| | | | | |___ | | | |___| _ <
/_/ \_\_| /_/ \_\____|_| |_|_____| \___/|_| |_|_____| |_| |_____|_| \_\ 5.4.2
Copyright (c) 1999-2021 The Apache Software Foundation
END Running Jmeter on Sun Jan 9 21:38:33 CET 2022
[32;1m$ mkdir testresults[0;m
[32;1m$ mkdir testresults/monyx[0;m
[32;1m$ mkdir testresults/moma2[0;m
[32;1m$ mkdir testresults/lynx_operational[0;m
[32;1m$ mkdir testresults/engine[0;m
[32;1m$ /entrypoint.sh -n -t ./JMeter/Monyx_API/Monyx_API.jmx -l ./monyx.log -e -o ./testresults/monyx -Jthreads=$MONYX_USERS -Jrampup=$MONYX_RAMPUP -JtestCases=$MONYX_LOOPS -Jhost=qa2-payapi.xxxx.com -f # collapsed multi-line command[0;m
START Running Jmeter on Sun Jan 9 21:38:33 CET 2022
JVM_ARGS=-Xmn502m -Xms2008m -Xmx2008m
jmeter args=-n -t ./JMeter/Monyx_API/Monyx_API.jmx -l ./monyx.log -e -o ./testresults/monyx -Jthreads=100 -Jrampup=0 -JtestCases=100 -Jhost=qa2-payapi.xxxx.com -f
jmeter ALL ARGS=-Dlog4j2.formatMsgNoLookups=true -n -t ./JMeter/Monyx_API/Monyx_API.jmx -l ./monyx.log -e -o ./testresults/monyx -Jthreads=100 -Jrampup=0 -JtestCases=100 -Jhost=qa2-payapi.xxxx.com -f
Creating summariser <summary>
Created the tree successfully using ./JMeter/Monyx_API/Monyx_API.jmx
Starting standalone test # Sun Jan 09 21:38:35 CET 2022 (1641760715608)
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
summary + 13617 in 00:01:23 = 163.8/s Avg: 577 Min: 5 Max: 49132 Err: 10000 (73.44%) Active: 100 Started: 300 Finished: 200
summary + 5932 in 00:00:30 = 197.2/s Avg: 503 Min: 256 Max: 1501 Err: 0 (0.00%) Active: 98 Started: 300 Finished: 202
summary = 19549 in 00:01:53 = 172.7/s Avg: 555 Min: 5 Max: 49132 Err: 10000 (51.15%)
summary + 489 in 00:00:31 = 15.7/s Avg: 2068 Min: 251 Max: 28079 Err: 0 (0.00%) Active: 100 Started: 400 Finished: 300
summary = 20038 in 00:02:24 = 138.7/s Avg: 591 Min: 5 Max: 49132 Err: 10000 (49.91%)
summary + 93 in 00:00:30 = 3.1/s Avg: 30500 Min: 19297 Max: 47158 Err: 0 (0.00%) Active: 100 Started: 400 Finished: 300
summary = 20131 in 00:02:54 = 115.7/s Avg: 730 Min: 5 Max: 49132 Err: 10000 (49.67%)
summary + 96 in 00:00:34 = 2.8/s Avg: 34369 Min: 22688 Max: 45334 Err: 0 (0.00%) Active: 100 Started: 400 Finished: 300
summary = 20227 in 00:03:28 = 97.3/s Avg: 889 Min: 5 Max: 49132 Err: 10000 (49.44%)
summary + 79 in 00:00:25 = 3.1/s Avg: 35523 Min: 23972 Max: 45774 Err: 0 (0.00%) Active: 100 Started: 400 Finished: 300
summary = 20306 in 00:03:53 = 87.1/s Avg: 1024 Min: 5 Max: 49132 Err: 10000 (49.25%)
summary + 81 in 00:00:33 = 2.4/s Avg: 36941 Min: 24322 Max: 46364 Err: 0 (0.00%) Active: 100 Started: 400 Finished: 300
etc
summary = 63674 in 01:35:23 = 11.1/s Avg: 8779 Min: 0 Max: 114698 Err: 18594 (29.20%)
summary + 613 in 00:00:30 = 20.4/s Avg: 4952 Min: 0 Max: 15629 Err: 204 (33.28%) Active: 100 Started: 600 Finished: 500
summary = 64287 in 01:35:53 = 11.2/s Avg: 8742 Min: 0 Max: 114698 Err: 18798 (29.24%)
DBCP DataSource configured without a 'username'
DBCP DataSource configured without a 'password'
etc
DBCP DataSource configured without a 'password'
DBCP DataSource configured without a 'username'
DBCP DataSource configured without a 'password'
DBCP DataSource configured withosummary + 377 in 00:00:30 = 12.6/s Avg: 7965 Min: 810 Max: 15971 Err: 188 (49.87%) Active: 100 Started: 1700 Finished: 1600
summary = 257193 in 02:40:23 = 26.7/s Avg: 3659 Min: 0 Max: 114698 Err: 61102 (23.76%)
summary + 388 in 00:00:30 = 12.9/s Avg: 7759 Min: 786 Max: 15878 Err: 191 (49.23%) Active: 100 Started: 1700 Finished: 1600
etc
summary + 3850 in 00:00:30 = 128.3/s Avg: 715 Min: 24 Max: 1653 Err: 3850 (100.00%) Active: 100 Started: 2700 Finished: 2600
summary = 383734 in 03:09:53 = 33.7/s Avg: 2835 Min: 0 Max: 114698 Err: 164923 (42.98%)
summary + 3864 in 00:00:30 = 128.7/s Avg: 765 Min: 44 Max: 3479 Err: 3864 (100.00%) Active: 100 Started: 2700 Finished: 2600
summary = 387598 in 03:10:23 = 33.9/s Avg: 2814 Min: 0 Max: 114698 Err: 168787 (43.55%)
summary + 2405 in 00:00:19 = 128.8/s Avg: 707 Min: 24 Max: 1436 Err: 2405 (100.00%) Active: 0 Started: 1 Finished: 1
summary = 390003 in 03:10:42 = 34.1/s Avg: 2801 Min: 0 Max: 114698 Err: 171192 (43.90%)
Tidying up ... # Mon Jan 10 00:49:18 CET 2022 (1641772158705)
... end of run
END Running Jmeter on Mon Jan 10 00:49:44 CET 2022
And here is the Dockerfile:
# inspired by https://github.com/hauptmedia/docker-jmeter and
# https://github.com/hhcordero/docker-jmeter-server/blob/master/Dockerfile
FROM alpine:3.12
ARG JMETER_VERSION="5.4.2"
ENV JMETER_HOME /opt/apache-jmeter-${JMETER_VERSION}
ENV JMETER_BIN ${JMETER_HOME}/bin
ENV JMETER_DOWNLOAD_URL https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-${JMETER_VERSION}.tgz
# Install extra packages
# Set TimeZone, See: https://github.com/gliderlabs/docker-alpine/issues/136#issuecomment-612751142
ARG TZ="Europe/Amsterdam"
ENV TZ ${TZ}
RUN apk update \
&& apk upgrade \
&& apk add ca-certificates \
&& update-ca-certificates \
&& apk add --update openjdk8-jre tzdata curl unzip bash \
&& apk add --no-cache nss \
&& rm -rf /var/cache/apk/* \
&& mkdir -p /tmp/dependencies \
&& curl -L --silent ${JMETER_DOWNLOAD_URL} > /tmp/dependencies/apache-jmeter-${JMETER_VERSION}.tgz \
&& mkdir -p /opt
RUN tar -xzf /tmp/dependencies/apache-jmeter-${JMETER_VERSION}.tgz -C /opt
RUN mkdir -p ${JMETER_HOME}/lib
RUN curl --ssl-no-revoke -L https://repo1.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc/9.4.1.jre8/mssql-jdbc-9.4.1.jre8.jar -o mssql-jdbc-9.4.1.jre8.jar
RUN curl --ssl-no-revoke -L https://repo1.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc_auth/9.4.1.x86/mssql-jdbc_auth-9.4.1.x86.dll -o mssql-jdbc_auth-9.4.1.x86.dll
RUN rm -rf /tmp/dependencies
COPY ./mssql-jdbc-9.4.1.jre8.jar /opt/apache-jmeter-${JMETER_VERSION}/lib/
COPY ./mssql-jdbc_auth-9.4.1.x86.dll /opt/apache-jmeter-${JMETER_VERSION}/lib/
# TODO: plugins (later)
# && unzip -oq "/tmp/dependencies/JMeterPlugins-*.zip" -d $JMETER_HOME
# Set global PATH such that "jmeter" command is found
ENV PATH $PATH:$JMETER_BIN
# Entrypoint has same signature as "jmeter" command
COPY entrypoint.sh /
WORKDIR ${JMETER_HOME}
ENTRYPOINT ["/entrypoint.sh"]
JMeter Report, showing errors:
I don't think you can use SQL Server integrated authentication from Linux machine.
In order to be able to use the approach with the Microsoft JDBC Driver and integrated authentication you need to run your builds on a Windows Runner
In case you have to use Linu you can work it around using JTDS JDBC Driver and connection string like:
jdbc:jtds:sqlserver://server_name:1433;databaseName=your-database;domain=your-domain;useNTLMv2=true;
and specify your username and password in the appropriate sections of the JDBC Connection Configuration.
Also be aware that according to 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure article you should always be using the latest version of JMeter so consider upgrading to JMeter 5.4.3 (or whatever is the latest version which is available at JMeter Downloads page) on next available opportunity.

Operation timed out after 10000 milliseconds with 11584 out of 1526223 bytes received

I am trying to get about 1.4 to 2 MB of JSON data from a server using the curl library in C.
I have increased the Buffer size, connection timeout
curl_easy_setopt(conn->easy, CURLOPT_CONNECTTIMEOUT, 10);
curl_easy_setopt(conn->easy, CURLOPT_BUFFERSIZE, 524288);
#ifdef CURL_MAX_WRITE_SIZE
#undef CURL_MAX_WRITE_SIZE
#define CURL_MAX_WRITE_SIZE 524288
#endif
I did take a capture on the client I am running and the packets are not received completely.
I also tried the url from a browser and the browser is able to get the data and display.
Any help on this is appreciated.
Thanks
So basically my code is based on the example https://curl.haxx.se/libcurl/c/crawler.html
The difference here is that the timer_cb is called frequently and the easy_handle is released only if there is something to read.
static void timer_cb(int fd, short kind, void * userp)
{
GlobalInfo * g = (GlobalInfo *)userp;
CURLMcode rc;
(void)fd;
(void)kind;
rc = curl_multi_socket_action(g->multi,
CURL_SOCKET_TIMEOUT, 0, &g->still_running);
mcode_or_die("timer_cb: curl_multi_socket_action", rc);
check_multi_info(g);
}
I read the response here in this api and new_conn_init will create/malloc a new easy handle and call the curl_multi_add_handle.
static void check_multi_info(GlobalInfo * g)
{
char * eff_url;
CURLMsg * msg;
int msgs_left;
ConnInfo * conn = NULL;
CURL * easy;
.....
while ((msg = curl_multi_info_read(g->multi, &msgs_left)))
{
if (msg->msg == CURLMSG_DONE)
{
easy = msg->easy_handle;
res = msg->data.result;
do
{
rc = ERROR;
conn = NULL;
contentType = NULL;
iValidContent = 1;
..........
}
while (0);
u8 ret = 0;
curl_multi_remove_handle(g->multi, easy);
curl_easy_cleanup(easy);
if (conn)
new_conn_init(&(conn->reqInfo), g, 1);
}
new_conn_init ()
{
conn->easy = curl_easy_init();
curl_easy_setopt(conn->easy, CURLOPT_URL, conn->url);
curl_easy_setopt(conn->easy, CURLOPT_BUFFERSIZE, 524288);
curl_easy_setopt(conn->easy, CURLOPT_WRITEFUNCTION, write_cb);
curl_easy_setopt(conn->easy, CURLOPT_WRITEDATA, &(conn->respData));
// curl_easy_setopt(conn->easy, CURLOPT_HEADERDATA, &(conn->headerData));
// curl_easy_setopt(conn->easy, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(conn->easy, CURLOPT_VERBOSE, (long)shmHttpClientConfig->debugLibCurl);
curl_easy_setopt(conn->easy, CURLOPT_FORBID_REUSE, 1L);
curl_easy_setopt(conn->easy, CURLOPT_ERRORBUFFER, conn->error);
curl_easy_setopt(conn->easy, CURLOPT_PRIVATE, conn);
curl_easy_setopt(conn->easy, CURLOPT_PROTOCOLS, CURLPROTO_HTTP | CURLPROTO_HTTPS);
curl_easy_setopt(conn->easy, CURLOPT_DNS_CACHE_TIMEOUT, -1);
curl_easy_setopt(conn->easy, CURLOPT_NOPROGRESS, 1);
curl_easy_setopt(conn->easy, CURLOPT_CONNECTTIMEOUT, 10);
rc = curl_multi_add_handle(g->multi, conn->easy);
}
I did some debugging and looks like the connections are getting closed quite late in if you can see the op of the curl debug. I am sending url queries every 20 seconds.
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#1)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:37:53 GMT
<
* Found bundle for host 12.0.0.25: 0x9f8d20
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#2)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:38:13 GMT
<
* Found bundle for host 12.0.0.25: 0x9f8d20
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#3)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:38:33 GMT
<
* Found bundle for host 12.0.0.25: 0x9f8d20
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#4)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
* Operation timed out after 10000 milliseconds with 160728 out of 1526223 bytes received
* Closing connection 1
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:38:53 GMT
<
* Closing connection 4
* Found bundle for host 12.0.0.25: 0x9f8d20
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Operation timed out after 10001 milliseconds with 11584 out of 1526223 bytes received
* Closing connection 2
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#5)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:39:13 GMT
<
* Closing connection 5
* Operation timed out after 10001 milliseconds with 845632 out of 1526223 bytes received
* Closing connection 3
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#6)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:43:33 GMT
<
* Found bundle for host 12.0.0.25: 0x9f31f0
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#7)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< C
[root#BENU TEMP/MEG-1] ~# cat /opt/benu-data/admin/benu_libcurl_op.txt | more
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#1)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:37:53 GMT
<
* Found bundle for host 12.0.0.25: 0x9f8d20
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#2)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:38:13 GMT
<
* Found bundle for host 12.0.0.25: 0x9f8d20
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#3)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:38:33 GMT
<
* Found bundle for host 12.0.0.25: 0x9f8d20
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#4)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*
* Operation timed out after 10000 milliseconds with 160728 out of 1526223 bytes received
* Closing connection 1
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 1526223
< Content-Type: application/json; charset=utf-8
< Date: Thu, 11 Apr 2019 11:38:53 GMT
<
* Closing connection 4
* Found bundle for host 12.0.0.25: 0x9f8d20
* Hostname 12.0.0.25 was found in DNS cache
* Trying 12.0.0.25...
* Hostname 12.0.0.29 was found in DNS cache
* Name '12.0.0.29' family 2 resolved to '12.0.0.29' family 2
* Local port: 0
* Operation timed out after 10001 milliseconds with 11584 out of 1526223 bytes received
* Closing connection 2
* Connected to 12.0.0.25 (12.0.0.25) port 1234 (#5)
> GET /xxxx HTTP/1.1
Host: 12.0.0.25:1234
Accept: */*

CORS enabled but still getting CORS error

I'm trying to get a JSON object from an API and the devs for the API said they just enabled CORS but I'm stilling getting the bellow error.
XMLHttpRequest cannot load http://example.com/data/action/getGame/9788578457657. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://dev.our-domain.local' is therefore not allowed access.
I'm using AngularJS to get the JSON in a Service with
app.service("gameService", function ($http, $q)
{
function getGame(GameId) {
var deferred = $q.defer()
var url = 'http://example.com/data/action/getGame/' + gameId;
// var url = 'https://jsonplaceholder.typicode.com/albums/' + gameId; // THIS WORKS
$http({
method: 'GET',
cache: true,
url: url,
headers: {
'Content-Type': 'application/json;charset=UTF-8'
}
}).
then(function(response) {
//your code when success
deferred.resolve(response);
console.log('gameService HTTP CORS SUCCESS!');
}, function(response) {
//your code when fails
console.log('gameService HTTP CORS ERROR!');
// deferred.resolve('');
deferred.reject(response);
});
return deferred.promise;
}
this.getGame = getGame;
})
My AngularJS service works when I test it with jsonplaceholder which has CORS enabled.
Am I missing something?
The API devs said that two CORS-Headers are added to data.service responses but I don't see them. This is what I see on the headers when I curl down the JSON object.
$ curl -X HEAD -i http://example.com/data/action/getGame/9788578457657
HTTP/1.1 200 OK
Date: Wed, 14 Dec 2016 10:39:17 GMT
Server: WildFly/8
Expires: Wed, 14 Dec 2016 10:39:17 GMT
X-Powered-By: Undertow/1
X-dmg-elapsed-time: 20ms
X-dmg-host-address: 1??.??.???.??
Vary: Accept-Encoding,Origin
X-dmg-generated-time: Wed, 14 Dec 2016 10:39:17 GMT
Content-Type: application/json;charset=UTF-8
Content-Language: en-
X-dmg-node-name: defg_node_1
X-Varnish-Bereq-Backend: real_backend_foo_bar_uk
X-Varnish-Bereq-Retries: 0
Last-Modified: Wed, 14 Dec 2016 10:39:17 GMT
Cache-Control: public, max-age=300
X-Varnish: 6876870
Age: 0
Via: 1.1 varnish-v4
X-Varnish-Cache: MISS
X-Varnish-Trimen: www.trimen.com
X-Varnish-Served-By-Host: snarf.foo.uk
X-Varnish-Served-By-IP: 100.100.10.80
X-Varnish-Pool: http_pages
X-Varnish-Req-Backend-Hint: dead
X-Varnish-Req-Restarts: 0
X-Varnish-Hash: /data/action/getGame/9788578457657
X-Varnish-Backend-Ourself: varnish_server_snarf_foo_uk
X-DMG-Version: 6.20.51.2358
Accept-Ranges: none
Connection: keep-alive
Is this what I should be seeing with CORS enabled or is there something more?
Do I need to add more to my AngularJS Service to http get with Cors enabled, as add in more to:
headers: {
'Content-Type': 'application/json;charset=UTF-8'
}
UPDATE
Passing Origin: in the header on my curl request as suggested by #t.niese
$ curl -H "Origin: http://our-production-domain.com/" --verbose \
> http://example.com/data/action/getGame/9788578457657
* Trying 1?.???.??.???...
* Connected to http://example.com/ (1?.???.??.???) port 80 (#0)
> GET /data/action/getGame/9788578457657 HTTP/1.1
> Host: http://example.com/
> User-Agent: curl/7.43.0
> Accept: */*
> Origin: http://our-production-domain.com/
>
< HTTP/1.1 200 OK
< Date: Wed, 14 Dec 2016 11:05:24 GMT
< Server: WildFly/8
< Expires: Wed, 14 Dec 2016 11:05:24 GMT
< X-Powered-By: Undertow/1
< X-dmg-elapsed-time: 27ms
< X-dmg-host-address: 1??.??.???.??
< Vary: Accept-Encoding,Origin
< X-dmg-generated-time: Wed, 14 Dec 2016 11:05:24 GMT
< Content-Type: application/json;charset=UTF-8
< Content-Language: en-
< X-dmg-node-name: defg_node_1
< X-Varnish-Bereq-Backend: real_backend_foo_bar_uk
< X-Varnish-Bereq-Retries: 0
< Last-Modified: Wed, 14 Dec 2016 11:05:24 GMT
< Cache-Control: public, max-age=300
< X-Varnish: 6876870
< Age: 0
< Via: 1.1 varnish-v4
< X-Varnish-Cache: MISS
< X-Varnish-Trimen: www.trimen.com
< X-Varnish-Served-By-Host: snarf.foo.uk
< X-Varnish-Served-By-IP: 100.100.10.80
< X-Varnish-Pool: http_pages
< X-Varnish-Req-Backend-Hint: dead
< X-Varnish-Req-Restarts: 0
< X-Varnish-Hash: /data/action/getGame/9788578457657
< X-Varnish-Backend-Ourself: varnish_server_snarf_foo_uk
< X-DMG-Version: 6.20.51.2358
< Accept-Ranges: none
< Transfer-Encoding: chunked
< Connection: keep-alive
<
{
"errorMessage" : null,
"expiry" : "2016-12-14T11:05:24.379+0000",
"data" : {
// json object data here
}
* Connection #0 to host http://example.com/ left intact
}
and..
$ curl -H "Origin: http://qa.our-qa-domain.com/" --verbose \
> http://example.com/data/action/getGame/9788578457657
* Trying 1?.???.??.???...
* Connected to http://example.com/ (1?.???.??.???) port 80 (#0)
> GET /data/action/getGame/9788578457657 HTTP/1.1
> Host: http://example.com/
> User-Agent: curl/7.43.0
> Accept: */*
> Origin: http://qa.our-qa-domain.com/
>
< HTTP/1.1 200 OK
< Date: Wed, 14 Dec 2016 11:06:11 GMT
< Server: WildFly/8
< Expires: Wed, 14 Dec 2016 11:06:11 GMT
< X-Powered-By: Undertow/1
< X-dmg-elapsed-time: 18ms
< X-dmg-host-address: 1??.??.???.??
< Vary: Accept-Encoding,Origin
< X-dmg-generated-time: Wed, 14 Dec 2016 11:06:11 GMT
< Content-Type: application/json;charset=UTF-8
< Content-Language: en-
< X-dmg-node-name: defg_node_1
< X-Varnish-Bereq-Backend: real_backend_foo_bar_uk
< X-Varnish-Bereq-Retries: 0
< Last-Modified: Wed, 14 Dec 2016 11:06:11 GMT
< Cache-Control: public, max-age=300
< X-Varnish: 1343699
< Age: 0
< Via: 1.1 varnish-v4
< X-Varnish-Cache: MISS
< X-Varnish-Trimen: www.trimen.com
< X-Varnish-Served-By-Host: snarf.foo.uk
< X-Varnish-Served-By-IP: 100.100.10.80
< X-Varnish-Pool: http_pages
< X-Varnish-Req-Backend-Hint: dead
< X-Varnish-Req-Restarts: 0
< X-Varnish-Hash: /data/action/getGame/9788578457657
< X-Varnish-Backend-Ourself: varnish_server_snarf_foo_uk
< X-DMG-Version: 6.20.51.2358
< Accept-Ranges: none
< Content-Length: 2988
< Connection: keep-alive
<
{
"errorMessage" : null,
"expiry" : "2016-12-14T11:06:11.927+0000",
"data" : {
// json data object here
}
* Connection #0 to host http://example.com/ left intact
and..
$ curl -H "Origin: http://dev.my-dev.local/" --verbose \
> http://example.com/data/action/getGame/9788578457657
* Trying 1?.???.??.???...
* Connected to http://example.com/ (1?.???.??.???) port 80 (#0)
> GET /data/action/getGame/9788578457657 HTTP/1.1
> Host: http://example.com/
> User-Agent: curl/7.43.0
> Accept: */*
> Origin: http://dev.my-dev.local/
>
< HTTP/1.1 200 OK
< Date: Wed, 14 Dec 2016 11:07:10 GMT
< Server: WildFly/8
< Expires: Wed, 14 Dec 2016 11:07:10 GMT
< X-Powered-By: Undertow/1
< X-dmg-elapsed-time: 28ms
< X-dmg-host-address: 1??.??.???.??
< Vary: Accept-Encoding,Origin
< X-dmg-generated-time: Wed, 14 Dec 2016 11:07:10 GMT
< Content-Type: application/json;charset=UTF-8
< Content-Language: en-
< X-dmg-node-name: defg_node_1
< X-Varnish-Bereq-Backend: real_backend_foo_bar_uk
< X-Varnish-Bereq-Retries: 0
< Last-Modified: Wed, 14 Dec 2016 11:07:10 GMT
< Cache-Control: public, max-age=300
< X-Varnish: 6619151
< Age: 0
< Via: 1.1 varnish-v4
< X-Varnish-Cache: MISS
< X-Varnish-Trimen: www.trimen.com
< X-Varnish-Served-By-Host: snarf.foo.uk
< X-Varnish-Served-By-IP: 100.100.10.80
< X-Varnish-Pool: http_pages
< X-Varnish-Req-Backend-Hint: dead
< X-Varnish-Req-Restarts: 0
< X-Varnish-Hash: /data/action/getGame/9788578457657
< X-Varnish-Backend-Ourself: varnish_server_snarf_foo_uk
< X-DMG-Version: 6.20.51.2358
< Accept-Ranges: none
< Content-Length: 2988
< Connection: keep-alive
<
{
"errorMessage" : null,
"expiry" : "2016-12-14T11:07:10.764+0000",
"data" : {
// JSON object data here
}
* Connection #0 to host http://example.com/ left intact
}
2nd UPDATE
I disables same origin policy in Chrome and these are the headers to my JSON request from Chrome's network panel.
GET data/action/getGame/9788578457657 HTTP/1.1
Host: example.com
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Accept: application/json, text/plain, */*
Origin: http://dev.my-dev.local/
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36
Referer: http://dev.my-dev.local//game/id-9788578457657
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
HTTP/1.1 200 OK
Date: Wed, 14 Dec 2016 15:38:38 GMT
Server: WildFly/8
Expires: Wed, 14 Dec 2016 15:38:38 GMT
X-Powered-By: Undertow/1
X-dmg-elapsed-time: 25ms
X-dmg-host-address: 172.16.0.70
Vary: Accept-Encoding,Origin
X-dmg-generated-time: Wed, 14 Dec 2016 15:38:38 GMT
Content-Type: application/json;charset=UTF-8
Content-Language: en-
X-dmg-node-name: defg_node_1
Content-Encoding: gzip
Content-Length: 1109
X-Varnish-Bereq-Backend: real_backend_foo_bar_uk
X-Varnish-Bereq-Retries: 0
Last-Modified: Wed, 14 Dec 2016 15:38:38 GMT
Cache-Control: public, max-age=300
X-Varnish: 6619151
Age: 0
Via: 1.1 varnish-v4
X-Varnish-Cache: MISS
X-Varnish-Trimen: www.trimen.com
X-Varnish-Served-By-Host: snarf.foo.uk
X-Varnish-Served-By-IP: 100.100.10.80
X-Varnish-Pool: http_pages
X-Varnish-Req-Backend-Hint: dead
X-Varnish-Req-Restarts: 0
X-Varnish-Hash: /data/action/getGame/9788578457657
X-Varnish-Backend-Ourself: arnish_server_snarf_foo_uk
X-DMG-Version: 6.20.51.2358
Accept-Ranges: none
Connection: keep-alive
3rd UPDATE
So after changing the http method to OPTIONS as in
$http({
method: 'OPTIONS',
...
I ge this error in the chrome consoler
XMLHttpRequest cannot load http://example.com/data/action/getGame/9788578457657. Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://dev.my-dev.local'' is therefore not allowed access. The response had HTTP status code 405.
And these are the headers:
OPTIONS /data/action/getGame/9788578457657 HTTP/1.1
Host: example.com
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Access-Control-Request-Method: OPTIONS
Origin: http://dev.my-dev.local/
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36
Access-Control-Request-Headers:
Accept: */*
Referer: http://dev.my-dev.local//game/id-9788578457657
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
HTTP/1.1 405 Method Not Allowed
Date: Wed, 14 Dec 2016 16:52:03 GMT
Server: Varnish
X-Varnish: 6619151
X-Varnish-Trimen: www.trimen.com
X-Varnish-Served-By-Host: snarf.foo.uk
X-Varnish-Served-By-IP: 100.100.10.80
X-Varnish-Pool:
X-Varnish-Req-Backend-Hint: dead
X-Varnish-Req-Restarts: 0
X-DMG-Version: 6.20.51.2358
Content-Type: text/html; charset=utf-8
Retry-After: 5
Content-Length: 49669
Connection: keep-alive
You need to receive the following headers:
Access-Control-Allow-Origin: * (or whatever host you want to restrict to)
Access-Control-Allow-Methods: * (or whatever methods you want to restrict to)
Access-Control-Allow-Headers: Content-Type
Note the last one which is also important because you are setting Content-Type: application/json;charset=UTF-8. If you have any other custom headers you will need to add those too.
These are all to be done on the server though, your app doesn't need to do anything else.
Alternatively (if possible) you can opt to not use application/json at all and set your Content-Type to application/x-www-form-urlencoded, multipart/form-data, or text/plain and no preflight (OPTIONS) request will be done and it won't matter if CORS is enabled on the server or not.
The answer by ed inspired me on my solution
you need to send the following headers during a preflight request (OPTIONS method for the endpoint)
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET,HEAD,PUT,PATCH,POST,DELETE
Access-Control-Allow-Headers: Content-Type, Access-Control-Allow-Origin, xxx
where xxx is whatever additional headers you are sending when doing the POST/PUT/DELETE/etc request
note that when this solution works, I strongly suggest changing the * to restrict it to your known origins

List windows route table

I want to list entries from windows route table. Same output as from route print. I use GetIpForwardTable2 function from IP Helper API. But I get some weird results which differ from route command output.
I run it in Windows 7 64bit in VirtualBox where I have 3 network cards (NAT, Bridge and Internal Network) and compile it under cygwin with following command:
gcc -D_WIN32_WINNT=0x0601 -DNTDDI_VERSION=0x06010000 win-iproute.c -liphlpapi
Those _WIN32_WINNT and NTDDI_VERSION are just to make functionality from Win7 available.
To make it simplier I consider ipv4 only now.
Here is the code:
#include <windows.h>
#include <winsock2.h>
#include <iphlpapi.h>
#include <Mstcpip.h>
#include <stdio.h>
int main(int argc, char *argv[])
{
DWORD retval;
MIB_IPFORWARD_TABLE2 *routes = NULL;
MIB_IPFORWARD_ROW2 *route;
int idx;
retval = GetIpForwardTable2(AF_INET, &routes);
if (retval != ERROR_SUCCESS)
{
fprintf(stderr, "GetIpForwardTable2 failed (0x%x)\n.", retval);
return 1;
}
printf("Route entries count: %lu\n", routes->NumEntries);
for (idx = 0; idx < routes->NumEntries; idx++)
{
printf("\n -- Entry #%d -- \n", idx);
route = routes->Table + idx;
printf("luid: \t\t Reserved: %u, NetLuidIndex %u, IfType %u\n",
route->InterfaceLuid.Info.Reserved,
route->InterfaceLuid.Info.NetLuidIndex,
route->InterfaceLuid.Info.IfType);
printf("protocol: \t %lu\n", route->Protocol);
printf("origin: \t %lu\n", route->Origin);
printf("loopback: \t %lu\n", route->Loopback);
printf("next hop: \t %s\n", inet_ntoa(route->NextHop.Ipv4.sin_addr));
printf("site prefix length: \t %u\n", route->SitePrefixLength);
printf("prefix length: \t %u\n", route->DestinationPrefix.PrefixLength);
printf("prefix : \t %s\n", inet_ntoa(route->DestinationPrefix.Prefix.Ipv4.sin_addr));
}
return 0;
}
And the output is:
Route entries count: 22
-- Entry #0 --
luid: Reserved: 0, NetLuidIndex 6, IfType 6
protocol: 0
origin: 0
loopback: 0
next hop: 0.0.0.0
site prefix length: 0
prefix length: 0
prefix : 0.0.0.0
-- Entry #1 --
luid: Reserved: 0, NetLuidIndex 0, IfType 0
protocol: 0
origin: 0
loopback: 0
next hop: 0.0.0.0
site prefix length: 0
prefix length: 3
prefix : 0.0.0.0
-- Entry #2 --
luid: Reserved: 0, NetLuidIndex 0, IfType 0
protocol: 4294967295
origin: 257
loopback: 0
next hop: 0.0.0.0
site prefix length: 0
prefix length: 10
prefix : 0.1.0.0
-- Entry #3 --
luid: Reserved: 17, NetLuidIndex 0, IfType 0
protocol: 11
origin: 0
loopback: 2
next hop: 0.0.0.0
site prefix length: 17
prefix length: 0
prefix : 2.0.0.0
-- Entry #4 --
luid: Reserved: 0, NetLuidIndex 0, IfType 0
protocol: 32
origin: 0
loopback: 2
next hop: 0.1.0.0
site prefix length: 0
prefix length: 255
prefix : 2.0.0.0
-- Entry #5 --
luid: Reserved: 0, NetLuidIndex 0, IfType 0
protocol: 0
origin: 256
loopback: 255
next hop: 0.0.0.0
site prefix length: 0
prefix length: 11
prefix : 255.255.255.255
-- Entry #6 --
luid: Reserved: 3, NetLuidIndex 65792, IfType 0
protocol: 201326592
origin: 2
loopback: 0
next hop: 0.0.0.0
site prefix length: 3
prefix length: 24
prefix : 0.0.6.0
-- Entry #7 --
luid: Reserved: 5855577, NetLuidIndex 89, IfType 0
protocol: 0
origin: 2
loopback: 0
next hop: 0.1.0.0
site prefix length: 89
prefix length: 0
prefix : 0.0.0.0
-- Entry #8 --
luid: Reserved: 0, NetLuidIndex 0, IfType 0
protocol: 0
origin: 4294967295
loopback: 0
next hop: 2.0.0.0
site prefix length: 0
prefix length: 0
prefix : 0.0.0.0
-- Entry #9 --
luid: Reserved: 16777215, NetLuidIndex 65791, IfType 0
protocol: 593
origin: 1572864
loopback: 0
next hop: 2.0.0.0
site prefix length: 255
prefix length: 0
prefix : 0.0.0.0
-- Entry #10 --
luid: Reserved: 1, NetLuidIndex 512, IfType 0
protocol: 0
origin: 0
loopback: 0
next hop: 255.255.255.255
site prefix length: 1
prefix length: 0
prefix : 0.0.0.0
-- Entry #11 --
luid: Reserved: 4, NetLuidIndex 512, IfType 0
protocol: 0
origin: 0
loopback: 0
next hop: 0.0.6.0
site prefix length: 4
prefix length: 81
prefix : 0.0.0.0
-- Entry #12 --
luid: Reserved: 0, NetLuidIndex 16776960, IfType 65535
protocol: 3
origin: 1
loopback: 0
next hop: 0.0.0.0
site prefix length: 0
prefix length: 0
prefix : 0.1.0.0
-- Entry #13 --
luid: Reserved: 0, NetLuidIndex 12, IfType 6
protocol: 4294967295
origin: 0
loopback: 0
next hop: 0.0.0.0
site prefix length: 0
prefix length: 0
prefix : 0.0.0.0
-- Entry #14 --
luid: Reserved: 0, NetLuidIndex 0, IfType 0
protocol: 0
origin: 0
loopback: 0
next hop: 0.0.0.0
site prefix length: 0
prefix length: 3
prefix : 0.0.0.0
-- Entry #15 --
luid: Reserved: 0, NetLuidIndex 0, IfType 0
protocol: 4294967295
origin: 257
loopback: 0
next hop: 0.0.0.0
site prefix length: 0
prefix length: 255
prefix : 0.1.0.0
-- Entry #16 --
luid: Reserved: 585, NetLuidIndex 0, IfType 0
protocol: 3449440
origin: 0
loopback: 0
next hop: 0.0.0.0
site prefix length: 73
prefix length: 0
prefix : 2.0.0.0
-- Entry #17 --
luid: Reserved: 3211321, NetLuidIndex 13056, IfType 65
protocol: 3342403
origin: 4325427
loopback: 49
next hop: 125.0.0.0
site prefix length: 53
prefix length: 68
prefix : 54.0.45.0
-- Entry #18 --
luid: Reserved: 3473453, NetLuidIndex 17408, IfType 54
protocol: 0
origin: 0
loopback: 0
next hop: 0.0.0.0
site prefix length: 0
prefix length: 0
prefix : 70.0.69.0
-- Entry #19 --
luid: Reserved: 0, NetLuidIndex 0, IfType 0
protocol: 7471205
origin: 7274610
loopback: 0
next hop: 115.0.97.0
site prefix length: 111
prefix length: 0
prefix : 0.0.0.0
-- Entry #20 --
luid: Reserved: 7274611, NetLuidIndex 26112, IfType 116
protocol: 3277144
origin: 50725
loopback: 0
next hop: 49.69.55.56
site prefix length: 51
prefix length: 56
prefix : 65.0.100.0
-- Entry #21 --
luid: Reserved: 3277144, NetLuidIndex 0, IfType 0
protocol: 0
origin: 0
loopback: 0
next hop: 0.0.0.0
site prefix length: 192
prefix length: 0
prefix : 16.0.0.0
While the output of route print -4 is following:
===========================================================================
Interface List
16...08 00 27 7e 98 16 ......Intel(R) PRO/1000 MT Desktop Adapter #3
14...08 00 27 86 3d 31 ......Intel(R) PRO/1000 MT Desktop Adapter #2
11...08 00 27 42 d2 16 ......Intel(R) PRO/1000 MT Desktop Adapter
1...........................Software Loopback Interface 1
12...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter
13...00 00 00 00 00 00 00 e0 Teredo Tunneling Pseudo-Interface
15...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #2
17...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #3
===========================================================================
IPv4 Route Table
===========================================================================
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 10.0.2.2 10.0.2.15 10
0.0.0.0 0.0.0.0 10.0.0.138 10.0.0.36 10
10.0.0.0 255.255.255.0 On-link 10.0.0.36 266
10.0.0.36 255.255.255.255 On-link 10.0.0.36 266
10.0.0.255 255.255.255.255 On-link 10.0.0.36 266
10.0.2.0 255.255.255.0 On-link 10.0.2.15 266
10.0.2.15 255.255.255.255 On-link 10.0.2.15 266
10.0.2.255 255.255.255.255 On-link 10.0.2.15 266
89.89.89.0 255.255.255.0 On-link 89.89.89.89 266
89.89.89.89 255.255.255.255 On-link 89.89.89.89 266
89.89.89.255 255.255.255.255 On-link 89.89.89.89 266
127.0.0.0 255.0.0.0 On-link 127.0.0.1 306
127.0.0.1 255.255.255.255 On-link 127.0.0.1 306
127.255.255.255 255.255.255.255 On-link 127.0.0.1 306
224.0.0.0 240.0.0.0 On-link 127.0.0.1 306
224.0.0.0 240.0.0.0 On-link 10.0.2.15 266
224.0.0.0 240.0.0.0 On-link 10.0.0.36 266
224.0.0.0 240.0.0.0 On-link 89.89.89.89 266
255.255.255.255 255.255.255.255 On-link 127.0.0.1 306
255.255.255.255 255.255.255.255 On-link 10.0.2.15 266
255.255.255.255 255.255.255.255 On-link 10.0.0.36 266
255.255.255.255 255.255.255.255 On-link 89.89.89.89 266
===========================================================================
Persistent Routes:
None
There is a lot of weird stuff in the code output. Many entries have undocumented values, for example:
Protocol should be within range 1-14 (almost non entry has such value)
Luid.IfType shouldn't be 0 (again almost all are zero)
almost non entry gives any reasonable Prefix
It's described here MIB_IPFORWARD_ROW2 and here NET_LUID
Should I just ignore those with invalid values? and if so where are the valid ones? Or am I doing something terribly wrong?
I also discovered that when I start Windows with cables unplugged it gives less entries (which makes sense). Then I plug in the cables and entries are added. But when I unplug again they are still there. route command works as expected, when cable is unplugged entries are reduced.
When I try older function GetIpForwardTable it works. But it doesn't support ipv6.
So it seems that the problem was in cygwin. When I compile the example code with Microsoft C compiler cl.ex it works as expected. And after update of cygwin it works when compiled using gcc too.
Interesting is that it was enough to update the packages using cygwin installer, cygwin1.dll can remain in older version.

Why actual bytes read from server's response less than the Content-Length?

I'm writing a proxy in C in Linux 3.0 kernel. The proxy forwards the request from browser to the target server and read response from server.
This is one request:
GET http://www.yongchuan.org/ HTTP/1.1
Host: www.yongchuan.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Proxy-Connection: keep-alive
This is one response the proxy read:
HTTP/1.1 200 OK
Content-Length: 3170
Content-Type: text/html
Content-Location: http://www.yongchuan.org/index.html
Last-Modified: Fri, 22 Jul 2011 01:28:50 GMT
Accept-Ranges: bytes
ETag: "72d9d7b5e48cc1:1a73"
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Date: Sun, 04 Mar 2012 16:26:05 GMT
<html>
<head>
..............
<tr>
<td width="43%" ></td>
<td width="46%"></td>
</tr>
<tr>
<td width="43%"><img src="images/triangle.gif"> ��ѧ�ڿγ̣�Ӧ��ͼ��</td>
<td width="46%" ><img src=
The problem is, the actual size of bytes the proxy read is 2880 and the data ends up suddenly, much less than Context length: 3170.
My code is as below:
241 int readlen;
242 char buffer[128 * 4096];
243 do {
244 readlen = read(servfd, buffer, 128 * 4096);
245 printf("readlen:%d\n", readlen);
246 if( readlen < 0 ) {
247 perror("read() from server failed");
248 }
249 printf("read content:\n%s", buffer2);
250 write(cliefd, buffer, readlen);
251 } while( readlen == 128 * 4096 );
Here servfd is the socket to server, cliefd is the socket to client.
What's the possible reason? Thanks!
while( readlen == 128 * 4096 );
The while condition is wrong. You should be checking readlen > 0 instead.
There's no requirement that read only hands you completely filled buffers. In the worst case (highly unlikely with current TCP stacks), it could hand you the bytes one by one.
At any rate, if the server doesn't close the connection after delivering the output you must keep reading until you get the Content-Length you were promised.

Resources