I have used Oracle Advanced Security to encrypt data during data transfer. I have successfully configured ssl with below parameters and I have restarted the instance. I am retrieving data from a Java class given below. But I could read the data without decrypting, the data is not getting encrypted.
Environment:
Oragle 11g database
SQLNET.AUTHENTICATION_SERVICES= (BEQ, TCPS, NTS)
SSL_VERSION = 0
NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)
SSL_CLIENT_AUTHENTICATION = FALSE
WALLET_LOCATION =
(SOURCE =
(METHOD = FILE)
(METHOD_DATA =
(DIRECTORY = C:\Users\kcr\Oracle\WALLETS)
)
)
SSL_CIPHER_SUITES= (SSL_RSA_EXPORT_WITH_RC4_40_MD5)
Java class:
try{
Properties properties = Utils.readProperties("weka/experiment/DatabaseUtils.props");
// Security.addProvider(new oracle.security.pki.OraclePKIProvider()); //Security syntax
String url = "jdbc:oracle:thin:#(DESCRIPTION =\n" +
" (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))\n" +
" (CONNECT_DATA =\n" +
" (SERVER = DEDICATED)\n" +
" (SERVICE_NAME = sal)\n" +
" )\n" +
" )";
java.util.Properties props = new java.util.Properties();
props.setProperty("user", "system");
props.setProperty("password", "weblogic");
// props.setProperty("javax.net.ssl.trustStore","C:\\Users\\kcr\\Oracle\\WALLETS\\ewallet.p12");
// props.setProperty("oracle.net.ssl_cipher_suites","SSL_RSA_EXPORT_WITH_RC4_40_MD5");
// props.setProperty("javax.net.ssl.trustStoreType","PKCS12");
//props.setProperty("javax.net.ssl.trustStorePassword","welcome2");
DriverManager.registerDriver(new OracleDriver());
Connection conn = DriverManager.getConnection(url, props);
/*8 OracleDataSource ods = new OracleDataSource();
ods.setUser("system");
ods.setPassword("weblogic");
ods.setURL(url);
Connection conn = ods.getConnection();*/
Statement stmt = conn.createStatement();
ResultSet rset = stmt.executeQuery("select * from iris");
///////////////////////////
while(rset.next()) {
for (int i=1; i<=5; i++) {
System.out.print(rset.getString(i));
}
}
Are you expecting that your SELECT statement would return encrypted data and that your System.out.print calls would result in encrypted output going to the screen? If so, that's not the way advanced security works-- Advanced Security allows you to encrypt data over the wire but the data is unencrypted in the SQLNet stack. Your SELECT statement, therefore, would always see the data in an unencrypted state. You would need to do a SQLNet trace or use some sort of packet sniffer to see the encrypted data flowing over the wire.
You'll find the documentation in "SSL With Oracle JDBC Thin Driver".
In particular you should probably use PROTOCOL = TCPS instead of PROTOCOL = TCP. I'd also suggest using a stronger cipher suite (and avoid the anonymous ones, since with them you don't verify the identity of the remote server).
Related
I'm trying to find out what protocol the SnowFlake JDBC library uses to communicate with SnowFlake. I see hints here and there that it seems to be using HTTPS as the protocol. Is this true?
To my knowledge, other JDBC libraries like for example for Oracle or PostgreSQL use the lower level TCP protocol to communicate with their database servers, and not the application-level HTTP(S) protocol, so I'm confused.
My organization only supports securely routing http(s)-based communication. Can I use this snowflake jdbc library then?
I have browsed all documentation that I could find, but wasn't able to answer this question.
My issue on GitHub didn't get an answer either.
Edit: Yes, I've seen this question, but I don't feel that it answers my question. SSL/TLS is an encryption, but that doesn't specify the data format.
It looks like the jdbc driver uses HTTP Client HttpUtil.initHttpClient(httpClientSettingsKey, null);, as you can see in here
The HTTP Utility Class is available here
Putting an excerpt of the session open method here in case the link goes bad/dead.
/**
* Open a new database session
*
* #throws SFException this is a runtime exception
* #throws SnowflakeSQLException exception raised from Snowflake components
*/
public synchronized void open() throws SFException, SnowflakeSQLException {
performSanityCheckOnProperties();
Map<SFSessionProperty, Object> connectionPropertiesMap = getConnectionPropertiesMap();
logger.debug(
"input: server={}, account={}, user={}, password={}, role={}, database={}, schema={},"
+ " warehouse={}, validate_default_parameters={}, authenticator={}, ocsp_mode={},"
+ " passcode_in_password={}, passcode={}, private_key={}, disable_socks_proxy={},"
+ " application={}, app_id={}, app_version={}, login_timeout={}, network_timeout={},"
+ " query_timeout={}, tracing={}, private_key_file={}, private_key_file_pwd={}."
+ " session_parameters: client_store_temporary_credential={}",
connectionPropertiesMap.get(SFSessionProperty.SERVER_URL),
connectionPropertiesMap.get(SFSessionProperty.ACCOUNT),
connectionPropertiesMap.get(SFSessionProperty.USER),
!Strings.isNullOrEmpty((String) connectionPropertiesMap.get(SFSessionProperty.PASSWORD))
? "***"
: "(empty)",
connectionPropertiesMap.get(SFSessionProperty.ROLE),
connectionPropertiesMap.get(SFSessionProperty.DATABASE),
connectionPropertiesMap.get(SFSessionProperty.SCHEMA),
connectionPropertiesMap.get(SFSessionProperty.WAREHOUSE),
connectionPropertiesMap.get(SFSessionProperty.VALIDATE_DEFAULT_PARAMETERS),
connectionPropertiesMap.get(SFSessionProperty.AUTHENTICATOR),
getOCSPMode().name(),
connectionPropertiesMap.get(SFSessionProperty.PASSCODE_IN_PASSWORD),
!Strings.isNullOrEmpty((String) connectionPropertiesMap.get(SFSessionProperty.PASSCODE))
? "***"
: "(empty)",
connectionPropertiesMap.get(SFSessionProperty.PRIVATE_KEY) != null
? "(not null)"
: "(null)",
connectionPropertiesMap.get(SFSessionProperty.DISABLE_SOCKS_PROXY),
connectionPropertiesMap.get(SFSessionProperty.APPLICATION),
connectionPropertiesMap.get(SFSessionProperty.APP_ID),
connectionPropertiesMap.get(SFSessionProperty.APP_VERSION),
connectionPropertiesMap.get(SFSessionProperty.LOGIN_TIMEOUT),
connectionPropertiesMap.get(SFSessionProperty.NETWORK_TIMEOUT),
connectionPropertiesMap.get(SFSessionProperty.QUERY_TIMEOUT),
connectionPropertiesMap.get(SFSessionProperty.TRACING),
connectionPropertiesMap.get(SFSessionProperty.PRIVATE_KEY_FILE),
!Strings.isNullOrEmpty(
(String) connectionPropertiesMap.get(SFSessionProperty.PRIVATE_KEY_FILE_PWD))
? "***"
: "(empty)",
sessionParametersMap.get(CLIENT_STORE_TEMPORARY_CREDENTIAL));
HttpClientSettingsKey httpClientSettingsKey = getHttpClientKey();
logger.debug(
"connection proxy parameters: use_proxy={}, proxy_host={}, proxy_port={}, proxy_user={},"
+ " proxy_password={}, non_proxy_hosts={}, proxy_protocol={}",
httpClientSettingsKey.usesProxy(),
httpClientSettingsKey.getProxyHost(),
httpClientSettingsKey.getProxyPort(),
httpClientSettingsKey.getProxyUser(),
!Strings.isNullOrEmpty(httpClientSettingsKey.getProxyPassword()) ? "***" : "(empty)",
httpClientSettingsKey.getNonProxyHosts(),
httpClientSettingsKey.getProxyProtocol());
// TODO: temporarily hardcode sessionParameter debug info. will be changed in the future
SFLoginInput loginInput = new SFLoginInput();
loginInput
.setServerUrl((String) connectionPropertiesMap.get(SFSessionProperty.SERVER_URL))
.setDatabaseName((String) connectionPropertiesMap.get(SFSessionProperty.DATABASE))
.setSchemaName((String) connectionPropertiesMap.get(SFSessionProperty.SCHEMA))
.setWarehouse((String) connectionPropertiesMap.get(SFSessionProperty.WAREHOUSE))
.setRole((String) connectionPropertiesMap.get(SFSessionProperty.ROLE))
.setValidateDefaultParameters(
connectionPropertiesMap.get(SFSessionProperty.VALIDATE_DEFAULT_PARAMETERS))
.setAuthenticator((String) connectionPropertiesMap.get(SFSessionProperty.AUTHENTICATOR))
.setOKTAUserName((String) connectionPropertiesMap.get(SFSessionProperty.OKTA_USERNAME))
.setAccountName((String) connectionPropertiesMap.get(SFSessionProperty.ACCOUNT))
.setLoginTimeout(loginTimeout)
.setAuthTimeout(authTimeout)
.setUserName((String) connectionPropertiesMap.get(SFSessionProperty.USER))
.setPassword((String) connectionPropertiesMap.get(SFSessionProperty.PASSWORD))
.setToken((String) connectionPropertiesMap.get(SFSessionProperty.TOKEN))
.setPasscodeInPassword(passcodeInPassword)
.setPasscode((String) connectionPropertiesMap.get(SFSessionProperty.PASSCODE))
.setConnectionTimeout(httpClientConnectionTimeout)
.setSocketTimeout(httpClientSocketTimeout)
.setAppId((String) connectionPropertiesMap.get(SFSessionProperty.APP_ID))
.setAppVersion((String) connectionPropertiesMap.get(SFSessionProperty.APP_VERSION))
.setSessionParameters(sessionParametersMap)
.setPrivateKey((PrivateKey) connectionPropertiesMap.get(SFSessionProperty.PRIVATE_KEY))
.setPrivateKeyFile((String) connectionPropertiesMap.get(SFSessionProperty.PRIVATE_KEY_FILE))
.setPrivateKeyFilePwd(
(String) connectionPropertiesMap.get(SFSessionProperty.PRIVATE_KEY_FILE_PWD))
.setApplication((String) connectionPropertiesMap.get(SFSessionProperty.APPLICATION))
.setServiceName(getServiceName())
.setOCSPMode(getOCSPMode())
.setHttpClientSettingsKey(httpClientSettingsKey);
// propagate OCSP mode to SFTrustManager. Note OCSP setting is global on JVM.
HttpUtil.initHttpClient(httpClientSettingsKey, null);
SFLoginOutput loginOutput =
SessionUtil.openSession(loginInput, connectionPropertiesMap, tracingLevel.toString());
isClosed = false;
authTimeout = loginInput.getAuthTimeout();
sessionToken = loginOutput.getSessionToken();
masterToken = loginOutput.getMasterToken();
idToken = loginOutput.getIdToken();
mfaToken = loginOutput.getMfaToken();
setDatabaseVersion(loginOutput.getDatabaseVersion());
setDatabaseMajorVersion(loginOutput.getDatabaseMajorVersion());
setDatabaseMinorVersion(loginOutput.getDatabaseMinorVersion());
httpClientSocketTimeout = loginOutput.getHttpClientSocketTimeout();
masterTokenValidityInSeconds = loginOutput.getMasterTokenValidityInSeconds();
setDatabase(loginOutput.getSessionDatabase());
setSchema(loginOutput.getSessionSchema());
setRole(loginOutput.getSessionRole());
setWarehouse(loginOutput.getSessionWarehouse());
setSessionId(loginOutput.getSessionId());
setAutoCommit(loginOutput.getAutoCommit());
// Update common parameter values for this session
SessionUtil.updateSfDriverParamValues(loginOutput.getCommonParams(), this);
String loginDatabaseName = (String) connectionPropertiesMap.get(SFSessionProperty.DATABASE);
String loginSchemaName = (String) connectionPropertiesMap.get(SFSessionProperty.SCHEMA);
String loginRole = (String) connectionPropertiesMap.get(SFSessionProperty.ROLE);
String loginWarehouse = (String) connectionPropertiesMap.get(SFSessionProperty.WAREHOUSE);
if (loginDatabaseName != null && !loginDatabaseName.equalsIgnoreCase(getDatabase())) {
sqlWarnings.add(
new SFException(
ErrorCode.CONNECTION_ESTABLISHED_WITH_DIFFERENT_PROP,
"Database",
loginDatabaseName,
getDatabase()));
}
if (loginSchemaName != null && !loginSchemaName.equalsIgnoreCase(getSchema())) {
sqlWarnings.add(
new SFException(
ErrorCode.CONNECTION_ESTABLISHED_WITH_DIFFERENT_PROP,
"Schema",
loginSchemaName,
getSchema()));
}
if (loginRole != null && !loginRole.equalsIgnoreCase(getRole())) {
sqlWarnings.add(
new SFException(
ErrorCode.CONNECTION_ESTABLISHED_WITH_DIFFERENT_PROP, "Role", loginRole, getRole()));
}
if (loginWarehouse != null && !loginWarehouse.equalsIgnoreCase(getWarehouse())) {
sqlWarnings.add(
new SFException(
ErrorCode.CONNECTION_ESTABLISHED_WITH_DIFFERENT_PROP,
"Warehouse",
loginWarehouse,
getWarehouse()));
}
// start heartbeat for this session so that the master token will not expire
startHeartbeatForThisSession();
}
I'm trying to send an array over TCP from a server-like script to a client-like one. The array is variable, so the data is sent using packets and then joined together at the client.
The data I'm trying to send is from the MNIST hand-written digits dataset for Deep Learning. The server-side code is:
tcp = '127.0.0.1'
port = 1234
buffer_size = 4096
(X_train, y_train), (X_test, y_test) = mnist.load_data()
test_data = (X_test, y_test)
# Client-side Deep Learning stuff
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((tcp, port))
x = pickle.dumps(test_data)
s.sendall(x)
s.close()
The client-side script loads a Neural Network that uses the test data to predict classes. The script for listening to said data is:
tcp = '127.0.0.1'
port = 1234
buffer_size = 4096
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((tcp, port))
print ('Listening...')
s.listen(1)
conn, addr = s.accept()
data_arr = []
while True:
data_pack = conn.recv(buffer_size)
if not data: break
data_pack += data
my_pickle = b"".join(data_pack)
test_data = pickle.loads(my_pickle)
print ("Received: " + test_data)
conn.close()
# Irrelevant Deep Learning stuff...
The server sends the data without a hitch, but the client crashes when trying to join the packets received by the client (my_pickle = ...) with the following error:
TypeError: sequence item 0: expected a bytes-like object, int found
How should I format the join in order to recreate the data sent and use it for the rest of the script?
I ended up using both Pickle and ZeroMQ to handle the comunication protocol. An advantage of this method is that I can send more than one data package.
On the client side:
ip = '127.0.0.1'
port = '1234'
# ZeroMQ context
context = zmq.Context()
# Setting up protocol (client)
sock = context.socket(zmq.REQ)
sock.bind('tcp://'+ip+':'+port)
print('Waiting for connection at tcp://'+ip+':'+port+'...')
sock.send(pickle.dumps(X_send))
X_answer = sock.recv()
sock.send(pickle.dumps(y_send))
print('Data sent. Waiting for classification...')
y_answer = sock.recv()
print('Done.')
And on the server side:
# ZeroMQ Context
context = zmq.Context()
# Setting up protocol (server)
sock = context.socket(zmq.REP)
ip = '127.0.0.1'
port = '1234'
sock.connect('tcp://'+ip+':'+port)
print('Listening to tcp://'+ip+':'+port+'...')
X_message = sock.recv()
X_test = pickle.loads(X_message)
sock.send(pickle.dumps(X_message))
y_message = sock.recv()
y_test = pickle.loads(y_message)
print('Data received. Starting classification...')
# Classification process
sock.send(pickle.dumps(y_message))
print('Done.')
I want to create a model in R using a connection to data stored in SQL Server datawarehouse.
I tried to use RevoScaleR library which returned
package RevoScaleR is not available (for R version 3.4.1)
so, I edited the connection string (given on the code below) for ODBC library:
install.packages("RevoScaleR")
#require("RevoScaleR")
if (!require("RODBC"))
install.packages("RODBC")
conn <- odbcDriverConnect(connection="Driver={SQL Server Native Client 11.0}; Server=CZPHADDWH01/DEV; Database=DWH_Staging; trusted_connection=true")
sqlWait <- TRUE;
sqlConsoleOutput <- FALSE;
cc <- RxInSqlServer(connectionString = conn, wait = sqlWait)
rxSetComputeContext(cc)
train_query <- "SELECT TOP(10000) * FROM dim.Contract"
formula <- as.formula("Cosi ~ ContractID + ApprovedLoanAmount + ApprovedLoadDuration")
forest_model <- rxDForest(formula = formula,
data = train_query,
nTree = 20,
maxDepth = 32,
mTry = 3,
seed = 5,
verbose = 1,
reportProgress = 1)
rxDForest_model <- as.raw(serialize(forest_model, connection = conn))
lenght(rxDForest_model)
However:
package 'RODBC' successfully unpacked and MD5 sums checked
The downloaded binary packages are in
C:\Users\sjirak\AppData\Local\Temp\Rtmpqa9iKN\downloaded_packages
Error in odbcDriverConnect(connection = "Driver={SQL Server Native
Client 11.0}; Server=CZPHADDWH01/DEV; Database=DWH_Staging;
trusted_connection=true") : could not find function
"odbcDriverConnect" In library(package, lib.loc = lib.loc,
character.only = TRUE, logical.return = TRUE, : there is no
package called 'RODBC'
Any help would be appreciated.
Looking at the documentation of the ODBC, I see the following functions
odbc-package
dbConnect,OdbcDriver-method
dbUnQuoteIdentifier
odbc
odbc-tables
OdbcConnection
odbcConnectionActions
odbcConnectionIcon
odbcDataType
OdbcDriver
odbcListColumns
odbcListDataSources
odbcListDrivers
odbcListObjects
odbcListObjectTypes
odbcPreviewObject
OdbcResult
odbcSetTransactionIsolationLevel
test_roundtrip
hence I dont see your function in this list. This could be the reason why...
Hence, check the documentation for the proper function.
I'm trying to connect to my local sql server database at my home network using the code that I found from this site Querying SQL Server with Google Apps Script via JDBC about 3 years ago which was marked being correct. However, I get the error message "We're sorry, a server error occurred. Please wait a bit and try again.". This error is from line 2 where the connection string is defined. I retried several times, but I always get the same error. When I searched this error, it seems like it could be too many things and I was not able to find any answers for my issue. Thanks.
This was the code that was marked being correct:
function readAzure() {
var conn = Jdbc.getConnection("jdbc:sqlserver://XYZ.database.windows.net:1433;databaseName=MYDATABSENAME","USERNAME","PASSWORD");
var stmt = conn.createStatement();
var rs = stmt.executeQuery("select * from helloworld");
var doc = SpreadsheetApp.create('azure');
var cell = doc.getRange('a1');
var row = 0;
while(rs.next()) {
cell.offset(row, 0).setValue(rs.getString(1));
cell.offset(row, 1).setValue(rs.getString(2));
row++;
}
rs.close();
stmt.close();
conn.close();
}
I also found another connection string code and when I try this format I get the same error.
var conn = Jdbc.getConnection("jdbc:sqlserver://IP-address:1433;" + "databaseName=DBName;user=username;password=password;");
Based on this documentation, you need to ensure that your database accepts connections from any of Apps Script's IP addresses. These are the address ranges you'll need to whitelist.
64.18.0.0 - 64.18.15.255
64.233.160.0 - 64.233.191.255
66.102.0.0 - 66.102.15.255
66.249.80.0 - 66.249.95.255
72.14.192.0 - 72.14.255.255
74.125.0.0 - 74.125.255.255
173.194.0.0 - 173.194.255.255
207.126.144.0 - 207.126.159.255
209.85.128.0 - 209.85.255.255
216.239.32.0 - 216.239.63.255
Also, from the documentation link above, there is a sample code that you can follow to set up external database via JDBC.
Here is a code that demonstrates how to write a single record to the database as well as a batch of 500 records.
// Replace the variables in this block with real values.
var address = 'database_IP_address';
var user = 'user_name';
var userPwd = 'user_password';
var db = 'database_name';
var dbUrl = 'jdbc:mysql://' + address + '/' + db;
// Write one row of data to a table.
function writeOneRecord() {
var conn = Jdbc.getConnection(dbUrl, user, userPwd);
var stmt = conn.prepareStatement('INSERT INTO entries '
+ '(guestName, content) values (?, ?)');
stmt.setString(1, 'First Guest');
stmt.setString(2, 'Hello, world');
stmt.execute();
}
// Write 500 rows of data to a table in a single batch.
function writeManyRecords() {
var conn = Jdbc.getConnection(dbUrl, user, userPwd);
conn.setAutoCommit(false);
var start = new Date();
var stmt = conn.prepareStatement('INSERT INTO entries '
+ '(guestName, content) values (?, ?)');
for (var i = 0; i < 500; i++) {
stmt.setString(1, 'Name ' + i);
stmt.setString(2, 'Hello, world ' + i);
stmt.addBatch();
}
var batch = stmt.executeBatch();
conn.commit();
conn.close();
var end = new Date();
Logger.log('Time elapsed: %sms for %s rows.', end - start, batch.length);
}
For more information, just read through to this documentation and check this thread and related SO question.
I have created a sample app by using both the oledb provider(SQLOLEDB and SQL Native OLEDB provider).
Case 1 : Provider = SQLOLEDB
hr = ::CoInitializeEx(NULL, COINIT_MULTITHREADED);
hr = cADOConnection.CreateInstance(__uuidof(Connection));
CString con_string = "provider=SQLOLEDB;server=MYPC;Database=MyDB";
CString SSlcon_string = "provider=SQLOLEDB;Encrypt=true;TrustServerCertificate=true;server=MYPC;Database=MyDB";
CString userName = "sa";
CString Password = "sa";
BSTR bsConnection = /*con_string*/SSlcon_string.AllocSysString();
BSTR uName = userName.AllocSysString();
BSTR uPassword = Password.AllocSysString();
hr = cADOConnection->Open(bsConnection, uName, uPassword, adConnectUnspecified);
printf("connection has been established");
VARIANT vaNoRecords;
memset(&vaNoRecords, 0, sizeof vaNoRecords);
CString sql = "SELECT * FROM salary";
BSTR query = sql.AllocSysString();
_RecordsetPtr rs;
rs = cADOConnection->Execute(query, &vaNoRecords, adCmdText);
printf("connection has been established\n");
Result : If certificate is installed on server machine then the connection is secure regardless of enabling
Encrypt=true and TrustServerCertificate=true from in connection string.
Case 2 : Provider = SQLNCLI10.1(SQL native client oledb provider)
HRESULT hr;
hr = ::CoInitializeEx(NULL, COINIT_MULTITHREADED);
hr = cADOConnection.CreateInstance(__uuidof(Connection));
CString con_string = "provider=SQLNCLI10.1;server=MYPC;Database=MyDB";
CString SSlcon_string = "provider=SQLOLEDB;Encrypt=true;TrustServerCertificate=true;server=MYPC;Database=MyDB";
CString userName = "sa";
CString Password = "sa";
BSTR bsConnection = con_string/*SSlcon_string*/.AllocSysString();
BSTR uName = userName.AllocSysString();
BSTR uPassword = Password.AllocSysString();
hr = cADOConnection->Open(bsConnection, uName, uPassword, adConnectUnspecified);
printf("connection has been established");
VARIANT vaNoRecords;
memset(&vaNoRecords, 0, sizeof vaNoRecords);
CString sql = "SELECT suppliernumber, name1 FROM zrs_supplier";
BSTR query = sql.AllocSysString();
_RecordsetPtr rs;
rs = cADOConnection->Execute(query, &vaNoRecords, adCmdText);
printf("connection has been established\n");
Result : If certificate is installed on server machine then the connection is secure regardless of enabling
Encrypt=true and TrustServerCertificate=true from in connection string.i.e Result is same as above.
In both the case i am getting same behavior.Am i missing something here??
Any Suggestion would be appreciated ??
Original question
Replace the connection string with
CString SSlcon_string = "provider=SQLOLEDB;Use Encryption for Data=True;server=MYPC;Database=MyDB";
The remaining steps will be same.Install the same certificate(present on server) on client machine's "truted root certificate authorities" folder.
If server and client both will have same certificate then connection will be established(SSL Connection) otherwise fail.