I can access the repl and I have a uri that I can connect with. How do I connect, see what entities and attributes exist, and see their values then drill down and so on? I am just using the free version that I did not have to register for.
So far this is what I have tried:
C:\programs\datomic-free-0.9.5327>.\bin\repl
Clojure 1.6.0
user=> (use '[datomic.api :only [q db] :as d])
nil
user=> (def conn (d/connect "datomic:dev://localhost:4334/somedb"))
IllegalArgumentExceptionInfo :db.error/unsupported-protocol Unsupported protocol :dev datomic.error/arg (error.clj:57)
user=>
user=> (def conn (d/connect "datomic://localhost:4334/somedb"))
IllegalArgumentExceptionInfo :db.error/invalid-db-uri Invalid database URI datomic://localhost:4334/somedb datomic.error/arg (error.clj:57)
user=> (def conn (d/connect "datomic:mem//somedb"))
IllegalArgumentExceptionInfo :db.error/invalid-db-uri Invalid database URI datomic:mem//somedb datomic.error/arg (error.clj:57)
With a MySql or an Oracle database an operator is able (given the right password) to list all the databases, connect to one of them, see its tables, then start things rolling by querying one of them.
By way of context even though this database ('somedb') is 'in memory', I still do expect it to exist, because there is a transactor running. However the transactor is a bit suspicious because this is what it looks like:
C:\programs\datomic-free-0.9.5327>.\bin\transactor .\config\samples\free-transactor-template.properties
Starting datomic:free://localhost:4334/<DB-NAME>, storing data in: data ...
System started datomic:free://localhost:4334/<DB-NAME>, storing data in: data
I am confused about querying a database that is supposed to be up and running and have values in it.
I can clarify these errors:
user=> (use '[datomic.api :only [q db] :as d])
nil
user=> (def conn (d/connect "datomic:dev://localhost:4334/somedb"))
IllegalArgumentExceptionInfo :db.error/unsupported-protocol Unsupported protocol :dev datomic.error/arg (error.clj:57)
Here the URI is formatted correctly, but the error indicates you're trying to access a pro exclusive storage (dev) rather than the storages allowed by free (free and mem). Note that you can access pro storage formats with the free level starter license (i.e. you don't need a paid license to get going with it).
user=> (def conn (d/connect "datomic://localhost:4334/somedb"))
IllegalArgumentExceptionInfo :db.error/invalid-db-uri Invalid database URI datomic://localhost:4334/somedb datomic.error/arg (error.clj:57)
Here you're not specifying a storage protocol at all, i.e. where dev is in the first call. This is required.
user=> (def conn (d/connect "datomic:mem//somedb"))
IllegalArgumentExceptionInfo :db.error/invalid-db-uri Invalid database URI datomic:mem//somedb datomic.error/arg (error.clj:57)
Here you're missing the : after mem. For the above errors, the API documentation for connect covers correct URI construction.
You can list databases with get-database-names, documented here.
To get up and running with queries, you might have a better first experience using the console, a web app/GUI which handles this automatic listing of databases, connecting to one, querying, etc.
Related
I am trying to duplicate a database called database_A in the same cluster.
The result should be a new database called database_B with the exact same data.
I am trying to follow the instructions here.
It explains how to do this locally:
Use mongodump to dump the test database to an archive mongodump-test-db:
mongodump --archive="mongodump-test-db" --db=test
Use mongorestore with --nsFrom and --nsTo to restore (with database name change) from the archive:
mongorestore --archive="mongodump-test-db" --nsFrom='test.'
--nsTo='examples.'
In order to achieve the same result in the cluster, it says:
Include additional options as necessary, such as to specify the uri or
host, username, password and authentication database.
However, when I try to add the uri to achieve the first step:
mongodump --uri mongodb+srv://hihi:password#cluster0.fklgt.mongodb.net --archive="mongodump-test-db" --db=test_db
I get this error:
illegal argument combination: cannot specify --db and --uri
Any idea how to solve this?
You have to add db in URI- like connection uri/[db]
mongodb+srv://hihi:password#cluster0.fklgt.mongodb.net/test_db
mongodump --uri mongodb+srv://hihi:password#cluster0.fklgt.mongodb.net/test_db --archive="mongodump-test-db"
https://docs.mongodb.com/v3.6/reference/program/mongodump/#cmdoption-mongodump-uri
/database Optional. The name of the database to authenticate if the connection string includes authentication credentials in the form of username:password#. If /database is not specified and the connection string includes credentials, the driver will authenticate to the admin database. See also authSource.
https://docs.mongodb.com/v3.6/reference/connection-string/
I am trying to get a spark cluster to write to SQL server using JavaKerberos with Microsoft's JDBC driver (v7.0.0) (i.e., I specify integratedSecurity=true;authenticationScheme=JavaKerberos in the connection string) with credentials specified in a keyTab file and I am not having much success (the problem is the same if I specify credentials in the connections string).
I am submitting the job to the cluster (4-node YARN mode v 2.3.0) with:
spark-submit --driver-class-path mssql-jdbc-7.0.0.jre8.jar \
--jars /path/to/mssql-jdbc-7.0.0.jre8.jar \
--conf spark.executor.extraClassPath=/path/to/mssql-jdbc-7.0.0.jre8.jar \
--conf "spark.driver.extraJavaOptions=-Djava.security.auth.login.config=/path/to/SQLJDBCDriver.conf" \
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=/path/to/SQLJDBCDriver.conf" \
application.jar
Things work partially: the spark driver authenticates correctly and creates the table, however when any of the executors come to write to the table they fail with an exception:
java.security.PrivilegedActionException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
Observations:
I can get everything to work if I specify SQL server credentials (however I need to use integrated security in my application)
The keytab and login module file “SQLJDBCDriver.conf” seem to be specified correctly since they work for the driver
I can see in the spark UI the executors pick up the correct command line options :
-Djava.security.auth.login.config=/path/to/SQLJDBCDriver.conf
After a lot of logging/debugging the difference in spark driver and executor behaviour, it seems to come down to the executor trying to use the wrong credentials even though the options specified should make it use those specified in the keytab file as it does successfully for the spark driver. (That is why it generates that particular exception which is what it does if I try deliberately incorrect credentials.)
Strangely, I can see in the debug output the JDBC driver finds and reads the SQLJDBCDriver.conf file and the keytab has to present (otherwise I get file not found failure) yet it then promptly ignores them and tries to use default behaviour/local user credentials.
Can anyone help me understand how I can force the executors to use credentials provided in a keytab or otherwise get JavaKerberos/SQL Server authentication to work with Spark?
Just to give an update on this, I've just closed https://issues.apache.org/jira/browse/SPARK-12312 and now it's possible to do Kerberos authentication with built-in JDBC connection providers. There are many providers added and one of them is MS SQL. Please read the documentation how to use it: https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html
Please be aware Spark 3.1 is not yet released so it will take some time when the newly added 2 configuration parameters appear on the page (keytab and principal). I think the doc update will happen within 1-2 weeks.
Integrated authentication does not work with MS SQLServer JDBC driver in a secure cluster with AD integration as the containers will not have the context as the Kerberos tokens are lost when the mappers spawn (as the YARN transitions the job to its internal security subsystem).
Here is my repo that was used as work around to get Kerberos/AD authentication https://github.com/chandanbalu/mssql-jdbc-krb5 solution implements a Driver that overrides connect method of the latest MS SQL JDBC Driver (mssql-jdbc-9.2.1.jre8.jar), and will get a ticket for keytab file/principal, and gives this connection back.
You can grab the latest build of this custom driver from release folder here
Start spark-shell with JARS
spark-shell --jars /efs/home/c795701/.ivy2/jars/mssql-jdbc-9.2.1.jre8.jar,/efs/home/c795701/mssql-jdbc-krb5/target/scala-2.10/mssql-jdbc-krb5_2.10-1.0.jar
Scala
scala>val jdbcDF = spark.read.format("jdbc").option("url", "jdbc:krb5ss://<SERVER_NAME>:1433;databasename=<DATABASE_NAME>;integratedSecurity=true;authenticationScheme=JavaKerberos;krb5Principal=c795701#NA.DOMAIN.COM;krb5Keytab=/efs/home/c795701/c795701.keytab").option("driver","hadoop.sqlserver.jdbc.krb5.SQLServ, "dbo.table_name").load()
scala>jdbcDF.count()
scala>jdbcDF.show(10)
spark-submit command
com.spark.SparkJDBCIngestion - Spark JDBC data frame operations
ingestionframework-1.0-SNAPSHOT.jar - Your project build JAR
spark-submit \
--master yarn \
--deploy-mode cluster \
--jars "/efs/home/c795701/mssql-jdbc-krb5/target/scala-2.10/mssql-jdbc-krb5_2.10-1.0.jar,/efs/home/c795701/.ivy2/jars/scala-library-2.11.1.jar"
--files /efs/home/c795701/c795701.keytab
--class com.spark.SparkJDBCIngestion \
/efs/home/c795701/ingestionframework/target/ingestionframework-1.0-SNAPSHOT.jar
So apparently JDBC Kerberos authentication is just not possible currently on the executors according to an old JIRA here https://issues.apache.org/jira/browse/SPARK-12312. The behaviour is the same as of version 2.3.2 according to the spark user list and my testing.
Workarounds
Use kinit and then distribute the cached TGT to the executors as detailed here: https://github.com/LucaCanali/Miscellaneous/blob/master/Spark_Notes/Spark_Executors_Kerberos_HowTo.md. I think this technique only works for the user that spark executors run under. At least I couldn't get it to work for my use case.
Wrap the jdbc driver with a custom version that deals with the authentication and then calls and returns a connection from the real MS JDBC driver. Details here: https://datamountaineer.com/2016/01/15/spark-jdbc-sql-server-kerberos/ and the associated repo here: https://github.com/nabacg/krb5sqljdb. I got this technique to work though I had to modify the authentication code for my case.
as Gabor Somogyi said.
you need to use spark > 3.1.0 and keytab and principal arguments
I have 3.1.1.
Throw a keytab on the same path for ALL HOST and machine where you use your code - and keep keytab up to date
add to connection string value integratedSecurity=true;authenticationScheme=JavaKerberos;
reading block will look like:
jdbcDF = (spark.read
.format("com.microsoft.sqlserver.jdbc.spark")
.option("url", url)
.option("dbtable", table_name)
.option("principal", "username#domen")
.option("keytab", "sameALLhostKEYTABpath")
.load()
)
I know how to connect using sql auth. But how to use windows auth?
(use 'clojure.java.jdbc)
(require '[clojure.java.jdbc :as j])
(def db {:classname "com.microsoft.jdbc.sqlserver.SQLServerDriver"
:subprotocol "sqlserver"
:subname "//localhost;database=test;user=sa;password=sa"
})
What abount :subname "//localhost;database=test;integratedSecurity=true; ?
reference : http://technet.microsoft.com/en-us/library/ms378428.aspxhttp://technet.microsoft.com/en-us/library/ms378428.aspx#Connectingintegrated
I've recently set up my first website on Azure. For now, it's basically just the standard MVC4 template website with the Home and Account controllers. In Azure, I have one website, and one SQL database. I can access the SQL Azure database from SSMS, and have set up a login and user to be used by my website.
In my development environment, pointing at my development database, I can access the /Account/Login page fine. I can register, and I can see the new user in my local db. I can also change the connection string to point my development website at my SQL Azure DB, and again I can access /Account/Login and register new users. I can then see those new users in the SQL Azure DB.
The problems happen when I deploy the website to Azure. I have a transform config associated with my publishsettings file, and I can see in the output window when publishing the site to Azure that this transform is applied during deployment. This amends the local development DB connection string to the SQL Azure connection string. I have also verified that this SQL Azure connection string is in the actual web.config file deployed (using FileZilla FTP to retrieve the actual web.config deployed). I can access the home page of my site on [mysite].AzureWebsites.net, but when I click the login link to go to the /Account/Login page, I get the following error:
[ArgumentException: Format of the initialization string does not conform to specification starting at index 0.]
System.Data.Common.DbConnectionOptions.GetKeyValuePair(String connectionString, Int32 currentPosition, StringBuilder buffer, Boolean useOdbcRules, String& keyname, String& keyvalue) +5313265
System.Data.Common.DbConnectionOptions.ParseInternal(Hashtable parsetable, String connectionString, Boolean buildChain, Hashtable synonyms, Boolean firstKey) +124
System.Data.Common.DbConnectionOptions..ctor(String connectionString, Hashtable synonyms, Boolean useOdbcRules) +95
System.Data.SqlClient.SqlConnectionString..ctor(String connectionString) +59
System.Data.SqlClient.SqlConnectionFactory.CreateConnectionOptions(String connectionString, DbConnectionOptions previous) +24
System.Data.ProviderBase.DbConnectionFactory.GetConnectionPoolGroup(DbConnectionPoolKey key, DbConnectionPoolGroupOptions poolOptions, DbConnectionOptions& userConnectionOptions) +167
System.Data.SqlClient.SqlConnection.ConnectionString_Set(DbConnectionPoolKey key) +61
System.Data.SqlClient.SqlConnection.set_ConnectionString(String value) +66
System.Data.Entity.Internal.LazyInternalConnection.InitializeFromConnectionStringSetting(ConnectionStringSettings appConfigConnection) +122
System.Data.Entity.Internal.LazyInternalConnection.TryInitializeFromAppConfig(String name, AppConfig config) +32
System.Data.Entity.Internal.LazyInternalConnection.Initialize() +127
System.Data.Entity.Internal.LazyInternalConnection.get_ProviderName() +13
System.Data.Entity.Internal.LazyInternalContext.InitializeContext() +346
System.Data.Entity.Internal.InternalContext.CreateObjectContextForDdlOps() +17
System.Data.Entity.Database.Exists() +36
[MyWebsite].Filters.SimpleMembershipInitializer..ctor() +105
This seems to suggest that there is an issue with my connection string, but like I mentioned earlier, this exact same connection string worked from my local website.
I've considered that the issue might be firewall-related, but I've checked the settings in the Azure management portal, and the Windows Azure Services firewall rule is applied to allow that access. Also, I've tried removing the firewall rule for my local machine to access the SQL Azure DB, to see would I get a similar exception, but the exception thrown was very obviously firewall-related.
I have also tried to add the SQL Azure connection string through the Azure Management portal (though I didn't see how to specify the provider) - needless to say, I got the same "Format of the initialization..." exception mentioned above.
My connection string in the web.config is in the following format:
<add name="[my connection name]"
connectionString="Server=tcp:abc123.database.windows.net,1433;Database=[my database];User ID=[my login]#abc123;Password=[my password];Trusted_Connection=False;Encrypt=True;Connection Timeout=30;"
providerName="System.Data.SqlClient" />
Any suggestions would be very welcome.
I've FINALLY got to the bottom of it. What I didn't realise until now was that there were 2 connection strings in the web.config that ultimately gets deployed to the Windows Azure website - my own custom one, but another DefaultConnection connection string as well, which has the format:
<add name="DefaultConnection" connectionString="DefaultConnection_ConnectionString" providerName="System.Data.SqlClient" />
obviously, not a valid connection string (and hence the format exception above). You can see this when you download the actual web.config from your Azure website using FTP.
This default connection string isn't anywhere in the web.configs or various transforms in my solution. Looking at the Output window during publishing, there are a number of transforms that get applied to the web.config. I've gone through the various versions of the files that get generated during the build / publish cycle, and none of them have the DefaultConnection connection string in them, not even in the obj\Release\Package\PackageTmp\ folder. So I'm guessing something in the Web Deploy phase is inserting it as the very last modification of the web.config. There is an MSDeployParameterValue element in the publishsettings file that mentions connection strings and the web.config - I guess it could be that.
In the AccountModels.cs file, a reference is made to this DefaultConnection:
public UsersContext()
: base("DefaultConnection")
{
}
This is how the particular connection string is chosen. Changing this parameter to your custom connection string name ensures that your custom database gets used for the various account stuff, and resolves the format exception seen above.
I know this is an old post, but I wanted to share my findings. In my .pubxml file, it stored my localdb connection string, and would not update on any builds, or publishing. I had to manually update the publish file with my azure DB connection strings in order for it to work. Hope this helps save someone time.
About the Connection strings of SQL Azure
The SQL Azure Database service is only available with TCP port 1433. Ensure that your firewall allows outgoing TCP communication on TCP port 1433.
SQL Azure does not support Windows Authentication. The Trusted Connection will always be set to False.
SQL Azure doesn’t support unencrypted connections. You need to specify in your connection string that you want to encrypt the connection.
Connecting to SQL Azure by using OLE DB is not officially supported.
Standard way
Server=tcp:[serverName].database.windows.net;Database=myDataBase;
User ID=[LoginForDb]#[serverName];Password=myPassword;Trusted_Connection=False;Encrypt=True;
Use 'username#servername' for the User ID parameter.
For more information check this out Connection strings for SQL Azure
I hope this will help to you.
Try removing "Encrypt=True;" or try adding "TrustServerCertificate=True;" - Read SqlConnection.ConnectionString Property for more details, but basically:
Beginning in .NET Framework 4.5, when TrustServerCertificate is false (the default)
and Encrypt is true, the server name (or IP address) in a SQL Server
SSL certificate must exactly match the server name (or IP address)
specified in the connection string.
We have a program where users can specify their database connection parameters. The usual suspects including host, port, username, password, and table name. We are connecting to the database using NHibernate. What we'd like to do is be able to build the configuration with NHibernate and then test the connection parameters before continuing with other operations; notifying the user of failure.
Is this possible to do through NHibernate, or will it require using each database type we support's specific driver and creating a custom TestConnection() method for each type?
I realize this is an old post - but I guess an answer to a question never hurts.
I don't think there is a way to explicitly tell NHibernate to test out the connection string. However, when you instantiate the SessionFactory it will attempt to connect to the database. You could wrap your SessionFactory creation in a Try/Catch and handle the error that way.
I use Fluent NHibernate, but I'm sure the following example will still explain the situation.
Dim sf as SessionFactory
Try
sf = CreateSessionFactory()
Catch ex as FluentNHibernate.Cfg.FluentConfigurationException
Messagebox.Show(ex.InnerException.Message)
End Try
The ex.InnerException.Message contains the actual error and will tell you if:
The connection string was invalid
The server could not be found
The user/pass could not be authenticated
To configure Nhibernate you have two options:
Set the dialect when you are building the session factory. This will assign reasonable default value to Nhibernate's ADO and other configuration values.
Manually set the configuration values.
That said, at some point, you need to configure Nhibernate to use the appropriate driver for the database you want to talk to. Which means you need to be able to build Session Factories of different types (your supported database types). To do this you need more than just "host, port, username, password, and table name". You need to know the database type(Dialect).
If you intend to just try to connect the database with every driver available to you not knowing what the database type is, you may run into problems when the database and the dialect don't match. Imagine you use a SqlServer2008 dialect on SqlServer2005 machine. The difference in dialect can cause a particular SqlServer2008 feature you are using not to, obviously, work. Moreover, if you don't stick to basic SQL through out all your code, you may be generating Sql that works, say, in PostgreSql but not in SqlServer (Think sequences and such).
To learn more about configuring Nhibernate read:
Chapter 3: Session Factory Configuration. Specially sections 3.3, 3.4, 3.5 which talk about configuration parameters.
Last note, Nhibernate supports multiple databases. But, for complex domain layers where you rely on database specific constructs, your code doesn't.