I have a problem loading data in database1 (default). You see, the system only has to load data that is in the database2 (source). the system works in the machine of my confessor but it has two different ports loaded and uses docker, I have the SQL server installed. The system starts, the problem is that when I want to load a data in the database1 it tells me that this data does not exist in the database2, then it does not. Now, if I try to load data that is not in the database2 if it loads correctly. I searched how to change the ports of the SQL Server but I did not get it. Can anyone help me?
DATABASES = {
'default': {
'ENGINE': 'sql_server.pyodbc',
'NAME': 'database1',
'HOST': 'name\\name',
'PORT': '',
'USER': 'user1',
'PASSWORD': 'password1',
'OPTIONS': {
'driver': 'ODBC Driver 13 for SQL Server',
}
},
'source': {
'ENGINE': 'sql_server.pyodbc',
'NAME': 'database2',
'HOST': 'name\\name',
'PORT': '',
'USER': 'user2',
'PASSWORD': 'password2',
'OPTIONS': {
'driver': 'ODBC Driver 13 for SQL Server',
}
}
This is the configuration:
def decide_on_model(model):
"""Small helper function to pipe all DB operations of a worlddata model to the world_data DB"""
return 'source' if model._meta.app_label == 'source' else None
class TutoriasRouter:
"""
Implements a database router so that:
* Django related data - DB alias `default` - MySQL DB `world_django`
* Legacy "world" database data (everything "non-Django") - DB alias `world_data` - MySQL DB `world`
"""
def db_for_read(self, model, **hints):
return decide_on_model(model)
# def db_for_write(self, model, **hints):
# return decide_on_model(model)
def db_for_write(self, model, **hints):
return 'default'
def allow_relation(self, obj1, obj2, **hints):
# Allow any relation if both models are part of the worlddata app
# if obj1._meta.app_label == 'source' and obj2._meta.app_label == 'source':
# return True
# # Allow if neither is part of worlddata app
# elif 'source' not in [obj1._meta.app_label, obj2._meta.app_label]:
# return True
# # by default return None - "undecided"
return True
def allow_migrate(self, db, app_label, model_name=None, **hints):
# allow migrations on the "default" (django related data) DB
if db == 'default' and app_label != 'source':
return True
# allow migrations on the legacy database too:
# this will enable to actually alter the database schema of the legacy DB!
# if db == 'source' and app_label == "source":
# return True
return False
`
Related
How do I connect to the PostgreSQL database with the following connection info?
I'm using Jupyter Notebook.
from sqlalchemy import create_engine
POSTGRES_DIALECT = 'postgresql'
POSTGRES_SERVER = 'server'
POSTGRES_DBNAME = 'db'
POSTGRES_SCHEMA = 'public'
POSTGRES_USERNAME = 'user'
POSTGRES_PASSWORD = 'password'
postgres_str = ('{dialect}://{username}:{password}#{server}:{schema}/{dbname}'.format(
dialect=POSTGRES_PROVIDER,
server=POSTGRES_SERVER,
dbname=POSTGRES_DBNAME,
schema=POSTGRES_SCHEMA,
username=POSTGRES_USERNAME,
password=POSTGRES_PASSWORD
))
# Create the connection
cnx = create_engine(postgres_str)
ValueError: invalid literal for int() with base 10: 'public'
You are subbing in "schema" where the port belongs. 'public' is not a valid port number.
I have a huge dataset in SQL server, I want to Connect the SQL server with python, then use pyspark to run the query.
I've seen the JDBC driver but I don't find the way to do it, I did it with PYODBC but not with a spark.
Any help would be appreciated.
Please use the following to connect to Microsoft SQL:
def connect_to_sql(
spark, jdbc_hostname, jdbc_port, database, data_table, username, password
):
jdbc_url = "jdbc:sqlserver://{0}:{1}/{2}".format(jdbc_hostname, jdbc_port, database)
connection_details = {
"user": username,
"password": password,
"driver": "com.microsoft.sqlserver.jdbc.SQLServerDriver",
}
df = spark.read.jdbc(url=jdbc_url, table=data_table, properties=connection_details)
return df
spark is a SparkSession object, and the rest are pretty clear.
You can also pass pushdown queries to read.jdbc
I use pissall's function (connect_to_sql) but I modified it a little.
from pyspark.sql import SparkSession
def connect_to_sql(
spark, jdbc_hostname, jdbc_port, database, data_table, username, password
):
jdbc_url = "jdbc:mysql://{0}:{1}/{2}".format(jdbc_hostname, jdbc_port, database)
connection_details = {
"user": username,
"password": password,
"driver": "com.mysql.jdbc.Driver",
}
df = spark.read.jdbc(url=jdbc_url, table=data_table, properties=connection_details)
return df
if __name__=='__main__':
spark = SparkSession \
.builder \
.appName('test') \
.master('local[*]') \
.enableHiveSupport() \
.config("spark.driver.extraClassPath", <path to mysql-connector-java-5.1.49-bin.jar>) \
.getOrCreate()
df = connect_to_sql(spark, 'localhost', <port>, <database_name>, <table_name>, <user>, <password>)
or you can use SparkSession .read method
df = spark.read.format("jdbc").option("url","jdbc:mysql://localhost/<database_name>").option("driver","com.mysql.jdbc.Driver").option("dbtable",<table_name>).option("user",<user>).option("password",<password>).load()
I am new to Capistrano.
I need to get the server properties in tasks using a loop. I am using this code:
server 'IP_address', user: 'root', password: 'pass', roles: %w{web}, database: 'production1'
server 'IP_address', user: 'root', password: 'pass', roles: %w{web}, database: 'production2'
task :backup_FilesDatabaseServerfiles do
on roles (:web) do |h|
puts h.database
end
end
How can I fetch database options in the above task?
This should do it.
task :backup_FilesDatabaseServerfiles do
on roles :web do |server|
p server.properties.database
end
end
Per Capistrano 3: use server custom variable in task
I am trying to connect to a redshift server and run some sql commands. Here is the code that I have written:
Class.forName("org.postgresql.Driver")
val url: String = s"jdbc:postgres://${user}:${password}#${host}:${port}/${database}"
val connection: Connection = DriverManager.getConnection(url, user, password)
val statement = connection.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY)
val setSearchPathQuery: String = s"set search_path to '${schema}';"
statement.execute(setSearchPathQuery)
But I am getting the following error:
java.sql.SQLException: No suitable driver found for jdbc:postgres://user:password#host:port/database
But when I am using play framework's default database library with the same configuration, then I am able to connect to database successfully. Below is the configuration for the default database:
db.default.driver=org.postgresql.Driver
db.default.url="postgres://username:password#hostname:port/database"
db.default.host="hostname"
db.default.port="port"
db.default.dbname = "database"
db.default.user = "username"
db.default.password = "password"
The problem was with the url. The correct format for the url is:
jdbc:postgresql://hostname:port/database
I need to config Laravel 4 to use the ODBC PDO Driver for SQL Server 2000
I have tested it in a plain PHP file and it works perfectly, However, I can't get the right config inside Laravel.
This is my connection string >
$conn = new PDO("odbc:Driver={SQL Server};Server=MyHOST;Database=myDb;User Id=sa;Password=;");
I got this so far in the Laravel config/database.php
Edit
Ok, i followed the instructions from https://github.com/ccovey/odbc-driver and i configured:
I changed it to
'odbc' => array(
'driver' => 'odbc',
'dsn' => 'Driver={SQL Server};Server=MyServer',
'grammar' => 'SqlServerGrammar',
'username' => 'user',
'password' => 'pass',
'database' => 'staPPM',
),
Im getting an error FatalErrorException, Grammar not Found
Laravel 4 still does not support ODBC directly, you'll have to do it yourself or you can try to use this driver: https://github.com/ccovey/odbc-driver.
You'll have to add connection that should look something like:
'odbc' => array(
'driver' => 'odbc',
'dsn' => 'Driver={iSeries Access ODBC Driver};System=my_system_name;',
'grammar' => 'DB2',
'username' => 'foo',
'password' => 'bar',
'database' => '',
'grammar' => 'SqlServerGrammar',
),
As noted in the package docs, you have to provide a valid DSN to connect to your server: those ones are examples of valid connection strings:
"Driver={SQL Server};Server=(local);Trusted_Connection=Yes;Database=AdventureWorks;"
"Driver={Microsoft ODBC for Oracle};Server=ORACLE8i7;Persist Security Info=False;Trusted_Connection=Yes"
"Driver={Microsoft Access Driver (*.mdb)};DBQ=c:\bin\Northwind.mdb"
"Driver={Microsoft Excel Driver (*.xls)};DBQ=c:\bin\book1.xls"
"Driver={Microsoft Text Driver (*.txt; *.csv)};DBQ=c:\bin"
To know if your DSN is valid, you better test the DNS outside Laravel.
If you have access to a Linux, you can test it by doing:
apt-get install unixodbc
isql -v DSN_NAME db_username db_password
And it should answer with:
+---------------------------------------+
| Connected! |
| |
| sql-statement |
| help [tablename] |
| quit |
| |
+---------------------------------------+
SQL>
There's a bug in ccovey's source code and, for now, you should alter the source of ODBCDriverConnection to:
/**
* Default grammar for specified Schema
* #return Schema\Grammars\Grammar
*/
protected function getDefaultSchemaGrammar()
{
return $this->withTablePrefix(new \Illuminate\Database\Schema\SqlServerGrammar);
}
I'll open an issue in the package Github so they get this fixed.