I'm trying to reverse generate a schema from a MSSQL database using Propel 2. I've set up my YAML configuration file as usual:
dbname:
adapter: mssql
classname: Propel\Runtime\Connection\ConnectionWrapper
dsn: "dblib:host=123.456.789.012;dbname=dbname"
user: username
password: password
attributes:
When I run the command propel reverse 'dbname' I receive the error:
[Propel\Generator\Exception\EngineException]
Column "" declared twice in table "Status"
Which is obviously thrown here:
https://github.com/propelorm/Propel2/blob/master/src/Propel/Generator/Model/Table.php#L499
#r499
Why does Propel attempt to add 'empty' columns? My SQL server management studio does not display empty columns at all when I look at the design of the DB table Status, it only displays the two columns it contains (uid and name).
Edit:
So I went digging into the code of Propel, and it seems to go wrong here:
https://github.com/propelorm/Propel2/blob/62859fd0ed3520b7d7afbbdeac113edaf160982b/src/Propel/Generator/Reverse/MssqlSchemaParser.php#L124
protected function addColumns(Table $table)
{
$dataFetcher = $this->dbh->query("sp_columns '" . $table->getName() . "'");
foreach ($dataFetcher as $row) {
$name = $this->cleanDelimitedIdentifiers($row['COLUMN_NAME']);
$table->getName() correctly returns the right table name. When I print dataFetcher it's a PDO object. However:
$row gives the following array:
Array(
[0] => My DBname
[1] => My DBprefix
[2] => Status
[3] => uid
[4] => 4
[5] => int identity
etc. no string indices hence COLUMN_NAME is empty.
(Posted on behalf of the OP):
This is a bug in the Propel MSSQL schema parser: https://github.com/propelorm/Propel2/issues/863.
Related
I use to use sqlsrv_connect but changed it to PDO.
Now i got this syntax error when using åäö.
When i used sqlsrv_connect i could do this:
SELECT Order, [Benämning], [Vår ref] FROM table
and it worked.
Now i'm trying to figure out how to do it with PDO.
So i tried:
SELECT Order, [Benämning], Antal FROM table
And got this error:
Operand type clash: text is incompatible with float
And i tried:
SELECT Order, Benämning, Antal FROM table
And i got error:
Incorrect syntax near '�'.
In the connection i added utf8:
$sql = new PDO("odbc:Driver=$driver;server=$serverName,$port;Database=$database;ConnectionPooling=0", $uid, $pwd,
array(
PDO::ATTR_PERSISTENT => true,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES utf8"
)
);
Now, when pasting this i can se: PDO::MYSQL_ATTR_I... I'm connecting to SQL.. not MYSQL. Can this be the problem?
If i remove the "Benämning" column and just select columns without åäö or space the select works just fine.
UPDATE
I got åäö to work with sqlsrv instead of odbc.
You simply need to use the same driver (PHP Driver for SQL Server) and the same connection options when you create the PDO instance.
<?php
...
try
$sql = new PDO("sqlsrv:Driver=$driver;server=$server,$port;Database=$database", $uid, $pwd);
$conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
...
} catch( PDOException $e ) {
...
}
try {
$sql = "SELECT [Vår ref], ... FROM ...";
$stmt = $conn->prepare($sql);
$stmt->execute();
} catch( PDOException $e ) {
...
}
?>
Some additional notes:
You need to quote the column name ([Vår ref]) to make the string literal a valid SQL Server identifier.
Use driver specific PDO connection attributes (PDO::SQLSRV_xxx) to add driver specific features. The PDO::MYSQL_ATTR_INIT_COMMAND option is useful if you connect to MySQL instance.
For id i use uniqueidentyfier, and in queries insert i write: newid() in, but, how do this in Query builder?
{
$db = \Config\Database::connect();
$data = [
'id' => 'newid()',
'id_zgloszenia' => $idpp,
'response' => $response_pp,
'header_response' => $head_res,
'data_datetime' => 'getdate()',
];
$builder = $db->table('DOM5_PP_LOGI');
$builder->insert($data);
}
I try like above, and also: $builder->set('id', uniqid());
But i had error:
[Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Conversion failed when converting from a character string to uniqueidentifier.
You can try this to generate UUID by MSSQL:
$builder->set('id', 'NEWID()', FALSE);
Explain: set() will also accept an optional third parameter ($escape), that will prevent data from being escaped if set to FALSE. To illustrate the difference, here is set() used both with and without the escape parameter.
You can get find out more here: https://codeigniter.com/userguide3/database/query_builder.html#id8
I need to connect a Yii2 model to an Oracle database but it is not working despite reading several answers on this topic.
The Oracle login credentials that I use to manually log in are similar to:
host: hosturl.abc.com
port: 1521
SID: sidname
user: username
password: passABC
In the manual login process, I then need to select the appropriate Schema, after which I can see all tables.
I am not clear how to include these details in the oci connection. Here is what I have done without success:
'db2' => [
'class' => 'yii\db\Connection',
'dsn' => 'oci:dbname=//hosturl.abc.com:1521/sidname;charset=UFT-8',
'username' => 'username',
'password' => 'passABC'
],
I get an error "The table does not exist: TableName" however, I get this message even if I supply completely random details ie wrong username, wrong host, wrong password etc)
I am not clear why I cannot connect. Perhaps the Oracle database is rejecting all queries? Do I need to include the SID in the connection credentials?
Any help appreciated
UPDATE
I think there must be a more general problem with my installation. I am trying a raw connection as follows:
$conn = oci_connect("username", "passABC", "//hosturl.abc.com/sidname");
if (!$conn) {
print "There is a problem";
} else {
print "Connected to Oracle!";
}
This returns the following error:
Call to undefined function oci_connect()
Any suggestions? This looks like a problem with my php setup??
SOLVED!
1) oci_connect not working: SOLVED
Please see this post for more details How do I connect PHP 7.x to Oracle database on RedHat / CentOS?. This fix was required for Yii2 to connect to Oracle.
2) Yii2 db credentials for an Oracle database
/common/config/main-local:
'db' = [
'class' => 'yii\db\Connection',
'dsn' => 'oci:dbname=//hosturl:1521/SID;charset=UTF8',
'username' => 'dbusername',
'password' => 'dbpassword',
]
I tried a number of Yii2 oci8 extensions but none of them worked for me. Instead, the fix (1) for the oci_connect worked and no additional Yii2 drivers were required. I was able to use the standard 'yii\db\Connection' class.
3) Database Schema
Within the Oracle database we have a number of schemas and I wasn't sure how to query a specific table within a particular schema. Ultimately the solution was simple: Within each model, prepend the schema name to the table name. For example:
class myTable extends ActiveRecord
{
public static function tableName()
{
return 'schemaName.myTable';
}
I'd like to export tom_test2 postgresql table to elastic search. The table has 176805 rows:
=> select count(*) from tom_test2;
count
--------
176805
(1 row)
The following logstach conf file import correctly my data to elastic search:
input {
jdbc {
# Postgres jdbc connection string to our database, mydb
jdbc_connection_string => "xxx"
# The user we wish to execute our statement as
jdbc_user => "xxx"
jdbc_password => "xxx"
# The path to our downloaded jdbc driver
jdbc_driver_library => "xxx"
# The name of the driver class for Postgresql
jdbc_driver_class => "org.postgresql.Driver"
# our query
statement => "select * from tom_test2"
}
}
output {
elasticsearch {
hosts => ["xxx"]
index => "tom"
document_type => "tom_test"
}
}
In elastic search:
GET tom/tom_test/_search
"hits": {
"total": 176805,
"max_score": 1,
}
I'm deleting my index in elastic search:
delete tom
And I now would like to do the same operation using jdbc_page_size in case my data becomes bigger, my logstach conf file is now:
input {
jdbc {
# Postgres jdbc connection string to our database, mydb
jdbc_connection_string => "xxx"
# The user we wish to execute our statement as
jdbc_user => "xxx"
jdbc_password => "xxx"
# The path to our downloaded jdbc driver
jdbc_driver_library => "xxx"
# The name of the driver class for Postgresql
jdbc_driver_class => "org.postgresql.Driver"
# our query
statement => "select * from tom_test2"
jdbc_page_size => 1000
jdbc_paging_enabled => true
}
}
output {
elasticsearch {
hosts => ["xxx"]
index => "tom"
document_type => "tom_test"
}
}
My count is now wrong:
GET tom/tom_test/_search
"hits": {
"total": 106174,
"max_score": 1,
}
as 176805-106174=70631 rows are missing
The reason you are facing this - you have ordering problems: your query doesn't controlls the order in which the data is received, and in general postgresql should not guarantie that in unordered consequent paging calls you don't fetch the same data: this produces situation when some data will be not fetched at all, and some data will be fetched multiple times :( even when the data is not modified during these calls, the background vacuum worker may change the order of the data in the physical file, and thus reproduce described situation.
Either add order to your statement SELECT * FROM tom_test2 ORDER BY id and page your data. But be aware: in this case your upload to elasticsearch will not assure the exact replica of the table at moment of time. The cause of that will be, that during logstash processing of consequent paging request the update of data in upcoming page introduced, i.e. you are uploading at the moment page 1 to 10000 and update happened at data on page 10001 and 20000, and then later otherwise... so you have problem in consistency of your data.
Or if you want to fetch all the data and generously use memory on logstash... , then you need to control the jdbc_fetch_size parameter: i.e. you are performing the same SELECT * FROM tom_test2. With this approach you will create a single query resultset, but will "pump" it in pieces, and data modification during your "pumping" will not cause you: you will be fetching the state at the moment of query start.
Because ordering is not guaranteed between queries in jdbc_page_size as WARNED in the documentation of jdbc_paging_enabled.
I recommend using jdbc_fetch_size instead of using jdbc_page_size as the documentation also says that for large result-sets.
P.S: sometimes ;) asking your questions at http://discuss.elastic.co is better answered by elastic maintainers
I have a trouble with dbutil and db.
Below is my code (a constructor of my model):
public function __construct(){
// I save my database configuration in CI session
$this->load->library('session');
$config = $this->session->userdata("db_config");
if($config){ // $config is now equal contains of my database configuration
$this->db = $this->load->database($config, TRUE);
}else{
$this->db = $this->load->database();
}
$this->load->dbutil();
echo '<pre>'.print_r($this->db, TRUE).'</pre>';
echo '<pre>'.print_r($this->dbutil, TRUE).'</pre>';
}
I found that $this->db and $this->dbutil point to different databases.
$this->db point to database config from session:
....
[username] => root
[password] => toor
[hostname] => localhost
[database] => coba
....
While $this->dbutil point to database config from my configuration file (application/config/database.php):
....
[username] => root
[password] => toor
[hostname] => localhost
[database] => module_devel
....
This is not expected, since I expect both of $this->db and $this->dbutil point to the same database
I've also check PHP: CodeIgniter; Managing two db connections; variable database parameters. But, for me the solution doesn't work at all.
So, anyone can discover what's wrong here?
It seems that it is a bug of CodeIgniter. But, dbutil is not the only we can use.
There are many alternatives to emulate what dbutil can do.
In my case, I want to generate a "create table" script. I end up using
$query = $this->db->query("SHOW CREATE TABLE `$table_name`");
And to generate SQL insert script, I do manual select, loop through the records and produce the script.
Hope this can help anyone else with similar problem