What is wrong with my geomtery? geometry is here.
When I run this command:
SELECT st_geomfromgml('content of file geometry.xml');
this error was thrown:
ERROR: invalid GML representation
I'm using postgres 9.4.1 and PostGIS 2.1.6
Thank you for help
The documentation for ST_GeomFromGML says that the function does "not support SQL/MM curves geometries". It's unfortunate that the error message does not articulate this shortcoming, as the GML has a valid representation to modern software. There is an enhancement ticket to enable this support, but there has not been any movement on this after several years.
As a workaround, you can use GDAL/OGR from (e.g.) Python to read the GML and export the WKB to PostGIS:
from osgeo import ogr
with open('geometry.xml', 'r') as fp:
g = ogr.CreateGeometryFromGML(fp.read())
g.GetArea() # 4519550457.106098
# There is more than one way to insert this to PostGIS, here is one way
import psycopg2
conn = psycopg2.connect('dbname=postgis host=localhost user=postgres port=5432')
curs = conn.cursor()
curs.execute('CREATE TABLE surf(geom geometry);')
curs.execute('INSERT INTO surf(geom) VALUES (%s)', (g.ExportToWkb().encode('hex'),))
conn.commit()
curs.execute('SELECT ST_Area(geom) FROM surf')
curs.fetchone()[0] # 4519550457.07643
The two area calculations (using different methods) are essentially the same, which is reassuring.
Related
I'm having an issue with inserting rows into a database. Just wondering if anyone has any ideas why this is happening? It works when I avoid using fast_executemany but then inserts become very slow.
driver = 'ODBC Driver 17 for SQL Server'
conn = pyodbc.connect('DRIVER=' + driver + ';SERVER=' + server+ \
';UID=' + user+ ';PWD=' + password)
cursor = conn.cursor()
cursor.fast_executemany = True
insert_sql = """
INSERT INTO table (a, b, c)
VALUES (?, ?, ?)
"""
cursor.executemany(insert_sql, insert_params)
---------------------------------------------------------------------------
MemoryError Traceback (most recent call last)
<ipython-input-12-e7e82e4d8c2d> in <module>
2 start_time = time.time()
3
----> 4 cursor.executemany(insert_sql, insert_params)
MemoryError:
There is a known issue with fast_executemany when working with TEXT or NTEXT columns, as described on GitHub here.
The problem is that when pyodbc queries the database metadata to determine the maximum size of the column the driver returns 2 GB (instead of 0, as would be returned for a [n]varchar(max) column).
pyodbc allocates 2 GB of memory for each [N]TEXT element in the parameter array, and the Python app quickly runs out of memory.
The workaround is to use cursor.setinputsizes([(pyodbc.SQL_WVARCHAR, 0, 0)]) (as described here) to coax pyodbc into treating [N]TEXT columns like [n]varchar(max) columns.
(Given that [N]TEXT is a deprecated column type for SQL Server it is unlikely that there will be a formal fix for this issue.)
While this issue was solved for the OP by Gord Thompson's answer, I wanted to note that the question as written applies to other cases where a MemoryError may occur, and fast_executemany actually can throw that in other circumstances beyond just usage of [N]TEXT columns.
In my case a MemoryError was thrown during an attempt to INSERT several million records at once, and as noted here, "parameter values are held in memory, so very large numbers of records (tens of millions or more) may cause memory issues". It doesn't necessarily require tens of millions to trigger, so YMMV.
An easy solution to this is to identify a sane number of records to batch per each execute. Here's an example if using a Pandas dataframe as a source (establish your insert_query as usual):
batch_size = 5000 # Set to a desirable batch size
with connection.cursor() as cursor:
try:
cursor.fast_executemany = True
# Iterate each batch chunk using numpy's split
for chunk in np.array_split(df, batch_size):
cursor.executemany(insert_query,
chunk.values.tolist())
# Run a single commit at the end of the transaction
connection.commit()
except Exception as e:
# Rollback on any exception
connection.rollback()
raise e
Hope this helps anyone who hits this issue and doesn't have any [N]TEXT columns on their target!
In my case, the MemoryError was because I was using a very old driver 'SQL Server'. Switched to the newer driver ('ODBC Driver 17 for SQL Server') as described in the link below and it worked:
link
I'm using pyodbc to connect to my *.mdb files and store them in a sqlite / spatialite database for further work and analysis. I'm passing DSN like this:
DSN="Driver={%s};DBQ=%s;"%(self.find_mdb_driver(),self.mdbPath)
and then:
conn = pyodbc.connect(DSN)
Problem is, that when I try to pass path with international characters cp1250 "čžš" I get error:
table=cursor.fetchall()
UnicodeEncodeError: 'ascii' codec can't encode character u'\u017e' in position 111: ordinal not in range(128)
Any pointers to this problem ?
Minimal example of the problem:
# -*- coding: utf-8 -*-
import pyodbc
if __name__ == '__main__':
DSN_1="Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=C:\Users\ep5065\Documents\GREDOS\Gredos 2015\Analize omrežja\Komen NOS 2015\Končno stanje\Analiza stanja Komen.mdb;"
DSN_2="Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=C:\Users\ep5065\Documents\GREDOS\Gredos 2015\Gredos modeli\9_2_2015\EP_9_2_2015_v1.mdb;"
conn = pyodbc.connect(DSN_1)
cursor=conn.cursor()
cursor.execute('select * from Branch')
table=cursor.fetchall()
for line in table:
print line
If I use DSN_1 it breaks with above mentioned error, if I use DSN_2 it happily spits table contents.
Maybe related: Stack overflow: Windoes does not support UTF-8
I have a script which I can source from Rstudio, but when I try to source it from Rgui.exe or try to BATCH CMD run, I get the following error in my Rout file:
Error in easy_clean$Sv_Karakter: $ operator is invalid for atomic vectors
The reason is that the database table I am quering have a latin charachter 'ø' in its name (se third line below). So the result of my query is this (as per str(easy)):
"";"x"
"1";"42000 102 [Microsoft][ODBC SQL Server Driver][SQL Server]Incorrect syntax near '¸'."
"2";"[RODBC] ERROR: Could not SQLExecDirect 'select *
from PrøveDeltager a
left outer join Aftaler b
ON b.Cpr = a.CPR
LEFT OUTER JOIN GODK c
ON c.GODK_ID = b.GODK_ID
where a.slut >= '20140808'
AND a.slut <='20140818'
AND a.Branche = 'vvs'
AND a.SaleID is not null
AND a.CPR in (select x.CPRNR from Statistik x)
order by Sv_Karakter'"
In rstudio the query works.
Sys.getlocale('LC_CTYPE')returns Danish_Denmark.1252 in both R.gui and Rstudio - so I don't know how to fix this.
I did find this link to developer.r-project which discus windows locales (quite old though).
For now I have created a database view without the 'ø' - that view I can call without problems from R.
From sessionInfo I can say that:
Rstudio R is 64 bit, and R.exe is 32 bit.
Other than that, the only difference is this, for Rstudio:
loaded via a namespace (and not attached): 1 tools_3.1.0
Since I can't write my database credentials, I can't create a reproducible example. But here is the script. http://pastebin.com/XwdZPhL7
Two ways I can imagine that would yield an error with that code: one is if this results in a single column:
easy[!is.na(easy$Sv_Karakter),]
In that case, the result would be a vector because of the default action of the "[" function to create atomic vectors from single column results. The attempts to extract that column would fail with that error. The other case of failure (but perhaps not that error) would be when there was no 'Sv_Karakter' column in the 'easy' dataframe.
Better efforts at documentation by offering str(easy) or dput(head(easy)) are needed.
Your query returned an error message and it was in the form of an R vector. So that explains the particular error. You now need to figure out how your db connection is getting messed up, as Andreas was saying.
Is there a way to read and write to sqlite3 from octave?
I'm thinking something along the lines of RODBC in R or the sqlite3 package in python, but for octave.
I looked on octave-forge http://octave.sourceforge.net/packages.php
But could only find the 'database' package, which only supports postgresql.
Details:
OS: Ubuntu 12.04
Octave: 3.6.2
sqlite: 3.7.9
I realise this is an old question, but most answers here seem to miss the point, focusing on whether there exists a bespoke octave package providing a formal interface, rather than whether it is possible to perform sqlite3 queries from within octave at all in the first place.
Therefore I thought I'd provide a practical answer for anyone simply trying to access sqlite3 via octave; it is in fact trivial to do so, I have done so myself many times.
Simply do an appropriate system call to the sqlite3 command (obviously this implies you have an sqlite3 client installed on your system). I find the most convenient way to do so is to use the
sqlite3 database.sqlite < FileContainingQuery > OutputToFile
syntax for calling sqlite3.
Any sqlite3 commands modifying output can be passed together with the query to obtain the output in the desired format.
E.g. here's a toy example plotting a frequency chart from a table which returns appropriate scores and counts in csv format (with headers and runtime stats stripped from the output).
pkg load io % required for csv2cell (used to collect results)
% Define database and Query
Database = '/absolute/path/to/database.sqlite';
Query = strcat(
% Options to sqlite3 modifying output format:
".timer off \n", % Prevents runtime stats printed at end of query
".headers off \n", % If you just want the output without headers
".mode csv \n", % Export as csv; use csv2cell to collect results
% actual query
"SELECT Scores, Counts \n",
"FROM Data; \n" % (Don't forget the semicolon!)
);
% Create temporary files to hold query and results
QueryFile = tempname() ; QueryFId = fopen( QueryFile, 'w' );
fprintf( QueryFId, Query ); fclose( QueryFId);
ResultsFile = tempname();
% Run query
Cmd = sprintf( 'sqlite3 "%s" < "%s" > "%s"', Database, QueryFile, ResultsFile );
[Status, Output] = system( Cmd );
% Confirm query succeeded and if so collect Results
% in a cell array and clean up temp files.
if Status != 0, delete( QueryFile, ResultsFile ); error("Query Failed");
else, Results = csv2cell( ResultsFile ); delete( QueryFile, ResultsFile );
end
% Process Results
Results = cell2mat( Results );
Scores = Results(:, 1); Counts = Results(:, 2);
BarChart = bar( Scores, Counts, 0.7 ); % ... etc
Et, voilà
According to Octave-Forge the answer is no.
Interface to SQL databases, currently only postgresql using libpq.
But you can write your own database package using the Octave C++ API with SQLite C API
As you already found out, the new version of the database package (2.0.0) only supports postgreSQL. However, old versions of the package also supported MySQL and SQLite (the last version with them was version 1.0.4).
Its problem is that the old database packages do not work with the new Octave and SWIG versions (I think the last version of Octave where the database package worked was 3.2.4). Aside the lack of maintainer (package was abandoned for almost 4 years) its use of SWIG was becoming a problem since it made more difficult for other developers to step in. Still, some users tried to fix it and some half fixes have been done (but never released). See bug #38098 and Octave's wiki page on the database package for some reports on making it work with SQLite in Octave 3.6.2.
The new version of the package is a complete restart of the package. Would be great if you could contribute with development for SQLite bindings.
Check out this link http://octave.1599824.n4.nabble.com/Octave-and-databases-td2402806.html which asks the same question regarding MySQL.
In particular this reply from Martin Helm points the way to using JDBC to connect to any JDBC supported database -
"Look at the java bindings in the octave java package (octave-forge), it is
maintained and it works. Java is very strong and easy for database handling.
Use that and jdbc driver for mysql to connect to mysql (or with the
appropriate jdbc friver everything else which you can imagine). That is what I
do when using db queries from octave. Much easier and less indirect than
invoking scripts and parsing output from databse queries.
As far as I remeber the database package is somehow broken (at least I never
was able to use it). "
I know this thread is pretty old, but for anybody else out there looking for a similar solution, this project seems to provide it.
https://github.com/markuman/go-sqlite
Please help me to load array value from database using odbc adapter with out using loop statement.
Thanks,
S.Somu
Since you didn't specify a language, here's an example in Python using mxODBC:
import mx.ODBC.unixODBC as mx
db = mx.DriverConnect('DSN=[your DSN here]')
c = db.cursor()
results = c.execute("SELECT * from tableName").fetchall()
If you have more detail - such as a specific language, you might get an answer closer to what you want.