I'm using pyodbc to connect to my *.mdb files and store them in a sqlite / spatialite database for further work and analysis. I'm passing DSN like this:
DSN="Driver={%s};DBQ=%s;"%(self.find_mdb_driver(),self.mdbPath)
and then:
conn = pyodbc.connect(DSN)
Problem is, that when I try to pass path with international characters cp1250 "čžš" I get error:
table=cursor.fetchall()
UnicodeEncodeError: 'ascii' codec can't encode character u'\u017e' in position 111: ordinal not in range(128)
Any pointers to this problem ?
Minimal example of the problem:
# -*- coding: utf-8 -*-
import pyodbc
if __name__ == '__main__':
DSN_1="Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=C:\Users\ep5065\Documents\GREDOS\Gredos 2015\Analize omrežja\Komen NOS 2015\Končno stanje\Analiza stanja Komen.mdb;"
DSN_2="Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=C:\Users\ep5065\Documents\GREDOS\Gredos 2015\Gredos modeli\9_2_2015\EP_9_2_2015_v1.mdb;"
conn = pyodbc.connect(DSN_1)
cursor=conn.cursor()
cursor.execute('select * from Branch')
table=cursor.fetchall()
for line in table:
print line
If I use DSN_1 it breaks with above mentioned error, if I use DSN_2 it happily spits table contents.
Related
I am using pyodbc to read a data frame from a sql query but keep getting an error saying 'utf-8' codec can't decode byte 0xa0 in position n: invalid start byte. Code as below. How to fix this error? Thanks
import pyodbc
import pandas as pd
conn = pyodbc.connect('DRIVER={SQL Server};SERVER=SSXBYXSQL107,5269;DATABASE=INV_Q1;Trusted_Connection=yes;')
conn.setencoding(encoding='utf-8')
conn.setdecoding(pyodbc.SQL_CHAR, encoding='utf-8')
conn.setdecoding(pyodbc.SQL_WCHAR, encoding = 'utf-8')
sql_query="select [inventory_desc] from [INV_Q1].[INV_Detail].[Inventory]"
df_inv=pd.read_sql(sql_query, conn)
At least for now, SQL Server does not send Unicode characters as UTF-8; it sends them as UTF-16LE, and UTF-16 is the default encoding expected by pyodbc. Those setencoding/setdecoding calls are not applicable for connections to SQL Server.
As mentioned in the pyodbc wiki:
SQL Server's recent drivers match the [ODBC] specification, so no configuration is necessary. Using the pyodbc defaults is recommended.
First off this is my setup:
Windows 7
MS SQL Server 2008
Python 3.6 Anaconda Distribution
I am working in a Jupyter notebook and trying to import a column of data from a MS SQL Server database using SQLAlchemy. The column in question contains cells which store long strings of text (datatype is nvarchar(max)). This is my code:
engine = create_engine('mssql+pyodbc://user:password#server:port/db_name?driver=SQL+Server+Native+Client+11.0'
stmt = 'SELECT componenttext FROM TranscriptComponent WHERE transcriptId=1265293'
connection = engine.connect()
results = connection.execute(stmt).fetchall()
This executes fine, and imports a list of strings. However when I examine the strings they are truncated, and in the middle of the strings the following message seems to have been inserted:
... (8326 characters truncated) ...
With the number of characters varying from string to string. I did a check on how long the strings that got imported are, and the ones that have been truncated are all limited at either 339 or 340 characters.
Is this a limitation in SQLAlchemy, Python or something else entirely?
Any help appreciated!
Same problem here!
Set up :
Windows Server 2012
MS SQL Server 2016/PostgreSQL 10.1
Python 3.6 Anaconda Distribution
I've tested everything I could, but can't overpass this 33x limitation in field length. Either varchar/text seems to be affected and the DBMS/driver doesn't seem to have any influence.
EDIT:
Found the source of the "problem": https://bitbucket.org/zzzeek/sqlalchemy/issues/2837
Seems like fetchall() is affected by this feature.
The only workaround i found was :
empty_list=[]
connection = engine.connect()
results = connection.execute(stmt)
for row in results:
empty_list.append(row['componenttext'])
This way i haven't found any truncation in my long string field(>3000 ch).
What is wrong with my geomtery? geometry is here.
When I run this command:
SELECT st_geomfromgml('content of file geometry.xml');
this error was thrown:
ERROR: invalid GML representation
I'm using postgres 9.4.1 and PostGIS 2.1.6
Thank you for help
The documentation for ST_GeomFromGML says that the function does "not support SQL/MM curves geometries". It's unfortunate that the error message does not articulate this shortcoming, as the GML has a valid representation to modern software. There is an enhancement ticket to enable this support, but there has not been any movement on this after several years.
As a workaround, you can use GDAL/OGR from (e.g.) Python to read the GML and export the WKB to PostGIS:
from osgeo import ogr
with open('geometry.xml', 'r') as fp:
g = ogr.CreateGeometryFromGML(fp.read())
g.GetArea() # 4519550457.106098
# There is more than one way to insert this to PostGIS, here is one way
import psycopg2
conn = psycopg2.connect('dbname=postgis host=localhost user=postgres port=5432')
curs = conn.cursor()
curs.execute('CREATE TABLE surf(geom geometry);')
curs.execute('INSERT INTO surf(geom) VALUES (%s)', (g.ExportToWkb().encode('hex'),))
conn.commit()
curs.execute('SELECT ST_Area(geom) FROM surf')
curs.fetchone()[0] # 4519550457.07643
The two area calculations (using different methods) are essentially the same, which is reassuring.
I'm having problems when connecting to a legacy database in SQL Server. The problem happens when read a string with special characters and try to insert it into another database (sqlite in development), which throws me the following error:
Encoding::UndefinedConversionError: "\xC3" from ASCII-8BIT to UTF-8
The database has a latin1-like collation. I have set client charset = UTF-8 in my freetds.conf file, and I have encoding: utf8 in my database.yml file.
It actually seems the freetds driver is making the right conversion, but for some reason the ruby library is not aware about it, because the problematic string is ESPA\xC3\x91A, which is actually the right binary representation for ESPAÑA, so calling force_encoding('UTF-8') on the string makes the error dissapear.
As an additional note, when I assign the string to the other model it gives me the following message:
Binary data inserted for `string` type on column `email`
Using https://github.com/rails-sqlserver/tiny_tds solved this and other problems I was having with the ODBC driver.
Please help me to load array value from database using odbc adapter with out using loop statement.
Thanks,
S.Somu
Since you didn't specify a language, here's an example in Python using mxODBC:
import mx.ODBC.unixODBC as mx
db = mx.DriverConnect('DSN=[your DSN here]')
c = db.cursor()
results = c.execute("SELECT * from tableName").fetchall()
If you have more detail - such as a specific language, you might get an answer closer to what you want.