I am trying to create a tkinter app from Python. In it I would nav to an excel sheet and load the data, Python will then select * into a tablename. The app reads the username and as such I will append that to the filename.
I cannot get past the select into.
My endgame is for users to have an app that they can confirm their datafiles, then simply upload them as reliable data.
here is what I have so far -
it picks up the filename successfully, once navigated to and it looks like the parameters are passed but fails on commit. my connection works, as I used it for specific files from pandas before and trying to open the scope now.
source= str(var_reason.get())
updatedby = os.getlogin()
filename = filedialog.askopenfilename(initialdir = "/",
title = "Select a File",
filetypes = (("Excel files",
"*.xlsx*"),
("all files",
"*.*")))
strSQL = """Select * into [{}] FROM OPENROWSET('Microsoft.ACE.OLEDB.12.0', 'Excel 12.0;Database=[{}];HDR=YES', 'SELECT * FROM [planning]')""".format(source,filename)
cur = conn.cursor()
cur.execute(strSQL)
conn.commit()
Related
So as the title suggest, I need to do an import of 2 Excel (.xlsx) files from my local machine (c:\temp) into one SQL Server table. Each of the files only contains one sheet, but the sheet names will differ. The columnnames and no of columns on each file is identical.
If I select one specific excel file through SSIS via Excel Connection Manager, it extracts the data perfectly and inserts it into my destination SQL table.
The problem comes in when I add a ForEach Loop Container and want to loop through the c:\temp directory to read the 2 files. Somewhere I am missing a setting and keep getting various "connect to Excel" errors.
Please assist with the following:
I am unsure how to specify the Excel file path. Is the below correct? I used to select the exact file here when loading only 1 file:
Then it seems I need to create variables, so I did below:
Then I am not sure if I should add an expression to my ForEach loop and which mappings would be correct?
And lastly, I am not sure whether to put the filename or sheetname as variable below. I tried the filepath, but get the following error:
Please help as I am totally lost with this.
UPDATE
OK, I have now done the following:
Added a SheetName variable (which I think the Value is maybe incorrect). I am trying to tell it to only read the first sheet.
Then my Excel connection string looks like this:
Provider=Microsoft.ACE.OLEDB.12.0;Data Source=;Extended Properties="EXCEL 12.0 XML;HDR=NO";
My ForEach loop:
And my Excel source:
I get the following error:
[Book 2] Error: Opening a rowset for "Sheet1$" failed. Check that the object exists in the database.
It seems like your biggest issue is in regards to getting the sheetname which can vary, and the only way I know how to do this is with a script task.
So inside your foreach loop (store filepath to the Excel file) as variable, add a script task before you enter the data flow.
First of all start with knowing you connection string (I use this site for help: https://www.connectionstrings.com/excel/)
Set your read/write variable to [SheetName] and read to FilePath
Code:
var cstr = string.Format(#"Provider=Microsoft.ACE.OLEDB.12.0;Data Source={0};Extended Properties=""Excel 12.0 Xml; HDR = YES"";"
, Dts.Variables["FilePath"].ToString()); //Enter your connection string for OleDB conn
using (var conn = new System.Data.OleDb.OleDbConnection(cstr))
{
conn.Open();
var sheets = conn.GetOleDbSchemaTable(System.Data.OleDb.OleDbSchemaGuid.Tables, null);
//Since there is only 1 for sure.
Dts.Variables["SheetName"] = sheets.Rows[0]["TABLE_NAME"].ToString();
}
Now you have the SheetName in a variable (this will have the $ in the sheetname that you need as well), set up another variable called SQL and define it as "Select * from [" + SheetName + "]".
Now use the variable SQL in your DataFlow Source.
conn = pyodbc.connect(r'DRIVER={SQL Server Native Client 11.0};SERVER=localhost\<redacted>;DATABASE=master;UID=<redacted>;PWD=<redacted>')
cursor = conn.cursor()
query = """SELECT <redacted> FROM <redacted> WHERE <redacted>"""
row = cursor.execute(query).fetchone()
dummyName = row[0]
cursor.close()
cursor = conn.cursor()
query = """SELECT <redacted> FROM <redacted> WHERE <redacted>"""
print query
row = cursor.execute(query)
print row.fetchone()
This code properly connects to the db and executes the first query on the first db. However, when it executes the second query on the other db, it doesn't return any data and I get a popup window saying python.exe has stopped working when I try to fetch any rows, after which my program crashes. I checked and the query I'm trying to execute is a valid query that works properly from the master db and from the same account I'm connected to in the code.
The problem was we were using an old version of pyodbc. I updated it to the new version and now it works perfectly.
Let's say I inserted a file into a varbinary(max) field like so:
CREATE TABLE myTable
(
FileName nvarchar(60),
FileType nvarchar(60),
Document varbinary(max)
)
GO
INSERT INTO myTable(FileName, FileType, field_varbinary)
SELECT
'Text1.txt' AS FileName,
'.txt' AS FileType,
*
FROM
OPENROWSET(BULK N'C:\Text1.txt', SINGLE_BLOB) AS Document
GO
Of course my file now looks like this:
0xFFD8FFE000104A46494600010101004800....
Is there a simple and elegant way to retrieve this file?
My preference is to open it right away in a temp folder rather than saving and then viewing and deleting. In MS Access, this is as simple as using an Attachment field and double clicking to upload/download.
Since there is no built-in functionality in SSMS for this task I usually write a simple LINQPad script that extracts my varbinary column and writes it to the file system.
Something like this:
var results = from p in myTable
where p.ID == ... //your condition here
select p;
foreach (var item in results)
{
File.WriteAllBytes("C:\\" + item.FileName + item.FileType , item.Document.ToArray());
}
I am working with C# and ASP.NET, and I ended up doing this using a Generic Handler, later linking to it elsewhere in my website project:
public class ImageProvider : IHttpHandler {
public string connString = "...";
public void ProcessRequest(HttpContext context)
{
context.Response.ContentType = "image/jpeg";
string sqlSelectQuery = "select img from table"
SqlConnection conn = new SqlConnection(connString);
conn.Open();
SqlCommand cmd = new SqlCommand(sqlSelectQuery, conn);
byte[] img = (byte[])cmd.ExecuteScalar();
context.Response.BinaryWrite(img);
}
I have a database in sql server called zd and a table called user_tab_columns. I want to export in bulk or write to excel the result of the query statement. The code that I tried to mimic from different sources ended up giving me error messages.
In the database zd and table user_tab_columns, the field are as below:
Here is an example of my code below:
ValueError with Pandas - shaped of passed values
error message SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape
import pyodbc
import pandas as pd
import os
cnxn = pyodbc.connect("Driver={SQL Server Native Client 11.0};"
"Server=DPC;"
"Database=zD;"
"trusted_connection=yes;")
cursor = cnxn.cursor()
script = """
SELECT *
FROM user_tab_columns
WHERE table_name = "A"
"""
cursor.execute(script)
columns = [desc[0] for desc in cursor.description]
data = cursor.fetchall()
df = pd.DataFrame(list(data), columns=columns)
writer = pd.ExcelWriter('C:\Users\PROGRAMs\TEST\export.xlsx')
df.to_excel(writer, sheet_name ='bar')
writer.save()
I would use pandas' built-in .read_sql(). Also, in order to put " in a string in python, you need to use ' as your quotation as opposed to ".
import pyodbc
import pandas as pd
import os
cnxn = pyodbc.connect("Driver={SQL Server Native Client 11.0};"
"Server=DPC;"
"Database=zD;"
"trusted_connection=yes;")
cursor = cnxn.cursor()
script = """
SELECT *
FROM user_tab_columns
WHERE table_name = 'A'
"""
df = pd.read_sql(script, cnxn)
writer = pd.ExcelWriter('C:\Users\PROGRAMs\TEST\export.xlsx')
df.to_excel(writer, sheet_name ='bar')
writer.save()
use forward slashes (/) instead of backslashes
I have more than 5000 records in my tblT1
I execute below queries in access 2010 with linked tables on MS Sql server database for my report.
Pass Through Query:
PTQuery1 = "SELECT tblT1.* From tblT1;"
But I need to execute above Pass Through query based on criteria (tempQuotationNo).
My friend suggested me below query:
A Select Query based on Pass Through Query:
Query1 = "SELECT PTQuery1.* FROM PTQuery1 WHERE PTQuery1.QuotationNo = tempQuotationNO;"
tempQuotationNO can be a variable or a field in a form.
As I know both queries load all records of tblT1 to client (second query filter result of PTQuery1 after loading).
Both work fine but I need to reduce the time of loading records to my client.
Is there anybody to suggest a solution for me?
The aim is (first) to filter the records on SQL Server then load the result to FE, based on a field in a form or user defined variable to reduce the time of loading data from SQL Server to client.
Simply modify the PT query like this:
Dim strSQL As String
strSQL = "select * from table where QuotationNo = " & lngQuoteNum
CurrentDb.QueryDefs("MyPass").SQL = strSQL
At this point you can launch your form etc. it will use this NEW sql for the pass though. The above assumes longQuoteNum is a VBA var. If quote number is text, then you need this:
strSQL = "select * from table where QuotationNo = '" & lngQuoteNum & "'"