I want to use the airflow 2.01 docker-compose file from apaches github.
here is the link docker-compose.yaml and here is the link to the dockerfiledockerfile
I want to use a Dag which should grab data out of my SQL Server database. Actually I get the following error:
no module named pymssql
After I manually installed it, I get an error like no module named pyodbc.
When I want to install this manually I get an gcc error, that it is not possible to install.
Does anyone have any clue about this?
Is there any docker-compose file which is able to handle SQL Server connection for Airflow 2?
Thanks in advance
I'm trying to install SQL Server Integration Services on RHEL 8. I have SQL Server 2019 (version v15) installed and running successfully, but when I try and follow this:
https://learn.microsoft.com/en-us/sql/linux/sql-server-linux-setup-ssis?view=sql-server-ver15#RHEL
The RHEL 8 repo doesn't have the mssql-server-is package in it, and using the RHEL 7 repo throws this error:
Error: Problem: conflicting requests
- nothing provides python needed by mssql-server-is-15.0.2000.5-4.x86_64 (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
I have both Python2 and Python 3 installed, and when I run this:
alternatives --config python
I see that /usr/bin/python2 is selected
Any ideas how I can get SSIS installed for RHEL 8 and SQL Server 2019?
I could install it by downloading the rpm and then installing it using --nodeps option as below:
sudo yum download mssql-server-is
sudo rpm -Uvh --nodeps mssql-server-is*rpm
Worked pretty smooth for me.
I wanted to do a realtime deployment of my model on azure, so I plan to create an image which firsts queries an ID in azure SQL db to get the required features, then predicts using my model and returns the predictions. The error I get from PyODBC library is that drivers are not installed
I tried it on the azure ML jupyter notebook to establish the connection and found that no drivers are being installed in the environment itself. After some research i found that i should create a docker image and deploy it there, but i still met with the same results
driver= '{ODBC Driver 13 for SQL Server}'
cnxn = pyodbc.connect('DRIVER='+driver+';SERVER='+server+';PORT=1433;DATABASE='+database+';UID='+username+';PWD='+ password+';Encrypt=yes'+';TrustServerCertificate=no'+';Connection Timeout=30;')
('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC
Driver 13 for SQL Server' : file not found (0) (SQLDriverConnect)")
i want a result to the query instead i get this message
and/or you could use pymssql==2.1.1, if you add the following docker steps, in the deployment configuration (using either Environments or ContainerImages - preferred is Environments):
from azureml.core import Environment
from azureml.core.environment import CondaDependencies
conda_dep = CondaDependencies()
conda_dep.add_pip_package('pymssql==2.1.1')
myenv = Environment(name="mssqlenv")
myenv.python.conda_dependencies=conda_dep
myenv.docker.enabled = True
myenv.docker.base_dockerfile = 'FROM mcr.microsoft.com/azureml/base:latest\nRUN apt-get update && apt-get -y install freetds-dev freetds-bin vim gcc'
myenv.docker.base_image = None
Or, if you're using the ContainerImage class, you could add these Docker Steps
from azureml.core.image import Image, ContainerImage
image_config = ContainerImage.image_configuration(runtime= "python", execution_script="score.py", conda_file="myenv.yml", docker_file="Dockerfile.steps")
# Assuming this :
# RUN apt-get update && apt-get -y install freetds-dev freetds-bin vim gcc
# is in a file called Dockerfile.steps, it should produce the same result.
See this answer for more details on how I've done it using an Estimator Step and a custom docker container. You could use this Dockerfile to locally create a Docker container for that Estimator step (no need to do that if you're just using an Estimator run outside of a pipeline) :
FROM continuumio/miniconda3:4.4.10
RUN apt-get update && apt-get -y install freetds-dev freetds-bin gcc
RUN pip install Cython
For more details see this posting :using estimator in pipeline with custom docker images. Hope that helps!
Per my experience, I think the comment as #DavidBrowne-Microsoft said is right.
There is a similar SO thread I am getting an error while connecting to an sql DB in Jupyter Notebook answered by me, which I think it will help you to install the latest msodbcsql driver for Linux on Microsoft Azure Notebook or Docker.
Meanwhile, there is a detail about the connection string for Azure SQL Database which you need to carefully note, that you should use {ODBC Driver 17 for SQL Server} instead of {ODBC Driver 13 for SQL Server} if your Azure SQL Database had been created recently (ignore the connection string shown in Azure portal).
you can use AzureML built in solution dataset to connect to your SQL server.
To do so, you can first create an azure_sql_database datastore. reference here
Then create a dataset by passing the datastore you created and the query you want to run.
reference here
sample code
from azureml.core import Dataset, Datastore, Workspace
workspace = Workspace.from_config()
sql_datastore = Datastore.register_azure_sql_database(workspace = workspace,
datastore_name = 'sql_dstore',
server_name = 'your SQL server name',
database_name = 'your SQL database name',
tenant_id = 'your directory ID/tenant ID of the service principal',
client_id = 'the Client ID/Application ID of the service principal',
client_secret = 'the secret of the service principal')
sql_dataset = Dataset.Tabular.from_sql_query((sql_datastore, 'SELECT * FROM my_table'))
You can also do it via UI at ml.azure.com where you can register an azure SQL datastore using your user name and password.
Getting this error while trying to run Package.dtsx from command file(.cmd).
To run SSIS package outside of SQL server Data Tools you must install %File_creation% of Integration services or higher
I've used package parameters & File System Tasks in the package. I used below script in command file.
DTEXEC /FILE "F:\SOME FOLDER\Package.dtsx" /CONFIGFILE "F:\SOME FOLDER\PackagedtsConfig"
Appreciate any help on this.
The main problem is that SSIS packages are deployed using a SSDT version newer that the one installed on the server you are trying to execute the package on it. You have to install the same version of SSDT on the server.
Useful Links
https://social.technet.microsoft.com/Forums/en-US/e5a149bf-a636-44bd-b825-36f9984ae720/to-run-a-ssis-package-outside-of-sql-server-data-tools-you-must-install-move-file-to-archive-of?forum=sqlintegrationservices
https://dba.stackexchange.com/questions/49786/error-to-run-a-ssis-package-outside-of-sql-server-data-tools-you-must-install
https://dba.stackexchange.com/questions/138850/ssis-to-run-the-package-outside-of-sql-server-data-tools-you-must-install-expre
I was forced to upgrade my SSIS packages using VS2015 + SSDT, however I'm unable to get the compiled dtsx files to run now through command line using 130\DTS\Binn\DTExec.exe
Full error:
Source: Process file data flow SSIS.Pipeline
Description: To run a SSIS package outside of SQL Server Data Tools you must install Standard Edition of Integration Services or higher.
I've reinstalled MSSQL2014 w/ integration services and I see SQL Server Integration Services 12 is running.
DTExec.exe gets installed when you pick the "Integration Services" option off the setup CD. Make sure that it's installed in the directory indicated in the command line. You also need to make sure that the versions match between the SSIS package and DTEXec.exe. "..\120\DTS\Binn\DTExec.exe" is used for MSSQL2014.
I received this error from double-clicking a DTSX file which opened the Execute Package Utility from C:\Program Files\Microsoft SQL Server\140. I couldn't get it to work until I:
Saved off the command line from the "Command-line" tab of the utility
Navigated to C:\Program Files\Microsoft SQL Server\130\DTS\Binn
Called dtexec.exe with the command line
I think the Execute Package Utility was somehow looking for SQL Server 17 while I only had SQL 16 installed. DTExec.exe was able to find SQL 16 ok.