I should preface by saying my experience with scripting or programming in OOP languages is limited.
I'm working on a method for programatically creating and executing SSIS packages using PowerShell. Unfortunately, most of the resources available for PowerShell and SSIS are for calling PS from SSIS, not the other way around.
I have, however, found a number of resources for VB/C# for creating SSIS packages.
Example resource here.
I've succeeded in converting most of the code by calling the DTS/SSIS assemblies, but it's failing now on converting the TaskHost object to a mainpipe.
Sample code:
[Void][Reflection.Assembly]::LoadWithPartialName('Microsoft.SqlServer.ManagedDTS')
[Void][Reflection.Assembly]::LoadWithPartialName('Microsoft.Sqlserver.DTSPipelineWrap')
# Create the Package and application, set its generic attributes
$Package = New-Object Microsoft.SqlServer.Dts.Runtime.Package
$Package.CreatorName = $CreatorName
$App = New-Object Microsoft.SqlServer.Dts.Runtime.Application
# Set connection info for our package
$SourceConn = $package.Connections.Add("OLEDB")
$SourceConn.Name = "Source Connection"
$SourceConn.set_ConnectionString("Data Source=$SourceServer;Integrated Security=True")
$TargetConn = $package.Connections.Add("OLEDB")
$TargetConn.Name = "Target Connection"
$TargetConn.set_ConnectionString("Data Source=$TargetServer;Integrated Security=True")
# Build the tasks
# Data Flow Task - actually move the table
[Microsoft.SQLServer.DTS.Runtime.Executable]$XferTask = $Package.Executables.Add("STOCK:PipelineTask")
$XferTaskTH = [Microsoft.SqlServer.Dts.Runtime.TaskHost]$XferTask
$XferTaskTH.Name = "DataFlow"
$XferTaskTH.Description = "Dataflow Task Host"
$DataPipe = [Microsoft.SQLServer.DTS.pipeline.Wrapper.MainPipeClass]($XferTaskTH.InnerObject)
Everything works fine til the last line, when I get the error:
Cannot convert the
"System.__ComObject" value of type
"System.__ComObject#{}" to
type
"Microsoft.SqlServer.Dts.Pipeline.Wrapper.MainPipeClass"
Any assistance or ideas are welcome!
Microsoft.SqlServer.DTSPipelineWrap makes heavy use of COM instances.
This forum post suggested using CreateWRapperOfType method:
http://social.technet.microsoft.com/Forums/en-US/ITCG/thread/0f493a31-fbf0-46ac-a6a5-8a10af8822cf/
You could try this:
$DataPipe = [System.Runtime.InteropServices.Marshal]::CreateWrapperOfType($XferTaskTH.InnerObject, [Microsoft.SQLServer.DTS.pipeline.Wrapper.MainPipeClass])
Doesn't error out and produces an object--I'm not sure of what type.
You could always just compile the working .NET version you referenced above into an exe, and allow it to accept parameters as needed in order to create the SSIS packages. Then, use Powershell to call the executable with the parameters as needed.
Related
I am developing application with vb.net (2015) and MS Access database. I can work fine with existing database. I have now situation where I need to create database programmatically, for billing purpose. It is the situation where each folder will contain database for company/firm selection.
After searching on the internet / StackOverflow I learned about ADOX. Even got the ready code for it. I applied it in my coding.
Adding reference of Microsoft ADO extend 2.8 and 6.0
Created variable Adx as new Adox.catalog
Then finally wrote Adx.create(olejet provider conn string with data source)
In this step I get an error
COM Class not registered
So I tried to register msadox.dll and msadox28.tlb with regsvr32 and regasm but at that time I get another error:
msadox.dll get registered successfully but error gives in msadox28.tlb
Fail to load -file- becuase it is not a valid .net assembly file
Now I am stuck at this point.
My system is Windows 10 64 bit. I tried to target cpu x86, and any cpu but it didn't work. I got many questions and answer here but didn't understand it.
EDIT:
I tried following connection string and it worked, but it creates old 2000-2003 mdb file. i want to use new access file .accdb
String is :
Provider=Microsoft.Jet.OLEDB.4.0;Data Source=D:\VBProj\Testing\test.mdb;Jet OLEDB:Engine Type=5
EDIT : on 20/9/2021 - MON
First of all Thank you very much #Jimi, your suggestion to use ACE.16 and cleaning solution worked. Thanks a lot
I use the following steps to create MS Access database using ADOX in VB.NET:
Project Menu > Add Reference > COM Section > Select Microsoft ADO Ext. 6.0 for DLL and security
Write connection string at program entry point (form load/sub main) -> Provider=Microsoft.ACE.OLEDB.16.0;Data Source=D:\VBProj\Testing\test.accdb, assign it to variable connString
Declare adox catalog globally like Public gAdxCat As New ADOX.Catalog
Use its method gAdxCat.create(connString)
That's all - DONE
Again thanks to #jimi
This is the answer to my question (helped by #jimi)
following are the steps to create msaccess database using ADOX in VB.NET and error occurs mention in original questions.
1-Project Menu > Add Reference > COM Section > Select Microsoft ADO Ext. 6.0 for DLL and security (remove ref of 2.8)
2-write connection string at program entry point (form load/sub main) -> "Provider=Microsoft.ACE.OLEDB.16.0;Data Source=D:\VBProj\Testing\test.accdb" assign it to variable connString
3-declare adox catalog globally like Public gAdxCat As New ADOX.Catalog
4-User its method gAdxCat.create(connString)
5-Thats all DONE
again thanks to #jimi
I've literally no experience in VB script or C#. I've created this SSIS package using some online tutorial which server my purpose but I've to fine-tune it to fit my requirements.
Current Scenario:
I'm trying to run an SSIS package which has a for-each loop container which imports the files with *.txt extension in a directory as the file names are not constant. This for-each loop container is followed by some other SQL tasks.
The package is executed successfully even when there are no files in the directory (May be I did something wrong while creating the container and data flow tasks, file system tasks). This is causing the SQL script at the end of the for-each loop container to execute successfully which is resulting in wrong data.
Requirement:
The package should fail if there is no file in directory. I've to implement a script before for-each loop container but not sure how to do it. Any leads would be appreciated!
I did something like this but not sure how to search wrt extension rather than file name:
Public Sub Main()
'
' Add your code here
'
Dim fileName As String
fileName = "filename.txt"
If System.IO.File.Exists(fileName) Then
Dts.Variables("User::bolFileExists").Value = True
Else
Dts.Variables("User::bolFileExists").Value = False
End If
Dts.TaskResult = ScriptResults.Success
End Sub
You should use System.IO.Directory.GetFiles() function.
If System.IO.Directory.GetFiles(<your path goes here>, "*.txt", SearchOption.AllDirectories).Length = 0 Then
Dts.Variables("User::bolFileExists").Value = False
Else
Dts.Variables("User::bolFileExists").Value = True
End If
Below would be my suggestion, I did the same in one of my requirements using event handling section that the underlying DFT is not run, then the script in event handler page would raise error. The point to be noted is that the DFT runs atleast once if there is any file in directory, raising error if it not runs would be simple rather than writing a complex script
Thanks,
Srinivas
I'm working on integrating SQL Server databases into our in-house version control/deployment utility, which is built with powershell,and uses Github as a repository.
Using the excellent sqlpackage.exe utility, I have been able to add a process whereby a developer can extract their current changes into a dacpac and store it in Github, then do the opposite in reverse when they want to get the latest version. However, because the .dacpac is a binary file, it's not possible to see differences in git. I have mitigated this somewhat by unzipping the dacpac before storing in in source control, so contained xml files are added instead. However, even though these files are text-based, they are still not easy to look through and find differences.
What I would like to do, is convert the dacpac into a folder structure similar to what would be seen in SSMS (with all the database objects such as triggers, sprocs etc in their respective folders), store that in Github, and then convert it back into a dacpac when a client checks out the code. However, there doesn't seem to be any function in sqlpackage.exe for this, and I can't find any documentation. Is there any command line tool I can use to this through Powershell?
Using the public APIs for DacFx you can load the dacpac, iterate over all objects, and script each one out. If you're willing to write your own code you could write each one to its own file based on the object type. The basic process is covered in the model filtering samples in the DacExtensions Github project. Specifically you'll want to do something like the ModelFilterer code that loads a dacpac, queries all objects, scripts them out - see the CreateFilteredModel method. I've put a sample that should mostly work below. Once you have this, you can easily do compare on a per-object basis.
using (TSqlModel model = new TSqlModel(dacpacPath))
{
IEnumerable<TSqlObject> allObjects = model.GetObjects(QueryScopes);
foreach (TSqlObject tsqlObject allObjects)
{
string script;
if (tsqlObject.TryGetScript(out script))
{
// Some objects such as the DatabaseOptions can't be scripted out.
// Write to disk by object type
string objectTypeName = tsqlObject.ObjectType.Name;
// pseudo-code as I didn't bother writing.
// basically just create the folder and write a file
this.MkdirIfNotExists(objectTypeName);
this.WriteToFile(objectTypeName, tsqlObject.Name + '.sql', script);
}
}
}
This can be converted into a powershell cmdlet fairly easily. The dacfx libraries are on nuget at https://www.nuget.org/packages/Microsoft.SqlServer.DacFx.x64/ so you should be able to install them in PS and then use the code without too much trouble.
Based on the other post I was able to get a script working. Caveat is you'll have to try the types till you get what you want... The way it is no it trys to put the full http or https value for some of the objects.
param($dacpacPath = 'c:\somepath' Debug', $dacpac = 'your.dacpac')
Add-Type -Path 'C:\Program Files (x86)\Microsoft SQL Server\120\DAC\bin\Microsoft.SqlServer.Dac.dll'
add-type -path 'C:\Program Files (x86)\Microsoft SQL Server\120\DAC\bin\Microsoft.SqlServer.Dac.Extensions.dll'
cd $dacpacPath
$dacPack = [Microsoft.SqlServer.Dac.DacPackage]::Load(((get-item ".\$dacpac").fullname))
$model =[Microsoft.SqlServer.Dac.Model.TSqlModel]::new(((get-item ".\$dacpac").fullname))
$queryScopes = [Microsoft.SqlServer.Dac.Model.DacQueryScopes]::All
$return = [System.Collections.Generic.IEnumerable[string]]
$returnObjects = $model.GetObjects([Microsoft.SqlServer.Dac.Model.DacQueryScopes]::All)
$s = ''
foreach($r in $returnObjects)
{
if ($r.TryGetScript([ref]$s))
{
$objectTypeName = $r.ObjectType.Name;
$d="c:\temp\db\$objectTypeName"
if(!(test-path $d ))
{
new-item $d -ItemType Directory
}
$filename = "$d\$($r.Name.Parts).sql"
if(! (test-path $filename))
{new-item $filename -ItemType File}
$s | out-file $filename -Force
write-output $filename
}
}
I working with Spock and Groovy in order to test my application. I should need to run a ddl script before to run every test.
To execute the script from Groovy I am using the following code:
def scriptToExecute = './src/test/groovy/com/sql/createTable.sql'
def sqlScriptToExecuteString = new File(scriptToExecute).text
sql.execute(sqlScriptToExecuteString)
The createTable.sql is a complex script that do several drop and create operation ( of course it is multiline ). When I try to execute it I got the following exception:
java.sql.SQLSyntaxErrorException: ORA-00911: invalid character
To notice that the ddl is correct since that it has been checked running it on the same DB that I am connecting with groovy.
Any Idea how to resolve the problem?
I think JDBC does not support this, but there are tools/libraries that could help, see this answer for Java.
In Groovy, using this JDBC script runner would be something like:
Connection con = ....
def runner = new ScriptRunner(con, [booleanAutoCommit], [booleanStopOnerror])
def scriptFile = new File("createTable.ddl")
scriptFile.withReader { reader ->
runner.runScript(reader)
}
Or, if your script is "simple enough" (ie no comments, no semicolons other than separating statements...), you can load the text, split it around ; and execute using sql.withBatch, something like that:
def scriptText = new File("createTable.ddl").text
sql.withBatch { stmt ->
scriptText.split(';').each { order ->
stmt.addBatch order.trim()
}
}
If you can't get it done in JDBC (See Hugues' answer), consider executing sqlplus from your Groovy program.
["sqlplus", CREDENTIALS, "#"+scriptToExecute].execute()
I'm trying to run a powershell script through batch job, I used the following code that works fine in a job :
System.Diagnostics.Process process;
System.Diagnostics.ProcessStartInfo startInfo;
;
process = new System.Diagnostics.Process();
startInfo = new System.Diagnostics.ProcessStartInfo();
startInfo.set_FileName("powershell.exe");
startInfo.set_Arguments("D:\\Documents\\OP3_FTP_Upload.ps1");
startInfo.set_UseShellExecute(false);
startInfo.set_RedirectStandardError(true);
process.set_StartInfo(startInfo);
process.Start();
when I use this code in a runbasebatch class, I have the following errors:
Failed to request the permission of type 'InteropPermission'.
Unable to create object 'CLRObject'
So I try to use the following to solve my permission problem:
Set permissionSet;
InteropPermission interopPermission;
;
interopPermission = new InteropPermission(InteropKind::ClrInterop);
permissionSet = new Set(Types::Class);
permissionSet.add(interopPermission);
CodeAccessPermission::assertMultiple(permissionSet);
...my first code example
CodeAccessPermission::revertAssert();
When I execute my batch job , I have no error message but nothing happens. The path is correct, the script also (parms corrects based on AOS)
I think the problem is my way to implement permissionSet and interopPermission classes, I know how to use it in case of CRUD operations on files, but how to use it in case of script execution? Can anyone explain me how (if possible)to manage those classes in my case of use?
Any other ideas to solve my problem are welcome.
This should be enough (in the method doing the CLR calls):
new InteropPermission(InteropKind::ClrInterop).assert();
Otherwise try to debug.