I have hundreds of .dbf files separated into different folders, basically organized as a bunch of different databases with the same structure. Some extend into the millions of rows. I need that data to end up in a SQL Server database (or multiple databases?), so my team can work with it more easily.
I can't do it manually via Access or the Import/Export Wizard as most tutorials suggest, because there are too many files. I have Access 2016, SSMS 17, Visual Studio 2017, and Windows 10 64 bit. I have been able to open individual tables in Access through the dBase V setting.
I don't know where to start, because I'm pretty new to everything. For example, should I write a console app, configure some SSMS setting, or do something else I'm unaware of?
Could you outline a high level step-by-step process I should use, and maybe point me to some resources? I've looked at a bunch of docs and forums through Google, but none quite seem to make sense to me. The most promising is this post, but I don't have the provider listed, like several others in the comments.
You may consider Integration Services for this kind of import job
https://learn.microsoft.com/en-us/sql/integration-services/connection-manager/integration-services-ssis-connections
And here is how to Connect to a dBASE or Other DBF File
https://learn.microsoft.com/en-us/sql/integration-services/connection-manager/connect-to-a-dbase-or-other-dbf-file
If this is one time operation you can convert dbf file to sql script and import data into sql server . There are some applications on windows store capable of doing so.
https://www.microsoft.com/en-us/store/search/apps?q=dbf+to+sql
Related
I have foxpro data table files ".dbf",".fpt",".cdx" .
I want to read data in the files and import it to SQL server 2008
I have windows 8 OS installed on my machine. I could not find compatible visual foxpro version for windows 8.
Also I have used dbfviewer tool to view data in .dbf file but I cannot check data in .fpt file
Any help is appreciated.
I don't know why people want to close an legit question... My guess they have no clue.
I am running windows 8.1 and use VFP 9 regularly. It may not claim to be directly compatible because M$ has discontinued support on the product, so why even test on platforms they have no concern with. Have you tried VFP9?
Another approach some people use is downloading the VFP OleDb Provider and use it via .net, or even through SQL-Server to use a connection, query the raw data into SQL Server, then run through conversion techniques while everything IS in SQL server.
VFP 9 also has from its top-level menu
Tools --> Wizards --> Upsizing
and guides you through how to pick either a directory of free tables, or pick a database and identify which server to connect to, etc, etc.
Hopefully some good starting points for you to run with and check back if you get stuck somewhere else down stream...
You only need to worry about the DBF file.
As noted in Microsoft's online documentation, Visual FoxPro's DBF, FPT, and CDX files are "Table", "Table Memo", and "Compound Index", respectively.
The DBF is the "core" file, containing the description of the fields as well as the information about each record and the actual data for most data types. For FoxPro's memo-style data types, the information for each record is written onto the equivalent FPT file instead of the DBF itself.
The CDX is a record of the FoxPro indexes, and rather that containing data just contains compiled and ordered pointers to which records match which indexed expressions.
You could purhcase a copy of Visual FoxPro 9, as it seems to work on Windows 8 albeit with some quirks. However, if you only want the data and do not have an exist VFP license, you are better off using the free OLE provider DRapp mentioned in his answer, and sending your data to a more modern format.
You can also use DBF Commander Pro to export DBF files to SQL Server.
To carry out this operation, you'll only need DBF and FPT files - they contain data of Memo field(s). CDX files are not needed since VFPOLEDB provider doesn't support indexes at all.
Install the app, open a DBF file, click File -> Export to DBMS. Then provide the connection string (or Build it). For MS SQL Server 2008 it will look as follows:
Provider=SQLOLEDB.1;Persist Security Info=False;User ID=userid;Initial Catalog=databasename;Data Source=servername
Click Connect button in order to check the connection, then choose the destination table from the dropdown list:
At the bottom of the window you'll see the command-line that can be used for uploading DBFs to a database in a batch mode. Click Export button to start the process.
DBF Commander Pro has 20-days full featured trial period, so you can do the job for free and you'll economize $300 on Visual FoxPro license :)
In Windows 7, I can create an ODBC connection to a Foxpro data source. I don't know if that capability still exists in Windows 8, but I don't see why it wouldn't.
Once that connection exists, the data becomes available for any other tool that can utilize ODBC. At that point, you could import the data into Excel, massage it, then load it into SQL Server. Alternatively, you could also access the data directly via VBScript to either dump it to a text file or import it directly into SQL Server.
I'm looking for the best approach (or a couple of good ones to choose from) for extracting from a Progress database (v10.2b). The eventual target will be SQL Server (v2008). I say "eventual target", because I don't necessarily have to connect directly to Progress from within SQL Server, i.e. I'm not averse to extracting from Progress to a text file, and then importing that into SQL Server.
My research on approaches came up with scenarios that don't match mine;
Migrating an entire Progress DB to SQL Server
Exporting entire tables from Progress to SQL Server
Using Progress-specific tools, something to which I do not have access
I am able to connect to Progress using ODBC, and have written some queries from within Visual Studio (v2010). I've also done a bit of custom programming against the Progress database, building a simple web interface to prove out a few things.
So, my requirement is to use ODBC, and build a routine that runs a specific query on a daily basis daily. The results of this query will then be imported into a SQL Server database. Thanks in advance for your help.
Update
After some additional research, I did find that a Linked Server is what I'm looking for. Some notes for others working with SQL Server Express;
If it's SQL Server Express that you are working with, you may not see a program on your desktop or in the Start Menu for DTS. I found DTSWizard.exe nested in my SQL Server Program Files (for me, C:\Program Files (x86)\Microsoft SQL Server\100\DTS\Binn), and was able to simply create a shortcut.
Also, because I'm using the SQL Express version of SQL Server, I wasn't able to save the Package I'd created. So, after creating the Package and running it once, I simply re-ran the package, and saved off my SQL for use in teh future.
Bit of a late answer, but in case anyone else was looking to do this...
You can use linked server, but you will find that the performance won't be as good as directly connecting via the ODBC drivers, also the translation of the data types may mean that you cannot access some tables. The linked server might be handy though for exploring the data.
If you use SSIS with the ODBC drivers (you will have to use ADO.NET data sources) then this will perform the most efficiently, and as well you should get more accurate data types (remember that the data types within progress can change dynamically).
If you have to extract a lot of tables, I would look at BIML to help you achieve this. BIML (Business Intelligence Markup Language) can help you create dynamically many SSIS packages on the fly which can be called from a master package. This master package can then be scheduled or run ad-hoc and so can any of the child packages as needed.
Can you connect to the Progress DB using OLE? If so, you could use SQL Server Linked Server to bypass the need for extracting to a file which would then be loaded into SQL Server. Alternately, you could extract to Excel and then import from Excel to SQL Server.
I'm looking for a way that I can keep a database in one single file, no server hosting it, and with the ability to use ADO (In delphi, specifically TADOConnection and/or TADOQuery). Please pardon my lack of terminology on this one. I'm only familiar with SQL Server databases, and nothing about any others. In fact, the only other ways I know to read/write files are Plain Text, INI, and XML. As for any official "databases", I know nothing.
So what I would like to do is keep a single file as a database, similar to how QuickBooks has a single "Company File". I should not have anything to host the data, such as SQL Server. And it needs to be compatible with ADO, so I can use simple select, update, delete, etc. It doesn't need to be so complex as to have relations, security, etc. But it does need to have some same syntax rules as SQL Server, like commands such as join, alter, distinct, etc.
I'm looking for the lightest-weight method to do so. The files need to be flexible enough to be able to copy/paste (so long as the application isn't using it), similar to an excel file. In fact, my original idea was to use Excel, as I know I can use ADO, but I also don't want to require Microsoft's excel drivers (it would have to presume that MS Office / Excel is installed on user's computer). It's obviously going to need some drivers, but I need the most standard method which is compatible everywhere.
You can use MS-Access MDB files. It can be used via Microsoft OLEDB Jet 4 engine (Which is build in into Windows since at least Win XP) and is perfect for local desktop DB applications, with the ability to create Tables, PKs, Indexes, Queries/Views, Transactions, Multi-User, replication, compact/repair and much more with almost perfect compatibility to MS SQL-Server SQL syntax (since MS-Access is the ancestor of MS SQL-Server).
MS-Access product (i.e MS Office) dose not have to be installed on the client machine.
No extra drivers or files to install, and completely integrable with existing MS-Office products.
Edit: MDB files could be also Protected/Encrypted.
You have several options for store your data in single database file.
SQLite
Firebird
Interbase
All of them can be accesed via ADO using a ODBC or OLEDB driver. my personal recomendation is Firebird, because is free, fast, stable and had a Embedded version.
This is a pretty useful comparison of a number of embedded databases. Of the ones tested these ones support (odbc), (oledb) or (both) and use a (single) file for the database:
Accuracer (odbc) (single)
NexusDB (odbc) (single v4 and newer)
Firebird (both) (single) - multiple odbc implementations and the commercial IBProvider supports three different ways to connect to the ADO components.
TurboDB (odbc) (single v4 and newer)
Note: Most of these also supply ADO.Net Providers as well.
The other's in the comparison (Advantage, ElevateDB, DBISAM and Apollo) use a file per table/index scheme.
I have a typical dev scenario: I have a SQL 2008 database that I want to copy every so often to my local instance of 2008 Express so that I can do dev, make changes, etc. to the local copy. I have some constraints though: the source db is part of a live e-commerce site in shared hosting so I can't detach it and the hosting service wants me to pay $5 for each ad hoc back up I invoke.
What I'd like is some tool that I can invoke ad hoc to take a snapshot (complete, not incremental) of the live db that I can then import in my local one. I've tried the SSMS 2008 Copy Database Wizard but it gives me an error saying I can't do that with Express. I tried the Generate Scripts tool and thought that was going to make it - the export to my local disk worked but when I went to import using SQLCMD (the script was 1GB so SSMS errored when I tried to open it there), it told me there was a syntax error a few thousand lines in.
Coming from the MySQL world, this process is trivial. All I want is an analog of mysqldump and then a command-line way to import that file into a db. Surely there's an easy way to do this in the SQL Server world? This seems like the most basic use-case for developers.
[ Yes, I've seen a few other questions here that seem similar but I didn't think they had the same constraints. ]
Best answer: full backup, restore, pay $5. Anything else seems to me like it'd waste a lot more than $5 worth of time.
If they don't charge you to run queries against the database these tools may help. Granted these are not free tools, but are handy on so many fronts it would be worth buying one. These tools can diff your source db and target db both data and structure or just one or the other, and optionally sync the target database to be just like the source.
http://www.innovartis.co.uk/
http://www.red-gate.com/products/sql%5Fdata%5Fcompare/index.htm
Try SQL Dumper.
SQL Server Dumper enables you to dump selected SQL Server database tables into SQL INSERT statements, that are saved as local .sql files and contain all the data required to create a duplicate table, or to be used for backup purposes. You can choose to create an individual .sql file for each table, or combine all selected tables into a single file.
SQL Server Database Publishing Wizard and osql usually do the trick for me with large databases.
I'm in the process of porting some SQL Server 2005 databases to SQL Server 2008. One of these databases has an associated import application (Windows task) which uses SSIS with a DTS package to import a large dataset from an MS Access database nightly.
In upgrading to SQL Server 2008, I discovered that I can't run the same console application which has been performing the imports due to the missing manageddts DLL in SQL Server 2008. It's several years old and in need of a rewrite for various reason, plus, I've been fairly unhappy with DTS in general. The original reason DTS was chosen was for speed (5 min import time compared to 30+ for ADO.NET).
The format of the data to import is out of my control (the client likes Access). I would also like to be able to run the import from a machine completely separate from the server hosting SQL Server and preferably with minimal SQL features installed.
Options I've considered:
Creating an Access application to connect to both databases (SQL Server and Access) and perform the import (Ugh!)
Revisiting ADO.NET to see if the original implementation was poorly written.
Updated SSIS packages.
What other technologies should I be considering for this job?
I received a suggestion that I use ADO.NET to convert the .mdb file to a bunch of flat files, then use tsql's bulk insert statement to import the flat files. If it works, I'll mark this as the answer.
EDIT:
This worked really well. Far speedier than the old DTS. One down side is that the import user must have bulkadmin rights, and the schema of the flat files must precisely match the tables, but a little coding solved the latter, and a secure, dedicated user account answers the former.
I'm fairly happy with the result.
I guess you should try SSIS. If you don't know enough of it, you will learn a new thing.