If I have SQL tables tblA and tblB and they have one to many relationship between them.. can we create multiple raven db documents automatically .. Can I automatically load documents in Raven DB... from SQL server tables any other way... using any tool?
There is a ETL demo in Raven's code base.
Additionally, you may have a look at how Raccoon imports a SQL database for Subtext into RavenDB.
Related
Here's the basic idea of what I want to do in SSIS:
I have a large query against a production Oracle database, and I need the following where clause that brings in a long list of ids from SQL Server. From there, the results are sent elsewhere.
select ...
from Oracle_table(s) --multi-join
where id in ([select distinct id from SQL_SERVER_table])
Alternatively, I could write the query this way:
select ...
from Oracle_table(s) --multi-join
...
join SQL_SERVER_table sst on sst.ID = Oracle_table.ID
Here are my limitations:
The Oracle query is large and cannot be run without the where id in (... clause
This means I cannot run the Oracle query, then join it against the ids in another step. I tried this, and the DBA's killed the temp table after it became 3 TB in size.
I have 160k id's
This means it is not practical to iterate through the id's one by one. In the past, I have run against ~1000 IDs, using a comma-separated list. It runs relatively fast - a few minutes.
The main query is in Oracle, but the ids are in SQL Server
I do not have the ability to write to Oracle
I've found many questions like this.
None of the answers I have found have a solution to my limitations.
Similar question:
Query a database based on result of query from another database
To prevent loading all rows from the Oracle table. The only way is to apply the filter in the Oracle database engine. I don't think this can be achieved using SSIS since you have more than 160000 ids in the SQL Server table, which cannot be efficiently loaded and passed to the Oracle SQL command:
Using Lookups and Merge Join will require loading all data from the Oracle database
Retrieving data from SQL Server, building a comma-separated string, and passing it to the Oracle SQL command cannot be done with too many IDs (160K).
The same issue using a Script Task.
Creating a Linked Server in SQL Server and Joining both tables will load all data from the Oracle database.
To solve your problem, you should search for a way to create a link to the SQL Server database from the Oracle engine.
Oracle Heterogenous Services
I don't have much experience in Oracle databases. Still, after a small research, I found something in Oracle equivalent to "Linked Servers" in SQL Server called "heterogeneous connectivity".
The query syntax should look like this:
select *
from Oracle_table
where id in (select distinct id from SQL_SERVER_table#sqlserverdsn)
You can refer to the following step-by-step guides to read more on how to connect to SQL Server tables from Oracle:
What is Oracle equivalent for Linked Server and can you join with SQL Server?
Making a Connection from Oracle to SQL Server - 1
Making a Connection from Oracle to SQL Server - 2
Heterogeneous Database connections - Oracle to SQL Server
Importing Data from SQL Server to a staging table in Oracle
Another approach is to use a Data Flow Task that imports IDs from SQL Server to a staging table in Oracle. Then use the staging table in your Oracle query. It would be better to create an index on the staging table. (If you do not have permission to write to the Oracle database, try to get permission to a separate staging database.)
Example of exporting data from SQL Server to Oracle:
Export SQL Server Data to Oracle using SSIS
Minimizing the data load from the Oracle table
If none of the solutions above solves your issue. You can try minimizing the data loaded from the Oracle database as much as possible.
As an example, you can try to get the Minimum and Maximum IDs from the SQL Server table, store both values within two variables. Then, you can use both variables in the SQL Command that loads the data from the Oracle table, like the following:
SELECT * FROM Oracle_Table WHERE ID > #MinID and ID < #MaxID
This will remove a bunch of useless data in your operation. In case your ID column is a string, you can use other measures to filter data, such as the string length, the first character.
We have a legacy database that has dozens of schemas in it, and we're looking to split that database up into several smaller distinct databases instead.
Is there any way I can create a new database on the same physical server, and then transfer an entire schema over to the new database?
Our tables look like:
Foo.Table1
Foo.Table2
Foo.Table3
...
Bar.Table1
Bar.Table2
...
Xxx.Table1
Xxx.Table2
...
...and I want to move Foo.* to a new database.
Typically this is recommended to be done by some kind of per-table export/import, but that's quite cumbersome with the 150+ tables in the schema.
As far as my trivial research goes the options appear to be:
Export/import each table individually.
Backup the entire database, restore in a different destination and delete everything else (painful, since the entire database is ~900GB).
Deploy the dacpac of the single schema to the new database, and do a cross database initial seeding, aka:
INSERT INTO newDb.Foo.Table1 SELECT * FROM oldDb.Foo.Table1;
INSERT INTO newDb.Foo.Table2 SELECT * FROM oldDb.Foo.Table2;
INSERT INTO newDb.Foo.Table3 SELECT * FROM oldDb.Foo.Table3;
...
All of these options are a lot of effort... is there any other approach that will simply move an entire schema into a new database?
I am not aware of any fully automated way but this can be done relatively simply with the help of Excel.
In SSMS you can use "Object Explorer Details" to easily (with few mouse clicks) script schema of multiple tables.
With the help of system views (sys.tables, sys.columns etc.) and Excel you should be able to generate 'INSERT INTO .... SELECT ...' scripts for all of your tables in minutes.
In Excel (or a similar application) you paste the list of your tables (obtained using sys.tables) and then write a formula to generate a script for each table.
you can create a filegroup for each schema and move the tables of a schema into the related filegroup. after that you backup each filegroup and restore.
I am new to SQL Server. On the previous job I used working with Postgres and MySQL. But now I was faced with the task connected with SQL Server. And I discovered very strange thing in the DB with that I should work. There don't exist any relations!
Is it normal to SQL Server? How can I automatically connect tables according to their primary keys? Any other ideas?
An screenshot of the ER diagram:
Unless you are talking about creating hundreds of FKs you may be better off by just adding those relationships manually either through SSMS database diagram or through a sql script.
If the number of possible relations is really large or you expect to have to do this again in the future, you may want to look to SMO (SQL Management Objects) and either use Powershell or a small C# program to script out the tsql that would join those tables. But you would need to make sure that there is a repeatable pattern / naming convention between the columns and tables that you can leverage. For Example:
Table1
ID
Name
Table2
ID
Table1ID
Name
Here you could consider Table1ID as a FK referencing Table1.ID
When I'm importing data from SQL Server 2008 DB to another using SSMS 2008, I get errors during the importation because it tries to insert data in a "read only" fields, or cuz some conflicts of relationships between tables' keys.
I'm wondering, how could I close the eyes of the SSMS until he finish the transformation :D
Thanks, Regards
Yes, if you reorder your table data insert statements you should be able to resolve foreign key/relationship issues. You could use an ER diagram (e.g. in SSMS Database Diagrams, select all tables) to first insert the data for the tables other tables depend on/point to with foreign keys, and then work your way down the dependencies.
I wonder how you generated these scripts; I'd imagine that any tool worth it's salt would generate data insert scripts in the proper order.
In the SQL Server 2005 books online section "Defining Named Queries in a Data Source View (Analysis Services)", it states:
A named query can also be used to join multiple database tables from one or more data sources into a single data source view table.
Does anyone know where I can find examples or tutorials on how this can be done?
EDIT: To provide some additional background...
I am working with an analysis services project in the SQL Server Business Intelligence Development Studio for SQL Server 2005. I have defined a data source for each of my databases which are on different servers. I am trying to create a named query which will be a union of a table from each data source. The problem is that the named query requires me to choose a single data source for the query. The query is executed against this data source which does not know anything about the data sources in my project. However, according to the SQL Server 2005 books online, what I am trying to accomplish should be possible based on my quote from above.
MSDN has this link describing Named Queries and this link walking you through the process of creating one.
Edit: I think that to use multiple datasources, you would need to fully qualify your table to hit other datasources when creating your query, like this:
SELECT user_id, first_name, 'DB1' as DB FROM users
UNION
SELECT user_id, first_name, 'DB2' as DB FROM Database2Name.dbo.users
to get results like
user_id first_name DB
1 Bob DB1
2 Joe DB1
11 Greg DB2
12 Mark DB2
If by "multiple data sources" you mean multiple databases, then you can do this if you fully qualify the database name.
For example if I have two databases I can do this:
SELECT * FROM DatabaseA.dbo.SomeTable
JOIN DatabaseB.dbo.OtherTable
ON DatabaseA.dbo.SomeTable.Id = DatabaseB.dbo.OtherTable.Id
Make sure that you don't forget the dbo bit (the owner), otherwise it won't work.
The only other sort of "multiple data sources" that I'm aware of is distributed queries which allows you to perform queries over multiple remote instances of sql server:
sp_addlinkedserver 'server\instance'
SELECT * FROM [server\instance].DatabaseA.dbo.SomeTable
JOIN DatabaseB.dbo.OtherTable
ON [server\instance].DatabaseA.dbo.SomeTable.Id = DatabaseB.dbo.OtherTable.Id