I finally managed to create a snapshot from a database that contains filestream tables. Now I am stuck selecting records from snapshot.
I have a simple table let's say( tblAttachments: ID, FileName, CreationTime , Value). When I want to select with a simple select query:
select * from tblAttachments
I receive:
Large object data (LOB) for table 'dbo.tblAttachments' resides on an
offline filegroup ('extraBlobs') that cannot be accessed.
Is there anyway to use snapshot with filestream enabled dbs.
If snapshots are not usable with databases with filestream data, why sql server allows to take snapshots from them?
You can use database snapshots with filestream enabled dbs.(although with some limitations)
for the sample you provided you can just ommit the blob column and select records from the table in snapshot:
select ID, FileName, CreationTime from tblAttachments
and you will get records without blob field. In many cases that suffic.
Related
is there a way to load all databases from the source SQL Server to the data lake as it is?
I tried to load each database with his tables but I am asking if there was a way to load all databases as it is to the data lake
One of the ways to load all databases from SQL to ADLG2 is by using Azure Data Factory
Follow the below procedure to load all databases from SQL to ADLG2:
First create one pipeline and take script activity in it add linked service with master database select script as Query And give the following query:
SELECT name, database_id, create_date
FROM sys.databases;
Then take ForEach activity and int its settings give Items as so it will fetch output of script activity
#activity('Script1').output.resultSets[0].rows
In for each activity take one lookup activity, create and add linked service for database with dynamic values
In that dataset add Db name parameter
Noe send this parameter value to linked service properties as below
Lookup activity settings
SELECT table_Schema, TABLE_NAME, TABLE_CATALOG
FROM information_Schema.tables
WHERE TABLE_TYPE = 'BASE TABLE'sql
now take execute pipeline activity click on new in that pipeline create lookupOP parameter with array datatype and in execute pipeline pass the value to it as output of lookup as #activity('Lookup1').output.value
In that new pipeline take ForEach activity and passthe parameter we created as items
In that for each activity take one copy activity and for the source dataset create linked service on SQL database with dynamic values as we created previously
In this dataset create parameters for database name, table name and schema name
now add these dynamic values to linked service properties and Table name, table schema
Copy activity source setting:
create parameters in sink dataset
now add these dynamic values to folder name, file name
Copy activity Sink settings
Output
creating folder of database name and in that folder loading tables of that particular database
I am in the process of building a grocery list app with Azure.
Currently, I have an azure function that writes json files to blob storage. Here is what my current blob storage looks like:
When Azure data factory copies over the data to my SQL DB, this is what it looks like:
Here is the code that I am using to create my table:
-- Create a new table called 'groceryitems' in schema 'grocerylistapp'
-- Drop the table if it already exists
IF OBJECT_ID('grocerylistapp.groceryitems', 'U') IS NOT NULL
DROP TABLE grocerylistapp.groceryitems
GO
CREATE SCHEMA grocerylistapp
GO
-- Create the table in the specified schema
CREATE TABLE grocerylistapp.groceryitems
(
id INT IDENTITY(1,1), -- Primary Key column
epoch DATE,
[name] VARCHAR(50),
PRIMARY KEY CLUSTERED([id] ASC)
-- Specify more columns here
);
GO
1
Oddly enough, when I originally did this with Data Warehouse / Synapse, I didn't get this issue, so I am assuming it is something to do particularly with SQL DB.
I also want to add that it randomly decides to distort one row, if I had 3,4, or 5 items in blob storage, it would pick one row and distort it as shown in the screenshot. It is not always the same item.
Thank you.
Firstly, make sure you have chosen the json format file as Soure dataset.
Here're the steps I tested, and it works well.
Source dataset(json format) from Blob storage:
Import the cchema of the source dataset:
Sink dataset settings:
Mapping settings:
Run the pipeline:
Check the data in table grocerylistapp.groceryitems:
Hope this helps.
I created database using SQL in hive.
And I looked for the database using HDFS.
But I couldn't find database in HDFS.
In hive:
CREATE DATABASE practice
LOCATION '/user/hive/warehouse'/
Checking:
hdfs dfs -ls /user/hive/warehouse
There is nothing in warehouse.
In addition, I created a table in a specific database in hive.
But, using Hue, I could see the table in the default location.
I wanna insert the table into a specific database location.
CREATE TABLE prac (
id INT,
title STRING,
salary INT,
posted TIMESTAMP
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
LOCATION '/user/hive/warehouse/practice.db/prac';
I couldn't find the table prac in the database practice in Hue and HDFS.
How can I see the database in HDFS?
And I also wanna know how to see the table in the specific database location.
Try by specifying db name while creating hive table prac by default hive creates tables in default database.
Example:
hive> CREATE DATABASE practice LOCATION '/user/hive/warehouse/practice.db';
hive> CREATE TABLE `practice.prac` ( id INT, title STRING, salary INT, posted TIMESTAMP ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION '/user/hive/warehouse/practice.db/prac';
Try using below command:
CREATE DATABASE practice
LOCATION '/user/hive/warehouse/practice.db';
hive by default uses '/user/hive/warehouse/' directory to create databases under this location. So while creating database, If you don't provide the location it will pick the database location like this '/user/hive/warehouse/practice.db'.
You can choose any location over hdfs until you have read and write permission over that location.
I have a SQL database (lets use northwind), that has a number of tables (unknown number of tables). I would like to import these tables into a MS access database as DATA (not tables) into a MTT_Table
All standard imports, creates the table as a physical table within ms access and not as data.
I have a table in MS Access that needs to store all the names of tables in other systems - not sure if that makes sense
Is there any way to read an infinite number of tables and populate them as data, using an odbc connection all through VBA
Expected output would be to see the table names as data values, and potentially able to populate the MS access row with metadata about the table
Use information schema to create a view in SQL server:
CREATE VIEW dbo.Sample_View
AS
SELECT TABLE_NAME
FROM [Your_Database].INFORMATION_SCHEMA.TABLES
WHERE TABLE_TYPE = 'BASE TABLE'
Now import this view to access following the steps in this link
Your question is a bit broad (what information do you want from tables), but generally can be achieved by querying the INFORMATION_SCHEMA meta-tables over ODBC.
SELECT * INTO MTT_Table
FROM [ODBC;Driver={SQL Server};Server=my\server;Database=myDb;Trusted_Connection=Yes;].INFORMATION_SCHEMA.TABLES
I'm trying to work out a specific way to copy all data from a particular table (let's call it opportunities) and copy it into a new table, with a timestamp of the date copied into the new table, for the sole purpose of generating historic data into a database hosted in Azure Data Warehousing.
What's the best way to do this? So far I've gone and created a duplicate table in the data warehouse, with an additional column called datecopied
The query I've started using is:
SELECT OppName, Oppvalue
INTO Hst_Opportunities
FROM dbo.opportunities
I am not really sure where to go from here!
SELECT INTO is not supported in Azure SQL Data Warehouse at this time. You should familiarise yourself with the CREATE TABLE AS or CTAS syntax, which is the equivalent in Azure DW.
If you want to fix the copy date, simply assign it to a variable prior to the CTAS, something like this:
DECLARE #copyDate DATETIME2 = CURRENT_TIMESTAMP
CREATE TABLE dbo.Hst_Opportunities
WITH
(
CLUSTERED COLUMNSTORE INDEX,
DISTRIBUTION = ROUND_ROBIN
)
AS
SELECT OppName, Oppvalue, #copyDate AS copyDate
FROM dbo.opportunities;
I should also mention that the use case for Azure DW is million and billions of rows with terabytes of data. It doesn't tend to do well at low volume, so consider if you need this product, a traditional SQL Server 2016 install, or Azure SQL Database.
You can write insert into select query like below which will work with SQL Server 2008 +, Azure SQL datawarehouse
INSERT INTO Hst_Opportunities
SELECT OppName, Oppvalue, DATEDIFF(SECOND,{d '1970-01-01'},current_timestamp)
FROM dbo.opportunities