I have a database with two file groups, the second file group (Default) being used for the database tables.
For some reason, when I check the file group sizes, the Primary file group is still being used to store all the data while the secondary appears to be as its initial size.
I have double checked that the tables are on the secondary file group by viewing the table properties Filegroup field.
Why is this database still using the primary file group ?
Additional information.
I ran #Vishal Gajjar's referenced script which identified that one table (the largest table) is not entirely using the secondary Filegroup as its still using the Primary for the "Text Filegroup".
The default options specified on a database are only applied to the objects created after the option is changed. Existing objects will keep the options they had when they were created.
So:
either modify the default options before creating any object
or modify the objects one by one (or run an script to do so) if you didn't do the first option
As you've mentioned, you can keep the "limited size" columns in a file group and the "blob columns" (those with MAX length, text, ntext...) on a different file group (the Text filegroup)
Related
I have a database on SQL Server on premises and need to regularly copy the data from 80 different tables to an Azure SQL Database. For each table the columns I need to select from and map are different - example, TableA - I need columns 1,2 and 5. For TableB I need just column 1. The tables are named the same in the source and target, but the column names are different.
I could create multiple Copy data pipelines and select the source and target data sets and map to the target table structures, but that seems like a lot of work for what is ultimately the same process repeated.
I've so far created a meta table, which lists all the tables and the column mapping information. This table holds the following data:
SourceSchema, SourceTableName, SourceColumnName, TargetSchema, TargetTableName, TargetColumnName.
For each table, data is held in this table to map the source tables to the target tables.
I have then created a lookup which selects each table from the mapping table. It then does a for each loop and does another lookup to get the source and target column data for the table in the foreach iteration.
From this information, I'm able to map the Source table and the Sink table in a Copy Data activity created within the foreach loop, but I'm not sure how I can dynamically map the columns, or dynamically select only the columns I require from each source table.
I have the "activity('LookupColumns').output" from the column lookup, but would be grateful if someone could suggest how I can use this to then map the source columns to the target columns for the copy activity. Thanks.
In your case, you can use the expression in the mapping setting.
It needs your provide an expression and it's data should like this:{"type":"TabularTranslator","mappings":[{"source":{"name":"Id"},"sink":{"name":"CustomerID"}},{"source":{"name":"Name"},"sink":{"name":"LastName"}},{"source":{"name":"LastModifiedDate"},"sink":{"name":"ModifiedDate"}}]}
So you need to add a column named as Translator in your meta table, and it's value should be like the above JSON data. Then use this expression to do mapping:#item().Translator
Reference: https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-schema-and-type-mapping#parameterize-mapping
After setting up a master-master replication on top of PostgreSQL , I tried to perform an initial load using:
./symadmin -engine octopusdb reload-node 2
My setup is:
1. I created all sym_x tables in a separate schema (replication).
2. I created all my application tables in other schemas of their own.
3. I inserted into sym_trigger.source_schema_name the suitable schema name for each application table.
Still, the initial load seem to create the application tables under the 'replication' schema instead of in their own schemas.
Is there some parameter I am missing for the properties file, or the initial load command?
So apparently for multi-schmea configuration,you need to create a separate record for each schema in sym_router (with a separate router_id, and appropriate target_schema_name), and for each table put a record in sym_trigger_router and sym_trigger with the appropriate router_id and schema name).
Also, once failed, I needed to remove everything from the tmp directory which is under the symmetric software, so the updates to sym tables will be recognized.
Background
I'm using Azure data factory v2 to load data from on-prem databases (for example SQL Server) to Azure data lake gen2. Since I'm going to load thousands of tables, I've created a dynamic ADF pipeline that loads the data as-is in the source based on parameters for schema, table name, modified date (for identifying increments) and so on. This obviously means I can't specify any type of schema or mapping manually in ADF. This is fine since I want the data lake to hold a persistent copy of the source data in the same structure. The data is loaded into ORC files.
Based on these ORC files I want to create external tables in Snowflake with virtual columns. I have already created normal tables in Snowflake with the same column names and data types as in the source tables, which I'm going to use in a later stage. I want to use the information schema for these tables to dynamically create the DDL statement for the external tables.
The issue
Since column names are always UPPER case in Snowflake, and it's case-sensitive in many ways, Snowflake is unable to parse the ORC file with the dynamically generated DDL statement as the definition of the virtual columns no longer corresponds to the source column name casing. For example it will generate one virtual column as -> ID NUMBER AS(value:ID::NUMBER)
This will return NULL as the column is named "Id" with a lower case D in the source database, and therefore also in the ORC file in the data lake.
This feels like a major drawback with Snowflake. Is there any reasonable way around this issue? The only options I can think of is to:
1. Load the information schema from the source database to Snowflake separately and use that data to build a correct virtual column definition with correct cased column names.
2. Load the records in their entirety into some variant column in Snowflake, converted to UPPER or LOWER.
Both options add a lot of complexity or even messes up the data. Is there any straight forward way to only return the column names from an ORC file? Ultimately I would need to be able to use something like Snowflake's DESCRIBE TABLE on the file in the data lake.
Unless you set the parameter QUOTED_IDENTIFIERS_IGNORE_CASE = TRUE you can declare your column in the casing you want:
CREATE TABLE "MyTable" ("Id" NUMBER);
If your dynamic SQL carefully uses "Id" and not just Id you will be fine.
Found an even better way to achieve this, so I'm answering my own question.
With the below query we can get the path/column names directly from the ORC file(s) in the stage with a hint of the data type from the source. This filters out colums that only contains NULL values. Will most likely create some type of data type ranking table for the final data type determination for the virtual columns we're aiming to define dynamically for the external tables.
SELECT f.path as "ColumnName"
, TYPEOF(f.value) as "DataType"
, COUNT(1) as NbrOfRecords
FROM (
SELECT $1 as "value" FROM #<db>.<schema>.<stg>/<directory>/ (FILE_FORMAT => '<fileformat>')
),
lateral flatten(value, recursive=>true) f
WHERE TYPEOF(f.value) != 'NULL_VALUE'
GROUP BY f.path, TYPEOF(f.value)
ORDER BY 1
I am new to Outsystems and SQL. I am try to create a Bus Application where the entities are
When I try to create a new rider with the same name and different Route and bus Id. I get
Cannot insert duplicate key row in object 'dbo.OSUSR_6SL_RIDER' with unique index 'OSIDX_OSUSR_6SL_RIDER_4NAME'. The duplicate key value is (ABC).
The statement has been terminated.
When I check Name field in the database table 'dbo.OSUSR_6SL_RIDER' it is not having the unique identifier set up. Can anybody please help me with this.
Open the Indexes tree under your table. You will find an Index named 'OSIDX_OSUSR_6SL_RIDER_4NAME'.
Script out that Index and you will see that it is a UNIQUE index on a "name" column that you are trying to create a duplicate value in.
You must either change that Index to include Route and Bus ID, or you must abandon your attempt to create a new row with a duplicate name.
It looks like you are are creating an exact duplicate, i.e. a record with the same Id value. The index name it refers to seems to be auto generated by the db system. Therefor it is not necessarily referring to the Name field. Have a look at your indexes and look at the fields they contain. I wouldn't be surprised if OSIDX_OSUSR_6SL_RIDER_4NAME contains the Id field.
If you are using the OutSystems platform, all the database management is done/generated when you publish from Service Studio, so it isn’t advisable to manipulate the database directly: you’re setting yourself up for a lot of maintenance pain and inconsistencies between different environments.
Double-click on the Entity Rider and it’ll open the edit window of your entity. In the Indexes tab you can define and change your indexes (unique or not) and the tool will (re)generate all the needed SQL commands.
See OutSystems Platform 9 Help | Indexes Tab for more details:
Good morning!
I'm with the need to look in my database, one column of a table that does not know the name at first, what happens is the following:
In my application created for each project, a table is created which takes the name of this project, taking the given name and concatenating with the date and time of creation. So the name of this table is stored in another table called projects that have a field that tells the client that belongs to that project. When I do SELECT want to see the names of application projects related to the ID's of customers, browse the database tables behind those those customers and bring me these tables, so that we can finally see the desired fields.
Do not know if I could be clear, if they need more details just talk!
Thanks!
If I understood you correctly, you need to find the exact names of the tables that were named like your project plus they have some additional characters in their names (that look like dates and times).
Well, you can list all the tables that start with the name of your project, using a query like this:
SELECT *
FROM sys.tables
WHERE name LIKE 'yourprojectname%'
sys.tables is a system view where all your tables are listed.
'yourprojectname%' is a mask used for filtering through the list of tables. The % character is neccessary. It means 'any character or characters, any number of them (or none of them)'. (Without % the output would show you only one table whose name is exactly like your project's name. If such a table exists, that is.)