I have a simple 4 column text file with billions of rows (say Huge_file.txt ). Now the Huge_File.txt after compressing is around 1 TB in zipped format with around 11 % compression rate ( after unzipping the size will balloon upto 10 TB or so.)
Is there a way to load Huge_File.txt without unzipping?(Space issues on server.) Like something similar to how Redshift imports a zipped file from S3 without unzipping it.
P.S. I am using SQL Server
Version : Microsoft SQL Server 2016 (SP1) (KB3182545) - 13.0.4001.0 (X64)
Oct 28 2016 18:17:30
Copyright (c) Microsoft Corporation
Enterprise Edition: Core-based Licensing (64-bit) on Windows Server 2016 Standard 6.3 (Build 14393: )
Any help is appreciated. Thank you.
Related
I recently started using sys.dm_os_enumerate_filesystem.
My SQL Server version is:
(Microsoft SQL Server 2017 (RTM-CU31) (KB5016884) - 14.0.3456.2 (X64) Sep 2 2022 11:01:50 Copyright (C) 2017 Microsoft Corporation Developer Edition (64-bit) on Windows Server 2012 Standard 6.2 (Build 9200: ) (Hypervisor) )
Everything worked fine until I started querying filesystems where the directories are very large.
The following query generated the error:
SELECT *
from sys.dm_os_enumerate_filesystem('F:\technician','*');
Msg 407, Level 16, State 1, Line 15
internal error. The string routine in file sql\ntdbms\storeng\dfs\alloc\storagedmv.cpp, line 799 failed with HRESULT 0x8007007a.
The directory is 460 GB. I have also checked all permission and SQL Server service has all permission on the above directory.
As you can see the SqlServer is running with the last CU31.
Is there a workaround?
In my team, a dev recently updated our project to use EF6.
All my colleagues can run the projet whitout problems, but in my machine, I'm getting the error:
"Spatial types and functions are not available for this provider because the assembly 'Microsoft.SqlServer.Types' version 10 or higher could not be found."
This package is not even being used in the project, no one of my team even has the dll in the project.
I cant seem to find out what is wrong with this.
Can anyone help me?
Using Visual Studio 19.6.3 and SQL Server Microsoft SQL Server 2016 (RTM) - 13.0.1601.5 (X64) Apr 29 2016 23:23:58 Copyright (c) Microsoft Corporation Enterprise Edition: Core-based Licensing (64-bit) on Windows Server 2012 Standard 6.2 (Build 9200: ) (Hypervisor)
To solve the problem, you need SQL Server installed in your machine.
If you have windows 10 in your system, check if you have SQL Server installed in Apps and features.
You can also check if you have any version of Microsoft.SqlServer.Types in your GAC_MSIL (Global Assembly Cache) directory on C:\Windows\assembly\GAC_MSIL\Microsoft.SqlServer.Types, if this folder is empty or does not exist, you will need to install Microsoft.SqlServer.Types in your machine.
I have 2 similar SQL Server installations on 2 similar GCP projects.
Everything is the same - configuration, CPU, RAM, disk drive layout, similar (but not the same) database with similar data and workload.
When I run alter index index [IndexName] on dbo.Tablename rebuild with (online=on) on the 1st server it takes about 30 minutes to rebuild the whole index. On the 2nd server it ran for more than 3.5 hours before I stopped the rebuild.
All disk metrics (throughput, queue length, etc.) look reasonable. The rebuild is being performed during a night maintenance window with no significant transaction load on the server.
My question is: How can I "debug" the rebuild process to see what is going on, why two similar databases on two similar servers act very different? Is there any trace flag, extended event, etc. which may help to investigate the problem?
Microsoft SQL Server 2016 (SP2) (KB4052908) - 13.0.5026.0 (X64)
Mar 18 2018 09:11:49
Copyright (c) Microsoft Corporation
Enterprise Edition: Core-based Licensing (64-bit) on Windows Server 2016 Datacenter 10.0 (Build 14393: ) (Hypervisor)
26CPU/20GB of RAM
Index size before rebuilding: ~110Gb, after rebuilding: ~30Gb
We use
Microsoft SQL Server 2008 R2 (RTM) - 10.50.1600.1 (X64) Apr 2 2010 15:48:46 Copyright (c) Microsoft Corporation Enterprise Edition (64-bit) on Windows NT 6.2 (Build 9200: ) (Hypervisor)
The machine has 70GB Memory
The SQL Server has
minimum Server memory: 20480 MB
maximum Server memory: 51200 MB
But when I open the Resourcemonitor and check the memory of sqlserver.exe-process, I see that the Commited Memory (Zugesichert) is about 51GB, but the WorkingSet (Arbeitssatz) and Privat are only about 1GB.
The SQL Server is under full load and it is running for 3 month without restart.
The Page Lifetime Expectancy is 14'146s (=about 4h)
For testing purpose, I have selected a table with 3.5Mio Rows (Storage Size: 4'600MB) - but the values for WorkingSet and Private in the Resourcemonitor do not change.
Now my questions:
Is there something wrong or are the values of Resourcemonitor not correct?
If yes, where can I get the real memory usages?
If no, where can I take in to solve the problem or get more information?
When ever I try to proceed to the second step in the comparison, it displays a message saying that it cannot connect with the login I provided (sqlAuth not win).
Note A: It will let me run queries and even do a schema comparison, but no luck with data comparison.
Note B: localhost works fine.
Versions:
Remote:
Microsoft SQL Server 2012 - 11.0.2100.60 (X64) Feb 10 2012 19:39:15
Copyright (c) Microsoft Corporation Enterprise Edition (64-bit) on
Windows NT 6.1 (Build 7601: Service Pack 1) (Hypervisor)
Local:
Microsoft SQL Server 2014 (RTM-CU14) (KB3158271) - 12.0.2569.0 (X64)
May 27 2016 15:06:08 Copyright (c) Microsoft Corporation Express
Edition (64-bit) on Windows NT 6.3 (Build 10586: )
There have been a number of connection-related fixes in the latest SSDT 16.5 release. I just worked through a similar issue with a partner building SSDT demos, and updating to latest resolved issues in Data and Schema compare for SQL Authentication with them. Specifically, there were issues where the "Remember Password" checkbox wasn't preserved which was causing issues, so you may need a one-time action to enter the password so that it's re-saved correctly.
Disclosure: I work on the SQL Server tools team.
I'm having similar issues with SQL Server auth.
Possible answers are listed on this similar question: https://dba.stackexchange.com/questions/102605/azure-sql-database-login-failed-for-user-in-application-but-works-fine-in-ssm
Which links to https://social.msdn.microsoft.com/Forums/sqlserver/en-US/74d2b7f0-09f2-4ef3-935b-e0a1e0e8fb82/schema-compare-target-unavailable?forum=ssdt