I need to export two schemas in my Oracle 11.2.0.4 database that totals 500GB in size. My problem is the duration of the export. It was still running 14 hours later before crashing out because of disk space. This is a standard edition database so I don't have the option to invoke the parallel parameter. I'm not an Oracle guru and would really appreciate some tips on how to speed this up. It seems to be taking a huge amount of time with the stats. Can I exclude them and regenerate them after I import. Please find some database parameter values below:
SGA Target: 6G
PGA_aggregate_target: 3G
db_cache_size: 0
stream_pool_size: 0
NOARCHVIVEMODE: enabled
Thanks
Related
I have been trying to do a PITR of a 2GB S0 Azure SQL Server db. It has been running for over 24hrs. The DB restore progress has been saying 50% complete for 18
Hrs without any errors. Should I upgrade the server DTUs and size or the actual service tier?
According to this post. On SQL Database, the "horsepower" is measured by Database Throughput Units, or just "DTUs". This unit is measured by an integer and may variate from 5 to 1750. Every database edition has an offer of one or more "Service Objectives", which are directly related to the number of DTUs and the price to be played.
In the following image, you can find the list of "Service Objectives" (S0, P3, Basic, P11, S3, etc…) per SQL Database Edition and its respective prices. Notice that Microsoft is always updating its offer, so those prices and Service Objectives per Edition may be outdated when you read this post:
One option is a more conservative, responsible and dignified way to choose the number of DTUs, and is based on real data about your database activity. It is the DTU Calculator (http://dtucalculator.azurewebsites.net/), an online service that helps us by advising about the most appropriate Service Objective for a database. You just need to download a PowerShell script, available on the DTU Calculator website, and run it in the server where your database is located. As soon as you run this script, the following data will be measured and recorded in a CSV file:
Processor – % Processor Time
Logical Disk – Disk Reads/sec
Logical Disk – Disk Writes/sec
Database – Log Bytes Flushed/sec
Once the collection is done, you just need to upload the file generated by the script and interpret the results. Here is a sample of one of the charts generated by the DTU Calculator, indicating that 89.83% of the database load would run well with the Service Objective S3, of the "Standard" SQL Database edition.
Here is a decision tree that will help you to reach the optimal point for your database.
So I think you can increase the DTU appropriately to speed up the process. :)
If you are on a S0 you are using Azure SQL Database, not a Managed Instance.
2GB is quite small, it should have recovered the point in time restore in an hour or so.
Contact Microsoft Support.
Need advise here: using Alteryx Designer, I'm pulling a large dataset from SQL Server (10M rows) and need to move into Greenplum DB
I tried both with connecting using Input Data (SQL Server) and Output Data (GP) and also Connect In-DB (SQL Server) and Write Data In-DB (GP)
Any approach is taking a life to complete at the point that i have to cancel the process (to give an idea, over the weekend it ran for 18hours and advanced no further than 1%)
Any good advice or trick to speed up these sort of massive bulk data loading would be very very highly appreciated!
I can control or do modifications on SQL Server and Alteryx to increase performance but not in Greenplum
Thanks in advance.
Regards,
Erick
I'll break down the approaches that you're taking.
You won't be able to use IN-DB tools as the Databases are different, hence you can't push the processing on to the DB...
Using the standard Alteryx Tools, you are bringing the whole table on to your machine and then pushing it out again, there are multiple ways that this could be done depending on where your blockage is.
Looking first at the extract from SQL, 10M rows isn't that much and so you could split the process and write it as a yxdb. If that fails or takes several hours, then you will need to look at the connection to the SQL Server or the resources available on the SQL Server.
Then for the push into Greenplum, there is no PostgreS bulk loader at present and so you can either just try and write the whole table, Or you can write segments of the table into temp tables in Greenplum and then execute a command to combine those tables.
We are pulling millions of rows daily from SQL servers to Greenplum and we use open source tool called Outsourcer. it's great tool and take care of cleansing and other.. We are using this tool for past 3.5 yrs and no issue till now.. It take care of all parallelism and millions of rows are loaded within minutes.
It support incremental or full load. If you need supports Jon Robert owner of the Outsourcers will response to your email within minutes. Here is the link for the tool
https://www.pivotalguru.com/
I have oracle 10g database, while export command is running to take scheduled backup, the database becomes very very very slow. A report which runs normally for 4 minutes (that too is high), takes 30-40 minutes, it is killing us. Firstly, we used to run the exp command on the server where database is installed but later moved it to another server to reduce load on the production server but no change, however it became more slow. We usually have average of 159 sessions on the database server. I think there's some thing wrong with my SGA, PGA memory allocation, here is my initXZ10.ora file.
XZ10.__db_cache_size=512876288
XZ10.__java_pool_size=33554432
XZ10.__large_pool_size=16777216
XZ10.__shared_pool_size=436207616
XZ10.__streams_pool_size=0
*.db_block_size=8192
*.job_queue_processes=10
*.open_cursors=5000
*.open_links=20
*.open_links_per_instance=20
*.pga_aggregate_target=4865416704
*.processes=500
*.sessions=2000
*.sga_max_size=11516192768
*.sga_target=11516192768
*.transactions=500
*.db_cache_size=512876288
*.java_pool_size=33554432
*.large_pool_size=16777216
*.shared_pool_size=436207616
*.streams_pool_size=0
I have 16GB RAM on the server and 64 bit oracle 10.2.0.4.0
Can somebody please help me here to optimize the init.ora file and any parameters to speed up the database.
Thanks everyone.
What is the fastest way to backup/restore Azure SQL database?
The background: We have the database with size ~40 GB and restoring it from the .bacbac file (~4GB of compressed data) in the native way by Azure SQL Database Import/Export Service takes up to 6-8 hours. Creating .bacpac is also very long and takes ~2 hours.
UPD:
UPD.
Creating the database (by the way transactional consistent) copy using CREATE DATABASE [DBBackup] AS COPY OF [DB] takes only 15 minutes with 40 GB database and the restore is simple database rename.
UPD. Dec, 2014. Let me share with you our experience about the fastest way of DB migration schema we ended up with.
First of all, the approach with data-tier application (.bacpac) turned out to be not viable for us after DB became slightly bigger and it also will not work for you if you have at least one non-clustered index with total size > 2 GB until you disable non-clustered indexes before export - it's due to Azure SQL transaction log limit.
We stick to Azure Migration Wizard that for data transfer just runs BCP for each table (parameters of BCP are configurable) and it's ~20% faster than approach with .bacpac.
Here are some pitfalls we encountered with the Migration Wizard:
We run into encoding troubles for non-Unicode strings. Make sure
that BCP import and export runs with same collation. It's -C ... configuration switch, you can find parameters with which BCP calling
in .config file for MW application.
Take into account that MW (at least the version that is actual at the moment of this writing) runs BCP with parameters that will leave the constraints in non-trusted state, so do not forget to check all non-trusted constraints after BCP import.
If your database is 40GB it's long past time to consider having a redundant Database server that's ready to go as soon as the main becomes faulty.
You should have a second server running alongside the main DB server that has no actual routines except to sync with the main server on an hourly/daily basis (depending on how often your data changes, and how long it takes to run this process). You can also consider creating backups from this database server, instead of the main one.
If your main DB server goes down - for whatever reason - you can change the host address in your application to the backup database, and spend the 8 hours debugging your other server, instead of twiddling your thumbs waiting for the Azure Portal to do its thing while your clients complain.
Your database shouldn't be taking 6-8 hours to restore from backup though. If you are including upload/download time in that estimate, then you should consider storing your data in the Azure datacenter, as well as locally.
For more info see this article on Business Continuity on MSDN:
http://msdn.microsoft.com/en-us/library/windowsazure/hh852669.aspx
You'll want to specifically look at the Database Copies section, but the article is worth reading in full if your DB is so large.
Azure now supports Point in time restore / Geo restore and GeoDR features. You can use the combination of these to have quick backup / restore. PiTR and Geo restore comes with no additional cost while you have to pay for
Geo replica
There are multiple ways to do backup, restore and copy jobs on Azure.
Point in time restore.
Azure Service takes full backups, multiple differential backups and t-log backups every 5 minutes.
Geo Restore
same as Point in time restore. Only difference is that it picks up a redundant copy from a different blob storage stored in a different region.
Geo-Replication
Same as SQL Availability Groups. 4 Replicas Async with read capabilities. Select a region to become a hot standby.
More on Microsoft Site here. Blog here.
Azure SQL Database already has these local replicas that Liam is referring to. You can find more details on these three local replicas here http://social.technet.microsoft.com/wiki/contents/articles/1695.inside-windows-azure-sql-database.aspx#High_Availability_with_SQL_Azure
Also, SQL Database recently introduced new service tiers that include new point-in-time-restore. Full details at http://msdn.microsoft.com/en-us/library/azure/hh852669.aspx
Key is to use right data management strategy as well that helps solve your objective. Wrong architecture and approach to put everything on cloud can prove disastrous... here's more to it to read - http://archdipesh.blogspot.com/2014/03/windows-azure-data-strategies-and.html
we run since 2 years a small application on SQL Server 2005 Express Edition the Database has gown from 75 MB up to nearly 400MB within this time, the there isn't a big amount of data.
But the log file has been arrived at 3,7GB now without changing Hardware, table structure or Program code we noted that the Import processes which required 10-15 minutes are now arrived at a couple of hours.
any idea where could be the Problem? Depends it on the log file may be? The 4GB Lock of Express Edition bear only on data files or also on log files?
Additional Informations: There isn't any RAID on the DB Server, There doesn't work concurrent users (only one user is logged in while the import process).
Thanks in Advance
Johannes
That the log file is so large is completely normal behavior; in the two years you have been running; sql has been keeping track of the events that happen in the database as it goes along its business.
Normally you might clear these logs off when you take a backup (as you most likely dont need them anyway.) If you are backing up all you need to change the sql script to checkpoint the logfile (its in books online) depending on how you are backing up your milage may vary.
To clear it down in the immediate make sure no one is in using the database; open management studio express find the database and run
backup log database_name with truncate_only
go
dbcc shrinkdatabase('database_name')
From MSDN:
"The 4 GB database size limit applies only to data files and not to log files. "
SQL Server Express is also limited in that it can only use 1 processor and 1GB of memory. Have you tried monitoring the processor/memory usage while the import is running to see if this is causing a bottleneck?