I am totally new to playfab and I got a question about how the data storage works.
I noticed that playfab allows you to explore the players' data using KQL in the data explorer, and that's ok.
I also noticed that the maximum retention period is 90 days.
I was wondering if I can connect playfab to a database service other than azure to store all the data about my app.
In poor words, I'm trying to find a way to save permanently ALL of my data efficiently, and I was wondering if setting for example 999,999 days in retention would be enough or if I need to do something else, like paying for azure data storaging services
Thanks for the tip and have a good time!
If you are looking for a data archival storage solution which can be integrated with PlayFab, then Azure Storage account is the best option.
You can create Container in Azure Blob Storage account and upload a blob in it (eg. bacpac file) and store it in Archive access tier. Refer image below.
Archive tier - An offline tier optimized for storing data that is rarely accessed, and that has flexible latency requirements, on the
order of hours. Data in the Archive tier should be stored for a
minimum of 180 days.
You can later change the access tier to cool or hot to access the file based on requirement.
Refer: Archive access tier, Archive a blob
Related
I'm completely stymied. Let me describe my situation.
We're a relatively small company and the vast majority of our operational data is contained in a vendor database. Our vendor offers a Data Warehousing service. They've taken all of our data and applied some OLAP-ish modeling to it. Each day, they place either a .bak or a .diff file (.bak once a week, .diff every other day) in a FTP endpoint that we pay to access. Currently, we use a PowerShell script to download this data to a server that we've got sitting at a local server farm, where we then use SQL Server to "rehydrate it" by restoring from it.
That's all fine and good, but we really want to move as many of our workloads into the cloud as possible (we use Azure). As far as I can tell, SQL Managed Instances are the only way we can restore from a .bak file in the cloud. This is waaaay more expensive than we need, and we really don't need the managed instance platform at all except to restore from this file.
Basically, everything about this current process is diametrically opposed to us moving it to the cloud, unless we want to pay even more than we are to rent out this server farm.
I'm trying to lobby them for a different method of getting their data, but I'm having trouble coming up with a method to propose. We need to, every day, transfer a ~40gb database from SQL Server (at our vendor) to Azure SQL (in our cloud). What's the least-intrusive way we could do this?
We are glad that you choose the Azure SQL on Azure VM as the solution. Thanks for the suggestions of Alex and Davaid too:
I've actually seen all of those resources already. The biggest
obstacle here is that the entire process has to be automated
end-to-end, which makes bacpac restores more difficult (they'd have
to write some sort of .NET app to back up to bacpac). I think SQL on Azure VM is the only real option, so I may have
to look at cost for that.
If others face the same scenario, we could reference this. This also can be beneficial to other community members.
I have recently completed two migrations from Hadoop to Snowflake on AWS. I am now moving on to work with another customer where snowflake is on Azure. I wanted to know if there are any feature difference between Snowflake on AWS and snowflake on Azure.
Thanks
not in terms of Snowflake - most customers pick depending on the other systems they use and the deployment (to avoid the extra costs related to data transfer - Snowflake doesn't charge for data load, but AWS and Azure do charge to get the data into the correct deployment if it's not there already)
Metadaten Operations in Azure are slightly slower.
And the maximum filesize ist limited to 256 MB while its 5 GB on AWS and Google.
I'm using a Cloud SQL instance to store two types of data: typical transactional data and large "read-only" data. Each of these read-only tables could have GBs of data and they work like snapshots that are refreshed once a day. The old data is totally replaced by the most recent data. The "read-only" tables reference data from the "transactional tables", but I don't necessarily need to perform joins between them, so they're kind of "independent".
In this context, I believe using Cloud SQL to store these kind of tables are going to be a problem in terms of billing. Because Cloud SQL is fully managed, I would be paying for maintenance work from Google and I wouldn't need any kind of maintenance for those specific tables.
Maybe there are databases more suitable for storing snapshot/temporary data. I'm considering to move those type of tables to another kind of storage, but it's possible that I would end up making the bill even higher. Or maybe I could continue using Cloud SQL for those tables and just unlog them.
Can anyone help me with this? Is there any kind of storage in GCP that would be great for storing large snapshots that are refreshed once a day? Or is there an workaround to make Cloud SQL not maintain those tables?
This is a tough question because there are a lot of options and a lot of things that could work. The GCP documentation page "Choosing a Storage Option" is very handy in this kind of cases. It has a flowchart to select a storage option based on the kind of data you want to store, a video that explains each storage option and a table with the description, strong points and use cases for each option. I would recommend to start there.
Also, if the issue with Cloud SQL is that is fully managed and pricy, you can set up MySQL on Google Compute Engine and manage it yourself. Is also fairly cheaper for the same machine. For a n1-standard-1, $0.0965 in Cloud SQL and $0.0475 in GCE (keep in mind that other charges may apply on top of the machine price)
I am considering options apart from Azure Import Export.
The data is more than 2Tb. but divided into multiple files. I would have 8-10 files ranging from 30Gb to 500Gb And we can use certificates for any web service that can be utilized for this transfer.
AzCopy is usually recommended for transfer of data upto 100GB or less.
Azure Data Factory, Data management Gateway by creating a pipeline and gateway and transferring data.
Please do suggest if there is any other way to do this transfer.
Based on my experience, Azcopy will provide you the best performance and is recommended even when data size is greater than 20 TB. Read this following KB article for more information.
One thing you should determine is which datacenter provides the lower BLOB storage latency based on your location. Use this URL for that purpose. The lower the values you get on that URL, the better.
I have been asked to move an Azure Table Storage service from the Microsoft Dublin Data Centre to a data centre in the UK (any secure data centre so long as it is UK based). My problem is that Azure table storage is only supported in Microsoft data centres (Dublin and Amsterdam in Europe).
There are about 50 tables plus queues and blobs. The data requirements are for highly scalable storage. I do not want to re-write the storage into SQL Server because this will require schema management and direct management of the indexes for performance. I would prefer a performant nosql database that operates to the standards of Azure table storage.
Does anyone have any experience in this area?
As far as migrating your data, there's no automated way to do it. Check out the AzCopy utility for blobs.
As far as which database to choose, that's really going to be app-dependent. You'll need to think about search, storage, indexing, map/reduce/aggregation, etc. Then there's document, column, graph, key/value, sql - you may choose one or a combination.
Just remember that Table Storage is storage-as-a-service, with triple-replicated storage providing durability, and upwards of 2000 tps per partition, 20K tps per storage account. You'll need to build this out yourself, whatever you choose (maybe Redis?).
Anyway: This is pretty wide-open as far as solving your architecture. But hopefully I gave you something to think about.
One more thing: You should really look into the reasons for moving your data. Many people talk about data sovereignty, but sometimes it turns out that the data location doesn't violate any local data laws (or that some data can actually remain where it is, with only specific data needing to be hosted within a country's boundaries).