I am trying to connect to snowflake from my laptop using the python connector. The problem is that my company has strict rules on connections and transferring data.
I couldn't find anything in their documentation on securing data transfer.
Anyone has done it at a company to help me with a solution that I can offer to our IT team?
Thanks
Related
For staging in Snowflake, we need S3 AWS layer or Azure or Local machine. Instead of this, can we FTP a file from a source team directly to Snowflake internal storage, so that, from there the Snowpipe can the file and load to our Snowflake table.
If yes, please tell how. If no, please confirm that as well. If no, won't that is a big drawback of Snowflake to depend on other platforms every time.
You can use just about any driver from Snowflake to move files to Internal stage on Snowflake. ODBC, JDBC, Python, SnowSQL, etc. FTP isn't a very common protocol in the cloud, though. Snowflake has a lot of customers without any presence on AWS, Azure, or GCP that are using Snowflake without issues in this manner.
We are planning to implement a project in Azure cloud where data storage will be Azure Data lake for now and in future HDP will be implemented and ADLS will be the extended datanode. From ADLS we want to expose data for Dashboard creation using Tableau. Initial plan was to use Hive and Tableau will connect to Data through Hive. But here comes the performance issue as:
There will be multiple users who will have access to Data through Tableau(100+)
We will also have to expose Data to different portal with API calls.
Which means multiple connectivity will be established at the same time which will hit hive . My question is:
Can hive serve the purpose with minimal time?
How can i measure the performance?
I dont want to let my users to sit back after running a query in tableau and wait for a long time to see the dashboard.
Would you please share your experiences in this design issue? Should we use Hive or should We use some other tools which have better performance to work with tableau and HDFS storage. Someone suggested me to use Azure SQL Server and connect Tableau to SQL server. But its again the old fashion and also matter of cost as price is related with the execution of each query.
If you have any better solution experience please share , would be greatly appreciated.
Thanks in advance.
Hive LLAP could work, if you can get it installed.
Otherwise, at my work, we've had good experience with PrestoDB and Tableau on S3 data.
Some teams use Spark SQL, and you can setup a Spark Thrift Server, that should be compatible with the Hive JDBC/ODBC drivers
I'm currently looking at metadata solutions for Microsoft SQL Server 2016. I'm interested in using Schemaspy. I think it looks like a great free way to document metadata, but I'm worried that it may not be secure for use in a company with sensitive data.
Question: Is Schemaspy secure to use? Can it be used as a loophole to hack into servers?
Thank you very much in advance!
SchemaSpy is solution that can be used with sensitive and confidential data.
Application code itself is not reading the database data but database structure and metadata from jdbc driver.
SchemaSpy is run as local program and everything what application is generating happen only on your computer and stay there.
In the code base we are not using any external servers or web services in the process of generating and preparing database documentation.
You can look on generated example by SchemaSpy: http://schemaspy.org/sample/index.html
I am working on my Java application in Eclipse that connects to the database and makes some queries with the data. This database has been created and managed in PHPMyAdmin. The database is stored locally and now I would like to make it accessible not only from my computer, but for anyone who will run my application on his computer. Can anyone tell me what is the procedure? How can I make the database "online"? Should I store the database on some remote server? If so, what is the procedure?
I know this question is not so much straight-forward, but I hope you will give me some idea how to deal with it.
Thanks in advance!
In this situation you should store the database on an external server who are better at managing databases with more security/backups etc.
If you're looking for scalability as well you could get an Amazon AWS Micro Instance in the cloud for free for a year which includes a large amount (20GB) of RDS (their relational database service based on MySQL) for free.
There is a great Stack Overflow question on Using PHPMyAdmin to administer Amazon RDS as well to get you started with something you are familiar with.
You can use MySQLDump to copy your current database from your local machine to RDS if you needed.
(A, completely non-extensive, list of) Mysql Hosting Providers
Webfaction - Good web user interface. Does more than just database hosting. Lots of documentation.
Amazons RDS - Cloud based. Reliable. Offers free tier
Rackspace - Cloud based. Excellent customer service.
Or, host your own on something like these
Linode - Has lots of documentation of getting Mysql installed
Digital Ocean - SSDs for great speed. Again great documentation on getting Mysql installed
I have some queries.
By default Google appengine (cloud based deployment platform) does not support mysql or any database for that matter. So we thought of using Amazon RDS as a option (Since it is in cloud.). After reading the documentations, i understood that amazon exposes the web services and provides API s for basic operations like creating a DB instance etc. But i am not sure whether it provides APIs for CRUD operations.? So that programatically i should configure amazon RDS and perform CRUD operations in it. Please answer.
Can i write a web service which is similar to amazon WS to perform CRUD operations in amazon RDS? is it feasible? please answer my questions ASAP.
Amazon RDS exposes MySQL databases using the standard MySQL protocol. App Engine can only make outgoing connections over HTTP, so it won't be possible to connect directly to RDS from App engine. You certainly could write a web service such as you describe, but you'd need to run it on a separate server (such as an EC2 instance), and you'd need to write your own interface for accessing the database on the client end, separate from the MySQL libraries.
Note that we're planning to introduce support for relational databases in the future with App Engine for Business.
I have never used them but RdbHost was built (so it seems) for this reason. You can make your SQL calls over https. This will be slower though.