I am trying to connect to Impala DB through Dask Library to fetch all data from a table using the read_sql_table(). Need the connection string to connect to, I have tried using the connection string that I generally connect in Dbeaver but it fails saying that not able to load the plugin: 'impala'.
Can someone help me connect to the Impala DB using Dask library or let me know if Dask supports Impala connection? Thanks.
Dask's read_sql_table uses the pandas version of it to fetch single chunks of data. So if your URI (connection string) works with pandas, it should also work with dask.
I found this tutorial which tells me, that the URI for impala should probably look like this apacheimpala:///?Server=127.0.0.1&Port=21050
I could not verify anything of this, because you have not posted a minimal working example and I don't have any impala DB flying around.
Related
I have a question I was hoping to get some feedback on being that it is my first time doing something like this.
I am building an intranet site and need to display a SQL view from SQL Server. I've done some research online and I've seen some people say use PHP to connect to the DB, then HTML to build a table... but those scenarios have been for MySQL or Access databases, not SQL Server.
My question is: what is the best way to to go about connecting and displaying a SQL Server view to an HTML 5 page? (I'm mainly just asking for some suggestions on where to start, any good documentation to look at, etc. not looking for someone to code it for me or anything like that)
First of all, you do not want to connect to the database directly from client side code, direct database queries should always be done on the server side.
I'd recommend using PHP because it's fairly simple, lightweight and widely supported.
You can query SQL Server with PHP, although you may need to download and install the PHP PDO drivers from Microsoft in order to do this. (Google 'sql server php pdo driver' and you'll be able to find them for your particular platform)
Alternatively you may be able to use an ODBC driver instead, although I haven't tried that personally.
In PHP you can use PDO to open a connection to your database, select from the view into an associative array using something like the following:
$pdo_object = new PDO($dsn, $user, $password);
$statement = $pdo_object->prepare("SELECT column1, column2 FROM view1");
$statement->execute();
$data = $statement->fetchAll(PDO::FETCH_ASSOC);
From here you can iterate over this data to render a table or pass it into javascript and have that populate a data grid.
I am struggling with the above - I've installed PowerQuery (64 bit Excel 2013 setup) and under the database connection options, despite following the instructions here to download the Ngpsql data provider for PostgreSQL:
https://support.office.com/en-ie/article/Connect-to-a-PostgreSQL-database-Power-Query-bf941e52-066f-4911-a41f-2493c39e69e4?ui=en-US&rs=en-IE&ad=IE
I can still only see a limited set of options under under the database list, which do not include a PostgreSQL database. Now having hunted around on the web I found this thread:
https://superuser.com/questions/950100/connect-to-postgresql-database-from-excel-2013-power-query-with-npgsql
Which seems to suggest that the reason I cannot see the Postgresql option is that I am not using an OfficePro installation (think it was home edition).
Does anyone have any pointers - any workaround for this? Or do I really have to get a different version of office to get data from a Postgresql db, short of converting the database into Access or something? Thanks
[I have Office 2013 Pro.]
I had to do this in addition to installing "PowerQuery_2.44.4675.281 (64-bit) [en-us].msi" (I ticked GAC installation on the installation dialog) and "Npgsql-3.2.3.msi". Also, rebooted the machine.
Everything then started to work connection-wise.
But when returning large amounts of data into Excel with Powerquery at times I would get "type cast" errors - I could not map it down to NULLs or anything easy to determine. Powerquery seems a fine tool for some usages and I am sure this error can be fixed with data transformation steps.
If you just want to get the postgres data - you can use VBA + ADO. I have just finished setting it up and it works.
Install "psqlodbc_x64.msi".
Add references to your vba project.
Finally I created the connection with this connecton string (no windows DSN setup required - modify the string below as per your setup):
cnn.Open "Driver={PostgreSQL Unicode(x64)};Server=127.0.0.1;Port=xxxx;UID=postgres;PWD=postgres; Database=db_name;"
I could also successfully add it as a data source using the same connection string.
I'd like to use SQLite3 with Agile Toolkit. I found it in the documentation that the SQLite driver is included with atk4, but I couldn't find any examples of how to connect to a SQLite DB.
I tried to change the DB connection of the example applications shipped with atk4 from MySql to SQLite using the format found in the Pear:DB DSN documentation:
$config['dsn']='sqlite:////var/www/webdev/atk4_example.db';
but I keep getting the could not find driver PDO error:
Application Error: Database Connection Failed
BaseException, code: 0
Additional information:
* PDO error: could not find driver
* DSN: :host=;dbname=;charset=utf8
/var/www/webdev/atk4/lib/DB.php:94
The SQLite PDO driver seems to be installed correctly on the server. Both the phpinfo() and an PHP/PDO/SQLite3 example page confirms that it works.
I'm learning Agile Toolkit, PHP and general web development, so I might be missing something trivial.
Can you try following syntax in your config.php?
$config['dsn']='sqlite://username:password#hostname/dbname';
Also please check if you have MySQL (yes, MySQL not SQLite) extension enabled in your php.ini. If not, then try to enable it and test again. If SQLite works with enabled MySQL extension then I guess I have found small bug in DB.php and could fix that.
Also it looks that you can pass DSN parameters in following format as array and maybe that's even better:
$config['dsn'] = array(
'sqlite:host=hostname;dbname=dbname;charset=utf8',
'username',
'password'
);
Please try 2 solutions described above and let me know if they work.
I can't test that myself now because I'm not at my developing computer.
Is it possible to get the data scraped from websites using Scrapy, and saving that data in an Microsoft SQL Server Database?
If Yes, are there any examples of this being done? Is it mainly a Python issue? i.e. if I find some code of Python saving to an SQL Server database, then Scrapy can do same?
Yes, but you'd have to write the code to do it yourself since scrapy does not provide an item pipeline that writes to a database.
Have a read of the Item Pipeline page from the scrapy documentation which describes the process in more detail (here's a JSONWriterPipeline as an example). Basically, find some code that writes to a SQL Server database (using something like PyODBC) and you should be able to adapt that to create a custom item pipeline that outputs items directly to a SQL Server database.
Super late and completely self promotion here, but I think this could help someone. I just wrote a little scrapy extension to save scraped items to a database. scrapy-sqlitem
It is super easy to use.
pip install scrapy_sqlitem
Define Scrapy Items using SqlAlchemy Tables
from scrapy_sqlitem import SqlItem
class MyItem(SqlItem):
sqlmodel = Table('mytable', metadata
Column('id', Integer, primary_key=True),
Column('name', String, nullable=False))
Add the following pipeline
from sqlalchemy import create_engine
class CommitSqlPipeline(object):
def __init__(self):
self.engine = create_engine("sqlite:///")
def process_item(self, item, spider):
item.commit_item(engine=self.engine)
Don't forget to add the pipeline to settings file and create the database tables if they do not exist.
http://doc.scrapy.org/en/1.0/topics/item-pipeline.html#activating-an-item-pipeline-component
http://docs.sqlalchemy.org/en/rel_1_1/core/tutorial.html#define-and-create-tables
I know that connection string questions are a dime-a-dozen, but I've got a new one.
I created a System DSN to talk to an Oracle database that I have locally on my machine. I put in all the info and hit the test button, and it says that it's successful. I'm using the OraDb11g_home1 driver.
When I try to put together a connection string for an application that uses ODBC, of course I can't get it to work.
One of the connection string attributes that they say that they require in their documentation is something called "Provider." What is this?
One of the most recent strings that I've used includes the following.
Driver={Oracle in OraDb11g_home1};Server=\\localhost:1521\local;Uid=mike;Pwd=password
Can anyone please offer any suggestions? Thanks,
mj
I figured it out. I was trying to use an application that was using 32-bit ODBC and the DSNs that I created were 64-bit.