Is there any way to access Redis data from relational databases, such as Oracle or SQL Server?
One use case I have in mind is ETL to a data warehouse.
Am trying to understand the question: You have data in a traditional RDBMs, and you want to extract information from here, and load into Redis? Or its the other way around?
Either way, since am not competent to talk about RDBMS, I would expect to create a program (Java in my case), which would extract information from Redis, and upload it to Oracle. There are options to interact with Redis using a Java Client library (JDBC Redis, and JRedis are examples)
You may get a better answer from the community, if you can elaborate on your question.
Well, if you use server side Java object on your ORA (and they can make REST calls, in the very least, if not socket io (don't know)) then you can call Redis from your stored procedures in Oracle.
[edit]
Should add that if you can make socket connections, then just include JRedis jar in your Oracle server's lib so server-side object can create clients.
Should that not be possible -- I would seriously question a DB that lets SProcs and triggers to open generic TCP connections -- then you are left with consuming Web Services.
JRedis doesn't support web services, but nothing is stopping you from wrapping JRedis and exposing whatever command you need as a RESTFul resource. So here, you would run Redis on server R, a java web server (Jetty/Jettison would do fine) running JRedis on server R or R`. Since Redis is single threaded, it is perfectly fine to run it on the same multi-core box as a JVM; its just a matter of resources, and it they are sufficient then you are using the loopback on the connection between Redis and JRedis and that's guaranteed to be faster than traversing network boundaries! But if the loads you require preclude colocation of Redis and JRedis (proxy), then use a second server.
And of course, you are running your DB on server D. So D <=> R` <=> R. You will pay the second hop's latency costs, of course.
Related
My understanding is that using Redis requires you to host it on its own server. So why even use it if the data being stored on it isn't being run on the same VM (thus using the same RAM) as your app server (eg Node)?
You're not required to host Redis on a separate server at all. In fact, it's not uncommon for application servers to run an in-memory store like Redis or Memcached on the same server for simple caching tasks.
However, what I think is at the heart of your question is a fundamental misunderstanding of how in-memory storage works. Even if you were to run Redis on the same server as your application, your application would never be able to directly access the RAM blocks that Redis uses to store your data--you would still need to send a request to the Redis instance to retrieve the data. Hosting Redis separately from your application server does introduce network latency, but there's zero difference in terms of accessing or modifying the data in RAM.
The name "Redis" is an acronym for REmote DIctionary Server - the "Remote" part indicates that it is intended to be used over a network. The main concept here is that the data stored in memory in Redis is accessible to multiple application instances, instead of having an in-app store for each.
That said, there is no requirement to have Redis on a separate server or use it with multiple application instance. On the other hand, it makes a lot of sense because that is what it was designed for.
I have a SQLite database on my local machine and my web services running on the same machine access it using SQLAlchemy like this:
engine = create_engine('sqlite:///{}'.format('mydatabase.db'), echo=True)
We are planning to host our web services on a separate machine from where the database is hosted. How can we make this 'mydabata.db' be accessible for our web services remotely for my web services? Thanks.
From SQLite when to use docs:
Situations Where A Client/Server RDBMS May Work Better
Client/Server Applications
If there are many client programs sending SQL to the same database over a network, then use a client/server database engine instead of SQLite. SQLite will work over a network filesystem, but because of the latency associated with most network filesystems, performance will not be great. Also, file locking logic is buggy in many network filesystem implementations (on both Unix and Windows). If file locking does not work correctly, two or more clients might try to modify the same part of the same database at the same time, resulting in corruption. Because this problem results from bugs in the underlying filesystem implementation, there is nothing SQLite can do to prevent it.
A good rule of thumb is to avoid using SQLite in situations where the same database will be accessed directly (without an intervening application server) and simultaneously from many computers over a network.
SQLite works well for embedded system or at least when you use it on the same computer. IMHO you'll have to migrate to one of the larger SQL solutions like PostgreSQL, MariaDB or MySQL. If you've generated all your queries though the ORM (SQLAlchemy) then there will be no problem migrating to another RDBMS. But even if wrote SQL queries too there should not be much problems because all these RDBMSes use very similar dialects (unlike Microsoft's T-SQL). And since SQLite is lite it supports only a subset of what other RDBMSes support so there should not be a problem.
Recently my clients have asked me if they can use they’re application remotely, disconnected from the local network and the company server.
One solution is to place the database in the cloud, but a connection to the database, and the cloud and an internet connection must be always available.
There not always the case.
So my question is - Is there any database sync system, or a synchronization library so that I can work disconnected with local database and when I connect synchronize the changes I have made and receive changes others have made?
Update:
The application is under Windows (7/xp) ( for now )
It's in Delphi 2007 win32
All client need to have Read/Write access
All Clients have internet connection, but not always ON
Security is not critical, but the Sync service should encrypt the communication
When in the presence of the companies network the system should sync and use the Server Database and not the local one.
You have a host of issues with thinking about such a solution. First, there are lots of possible solutions, such as:
Using database replication within a database, to mimic every update (like a "hot" backup)
Building an application to copy the database periodically (every night)
Using a third-party tool (which is what you are asking, I think)
With replication services, the connection does not have to always be up. Changes to the database are logged when the connection is not available and then applied when they can be sent.
However, there are lots of other issues when you leave a corporate network. What about security of the data and access rights? Do you have other options, such as making it easier to access the database from within the network? Do the users need only read-access to the database or read-write access? Would both versions need to be accessed at the same time. Would there be updates to both at the same time?
You may have other options that are more secure than just moving a database to the cloud.
I believe RemObjects DataAbstract allows offline mode and synchronization by using what they call Briefcases. All your other requirements (security, encrypted connections, etc.) are also covered.
This is not a drop-in replacement, thought, and may need extensive rewrite/refactoring of your application. There are lots of upsides, thought; business rules can/should be enforced on the server (real security), scriptable business rules, multiplatform architecture, etc.
There are some products available in the Java world (SymmetricDS lgpl license) - apart from actually being a working system it is documents how it achieved synchronization. Connects to any db with jdbc support. . There is a pro version but the user guide (downloadable pdf) gives you the db schema plus rules on push pull syncing. Useful if you want to build your own.
Btw there is a data replication so tag that would help.
One possibility that is free is the Microsoft Sync Framework: http://msdn.microsoft.com/en-us/sync/bb736753.aspx
It may be possible for you to use it, but you would need to provid some more detail about your application and operating environment to be sure.
IS it possible to share a database like .mdb and work fine? i try but sometimes the file where the databse is changes from DB to DB1 i use delphi Xe4 and Google Drive .
Thank´s
The new security abilities of couchdb mean you can dispense with your middle-ware and access your data directly from your client if you data fits into a key value store. What if your data needs a relational database? Is there a relational db with similar abilities? Should I just tell my db server to listen on port 80?
Oracle 10g and Oracle 11g come with an embedded HTTP server.
edit
Tim Hall has a succinct overview of the embedded PL/SQL Gateway, which is part of the XML DB implementation in 10g, on his Oracle-Base site. Read it here. In another example he discusses native PL/SQL web services in 11g. Find out more.
Your question is confusing, but I will try anyway:
A relational database (RDBMS and not embedded) generally has very granular security features which includes login and authentication mechanisms -- the details are beyond the scope of a SO answer.
Telling a DB to listen on a certain port doesn't have much to do with security (unless the port is mapped and accepting internet trafic, in which case mapping it away would prevent it to listen for trafic).
In a RDBMS the relational the execution environment is your middle man, and a RDBMS will have a back-end storage structure. You generally cannot directly access the underlying engine as the execution environment does a lot of complex things -- which you cannot hope to co-ordinate with through direct access. The architecture of couch-DB is very simple compared to a RDBMS and places a lot of low-level power in the hands of the developer.
-- edit: after first comment by author --
A relational database is meant to be directly accessed -- and layers in the middle are application specific architectural decisions and additions to the RDBMS.
--edit: after second comment by Author --
If you want to access their RDBMS directly via the internet they need make the database port reachable, once that is done you need to use the native drivers/API of the database vendor.
They may:
Open up the database's port to the internet by externally mapping it (bad bad bad).
Provide you with an SSH gateway which you could use to tunnel in.
provide you with a VPN endpoint to which you can establish a VPN connetion from your network
I am building an Asp.net MVC site where I have a fast dedicated server for the web app but the database is stored in a very busy Ms Sql Server used by many other applications.
Also if the web server is very fast, the application response time is slow mainly for the slow response from the db server.
I cannot change the db server as all data entered in the web application needs to arrive there at the end (for backup reasons).
The database is used only from the webapp and I would like to find a cache mechanism where all the data is cached on the web server and the updates are sent to the db asynchronously.
It is not important for me to have an immediate correspondence between read db data and inserted data: think like reading questions on StackOverflow and new inserted questions that are not necessary to show up immediately after insertion).
I thought to build an in between WCF service that would exchange and sync the data between the slow db server and a local one (may be an Sqllite or an SqlExpress one).
What would be the best pattern for this problem?
What is your bottleneck? Reading data or Writing data?
If you are concerning about reading data, using a memory based data caching machanism like memcached would be a performance booster, As of most of the mainstream and biggest web sites doing so. Scaling facebook hi5 with memcached is a good read. Also implementing application side page caches would drop queries made by the application triggering lower db load and better response time. But this will not have much effect on database servers load as your database have some other heavy users.
If writing data is the bottleneck, implementing some kind of asyncronyous middleware storage service seems like a necessity. If you have fast and slow response timed data storage on the frontend server, going with a lightweight database storage like mysql or postgresql (Maybe not that lightweight ;) ) and using your real database as an slave replication server for your site is a good choise for you.
I would do what you are already considering. Use another database for the application and only use the current one for backup-purposes.
I had this problem once, and we decided to go for a combination of data warehousing (i.e. pulling data from the database every once in a while and storing this in a separate read-only database) and message queuing via a Windows service (for the updates.)
This worked surprisingly well, because MSMQ ensured reliable message delivery (updates weren't lost) and the data warehousing made sure that data was available in a local database.
It still will depend on a few factors though. If you have tons of data to transfer to your web application it might take some time to rebuild the warehouse and you might need to consider data replication or transaction log shipping. Also, changes are not visible until the warehouse is rebuilt and the messages are processed.
On the other hand, this solution is scalable and can be relatively easy to implement. (You can use integration services to pull the data to the warehouse for example and use a BL layer for processing changes.)
There are many replication techniques that should give you proper results. By installing a SQL Server instance on the 'web' side of your configuration, you'll have the choice between:
Making snapshot replications from the web side (publisher) to the database-server side (suscriber). You'll need a paid version of SQLServer on the web server. I have never worked on this kind of configuration but it might use a lot of the web server ressources at scheduled synchronization times
Making merge (or transactional if requested) replication between the database-server side (publisher) and web side(suscriber). You can then use the free version of MS-SQL Server and schedule the synchronization process to run according to your tolerance for potential loss of data if the web server goes down.
I wonder if you could improve it adding a MDF file in your Web side instead dealing with the Sever in other IP...
Just add an SQL 2008 Server Express Edition file and try, as long as you don't pass 4Gb of data you will be ok, of course there are more restrictions but, just for the speed of it, why not trying?
You should also consider the network switches involved. If the DB server is talking to a number of web servers then it may be being constrained by the network connection speed. If they are only connected via a 100mb network switch then you may want to look at upgrading that too.
the WCF service would be a very poor engineering solution to this problem - why make your own when you can use the standard SQLServer connectivity mechanisms to ensure data is transferred correctly. Log shipping will send the data across at selected intervals.
This way, you get the fast local sql server, and the data is preserved correctly in the slow backup server.
You should investigate the slow sql server though, the performance problem could be nothing to do with its load, and more to do with the queries and indexes you're asking it to work with.