I want to store some data in the database. Then using those data I will answer the queries for the user using Dialog flow.
Any idea on implementing these
You will need to use a webhook to do fulfillment. In your webhook, you can make the database queries you want.
You may want to use an NLIDB (natural language interface to database). An NLIDB maps natural language questions over the database schema into SQL, solves such SQL queries and returns answers. Additional misconception and ambiguity resolution steps may be included.
NLIDBs are in contrast to dialog management systems (such as DialogFlow) which use interactive dialog to fill in slots for specific question types, and then execute these questions in specialized code. This specialized code may very well interact with a database, but it is relative to a specific question type so it is fairly straight forward to implement.
The advantage of NLIDBs however is that if the mapping tool is robust, a practically infinite number of questions may be answered over a complex database schema. The disadvantage is that the mapping tools are often sometimes less than robust. But this is an area under active R&D.
There are several companies currently offering NLIDB systems.
See for example: https://friendlydata.io/, http://c-phrase.com and http://kueri.me/.
AWS might be of help. I have some answers where I detail how to use API gateway for example, as a pseudo back-end so you can run this all from a front end ( or static ) page. DOing this, my hack would be to just write a JSON file or create a variable thats imported (key/vales) which would include your database info. I created a react page once where I used a long list of database data (SQL) which i just put in a json file and imported. worked great.
Of course if you have experience building a back end, you can figure all this out. if not, i would recommend looking into wix. They have a great platform, which you can use javascript in and it also has a node back end with access to node modules. they also have fully functional built in databases. good luck!
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I'm starting my first PhoneGap project, using AngularJS. It's a database driven app, using a REST API as the backend. To start with, I'm not going to store data locally at all, so it won't do much without Internet.
However, I would eventually like to have it store data locally, and sync when Internet is available, since I know I personally disable the Internet connections on my phone at times (air planes, low battery), or have no bars. I was wondering if you could point me toward some good resources for this type of syncing. Some recommended libraries? Or perhaps some discussions of the pitfalls and how to circumnavigate them. I've Googled a bit, but I think right now, I don't know the questions to ask.
Also, my intent to build it Internet-dependent first, and then add syncing.... Is that a good idea, or am I shooting myself in the foot? Do I need to build it syncing from the start?
I had someone suggest building the app as local-only first, rather that the Internet-only part first, which has a certain logic to it. The remote storage is kind of important to me. I know the decision there has a lot to do with my goals for the app, but from the stand point of building this, with the eventual goal being local storage + Internet storage, and two-way syncing, what's going to be easier? Or does it even make a difference?
To start with, I'm thinking of using UUIDs, rather than sequential integer primary keys. I've also thought about assigning each device an ID that is prefixed on any keys it generates, but that seems delicate. Anyone used either technique? Thoughts?
I guess I need a good system to tell what data's been synced. On the client side, I guess any records that get created/edited, can be flagged for syncing. But on the server-side, you have multiple clients, so that wouldn't work. I guess you could have a last_updated timestamp, and sync everything updated sync the last successful sync.
What about records edited in multiple places? If two client edit, and then want to sync, you have some ambiguity about merging, like when merging branches in git or other version control systems. How do you handle that? I guess git does it by storing diffs of every commit. I guess you could store diffs? The more I think about this, the more complicated it sounds. Am I over-thinking it or under-thinking it?
What about client side storage? I've thought about SQLite, or the PhoneGap local storage thing (http://docs.phonegap.com/en/1.2.0/phonegap_storage_storage.md.html). Recommendations? The syncing will be over a REST API, exchanging JSON, so I was thinking something that actually stores the data as JSON, or something JSON-like that's easy to convert, would be nice. On the other hand, if I'm going to have to exchange some sort of data diff format, maybe that's what I need to be storing?
Let me provide the answer to your question based on my experience related to the sync part as I don’t have enough experience with PhoneGap so will skip the question about PhoneGap local storage v SQLite.
I was wondering if you could point me toward some good resources for this type of syncing. Some recommended libraries?
There are a number of open source projects for syncing the PhoneGap app with the remote server. But you probably have to adjust them for your own needs or implement your own sync functionality. Below I listed some of the open-source projects. You must’ve already aware of them if you’d search the net.
PhoneGap sync plugin
Simple Offline Data Synchronization for Mobile Web and PhoneGap Applications
Synchronize a local WebSQL Db to a server
Couchbase Lite PhoneGap plugin
Additionally, you might consider the other options but that depends on your server side:
Microsoft Sync Framework Toolkit (Html5 sample is available)
OpenSync Framework - platform independent, general purpose synchronization engine
Also, my intent to build it Internet-dependent first, and then add syncing.... Is that a good idea, or am I shooting myself in the foot? Do I need to build it syncing from the start?
I believe the sync functionality is more like an additional module and shouldn’t be tightly coupled with the rest of your business logic. Once you start thinking about testing strategy for your sync you’ll realise it will be easier to test that if your sync facility is decoupled from the main code.
I think you can launch your app as soon as possible with the minimum required functionality without sync. But you’d better think about your architecture and the way you add the sync facility in advance.
To start with, I'm thinking of using UUIDs, rather than sequential integer primary keys. I've also thought about assigning each device an ID that is prefixed on any keys it generates, but that seems delicate. Anyone used either technique? Thoughts?
That depends on your project specifications and specifically your server side. For example, Azure mobile services allow only integer type for the primary keys. Although unique identifiers as primary keys are pretty handy in the distributed systems (has some disadvantages as well).
Related to assigning a device ID – I am not sure I understand the point although I don’t know your project specifics. Have a look at the sync algorithm that is used in our system (bidirectional sync using REST between multiple Android clients and central SQL Server).
What about records edited in multiple places? If two client edit, and then want to sync, you have some ambiguity about merging, like when merging branches in git or other version control systems. How do you handle that? I guess git does it by storing diffs of every commit. I guess you could store diffs? The more I think about this, the more complicated it sounds. Am I over-thinking it or under-thinking it?
This is where you need to think about how to handle the conflict resolution in your system.
If the probability of conflicts in your system will be high, e.g. users will be changing the same records quite often. Then you’d better track what fields (columns) of the records had been modified in your sync and then once the conflict is detected:
Iterate through each modified field of the server side record in conflict
Compare each modified field of the server record with the relevant field of the client.
If the client field was not modified then there is no conflict so just overwrite it with the server one.
Else there is a conflict so save the both field’s content into a temporary place for the report
At the end of sync produce the report of records in conflict.
I'm pretty new to database development in general and I've never used ORM before. I'm interested in the benefits of using one, specifically saving time writing boilerplate SQL queries. I'd like to use ORM for a project that I'm working on right now, but I'm not sure it's applicable.
This project is more akin to change tracking for very small (<= 500 characters) documents. I need to track edits and categorizations made by multiple users. Not really to see the specific changes they make, but more to see if the users agree with each other. I am using a SQL database for this (as opposed to actual document control) for a few reasons:
The documents are really small; and I'm only interested in the strings, not really in files.
I wanted the ability to perform ad-hoc queries against the data for development purposes, and didn't want an unpleasantly surprised halfway through that a particular document control package couldn't do what I wanted.
From most of what I've read it seems like you need a direct mapping from columns to data fields in an object to use ORM. What I have now does not even come close to this. To create objects representing documents in different stages of editing I have to cobble together data from columns in different tables, in different combinations.
So my question is: Does an ORM like Hibernate apply to this type of project? And if it does can one be added to an existing application/database?
If it makes a difference: I'm using Java, MySQL, and JDBC. The web app users have access to for edits is made with GWT and hosted via Tomcat6. If I need it, I have complete control of the webserver.
Thanks.
Does an ORM like Hibernate apply to
this type of project?
Yes
And if it does can one be added to an
existing application/database?
Yes
My opinion is that an ORM tool could be useful for you but you really need to delve into it to see for yourself. Remember when you use an ORM tool you are not forced to use only that to connect to your database. ORM tools in general make the most sense for applications that store data in a very object like structure. For instance your user code might be the place to start. Usually you only create 1 user at a time, you edit 1 user at a time, you check if 1 user is logged into. It also makes sense for things where you would return a list of results like Order Lines. Where I have run into issues with ORM tools is when you have complex data that requires multiple joins especially back to the table you started in. For those cases you might want to keep doing what you are doing. Overall, ORM tools are great but they are like a lot of other things in software development. Try them out on a small part of your code and use them where they work and don't where they don't. Ultimately, you are the one that will have to deal with and maintain whatever you make. Just educate yourself on Hibernate and I am sure you will know what to do!
I think that ORM (I would suggest using the JPA standard, probably with Hibernate as the provider) could suit your project.
It is fairly traditional, as you say, for database columns to map directly onto object fields. If you need to keep your existing database structure (which apparently doesn't map at all well to your objects), then you might find that its more trouble than it's worth to use ORM.
While it's certainly possible to use ORM to map to a specific database schema - perhaps because it's used by other systems - my view is that one of the biggest advantages of ORM is that you can almost ignore the schema. Once you design your objects, and tell hibernate about them, hibernate will create whatever tables it needs.
The question is more simple than what it looks. There are many use cases that are well known and people have put a lot of thought into them. For example: Audit trailing, login users, and so on. We are looking for a good resource site that present the DB design for those common use cases.
Check out
http://databaseanswers.org/
There are over 500 data models available for free there. It's not hard to convert them into a working database. For some of them, if you contact the webmaster, they will send you a working MS Access application with a built in database.
Even if you don't use Access this could serve as a prototype.
Try The Data Model Resource Book by Len Silverston. It has 3 volumes and not only shows but also explains the usual use cases.
So this is the scenario:
You have a bunch of data that needs to end up in SQL.
It needs to entered by hand.
It is not an "enter once and you're done" scenario: it will need to be modified and expanded by humans in an ongoing iterative way. Comments will be associated with entries. It is also useful for data entry people to be able to see related entries near each other.
Different parts of data will need to be worked on simultaneously by different people.
Some error checking also needs to happen. (Let the data entry people correct their mistakes before SQL picks them up)
I have one answer, which is how my project currently operates, but it occurred to me that maybe there are other awesome ways of doing this which don't have the problems of my current method.
Look at YAML as a way to represent the data as plain, human-readable, and human-fixable text.
A very simple program can parse the YAML, locate errors and (if there are no errors) update the database.
These are some really basic requirements, and you probably have more issues than those stated. Nonetheless, you need a simple admin utility to enter data into your database.
A straight SQL query/update utility doesn't cut it because your team needs validation and such. You need multi-user access to the same data with transactional support. You also want to annotate your data entries and allow "related entries" to be viewed by your other users.
You need a database-maintenance application.
Consider using something like Django and it's built admin utilities. It might be more than you're expecting, but I imagine you have more needs in your future than what you've stated here.
My answer is basically
Have the data entry work in Prolog files (Prolog facts)
Have multiple files, split up in a way that is sane for the data.
Have a script that converts the Prolog facts to SQL.
Have some tests in Prolog that validate the Prolog facts.
CONS of this approach:
a little bit annoying to have to check across multiple files to see if an entry already exists, or has been moved etc.
Writing Prolog, as simple as this is, is pretty scary for non-programmers (compared to say, filling out an Excel spreadsheet, or some guided process)
maybe: Merging is tricky, or maybe my VCS is just not very smart (see Which SCM/VCS cope well with moving text between files?)
So this works pretty well, but maybe there is something better that I've never thought of!
If the constraints you're referring to can be enforced at the database level, free software like Quest Toad could allow them enter data directly into the db. It feels very much like using a spreadsheet when in grid view and displays an error when constraints are violated.
Alternatively, depending on what existing stack you have available, .Net grid views make it easy to slap together crud screens in little time.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I have been developing web/desktop applications for about 6 years now. During the course of my career, I have come across application that were heavily written in the database using stored procedures whereas a lot of application just had only a few basic stored procedures (to read, insert, edit and delete entity records) for each entity.
I have seen people argue saying that if you have paid for an enterprise database use its features extensively. Whereas a lot of "object oriented architects" told me its absolute crime to put anything more than necessary in the database and you should be able to drive the application using the methods on those classes?
Where do you think is the balance?
Thanks,
Krunal
I think it's a business logic vs. data logic thing. If there is logic that ensures the consistency of your data, put it in a stored procedure. Same for convenience functions for data retrieval/update.
Everything else should go into the code.
A friend of mine is developing a host of stored procedures for data analysis algorithms in bioinformatics. I think his approach is quite interesting, but not the right way in the long run. My main objections are maintainability and lacking adaptability.
I'm in the object oriented architects camp. It's not necessarily a crime to put code in the database, as long as you understand the caveats that go along with that. Here are some:
It's not debuggable
It's not subject to source control
Permissions on your two sets of code will be different
It will make it more difficult to track where an error in the data came from if you're accessing info in the database from both places
Anything that relates to Referential Integrity or Consistency should be in the database as a bare minimum. If it's in your application and someone wants to write an application against the database they are going to have to duplicate your code in their code to ensure that the data remains consistent.
PLSQL for Oracle is a pretty good language for accessing the database and it can also give performance improvements. Your application can also be much 'neater' as it can treat the database stored procedures as a 'black box'.
The sprocs themselves can also be tuned and modified without you having to go near your compiled application, this is also useful if the supplier of your application has gone out of business or is unavailable.
I'm not advocating 'everything' should be in database, far from it. Treat each case seperately and logically and you will see which makes more sense, put it in the app or put it in the database.
I'm coming from almost the same background and have heard the same arguments. I do understand that there are very valid reasons to put logic into the database. However, it depends on the type of application and the way it handles data which approach you should choose.
In my experience, a typical data entry app like some customer (or xyz) management will massively benefit from using an ORM layer as there are not so many different views at the data and you can reduce the boilerplate CRUD code to a minimum.
On the other hand, assume you have an application with a lot of concurrency and calculations that span a lot of tables and that has a fine-grained column-level security concept with locking and so on, you're probably better off doing stuff like that directly in the database.
As mentioned before, it also depends on the variety of views you anticipate for your data. If there are many different combinations of columns and tables that need to be presented to the user, you may also be better off just handing back different result sets rather than map your objects one-by-one to another representation.
After all, the database is good at dealing with sets, whereas OO code is good at dealing with single entities.
Reading these answers, I'm quite confused by the lack of understanding of database programming. I am an Oracle Pl/sql developer, we source control for every bit of code that goes into the database. Many of the IDEs provide addins for most of the major source control products. From ClearCase to SourceSafe. The Oracle tools we use allow us to debug the code, so debugging isn't an issue. The issue is more of logic and accessibility.
As a manager of support for about 5000 users, the less places i have to look for the logic, the better. If I want to make sure the logic is applied for ALL applications that use the data , even business logic, i put it in the DB. If the logic is different depending on the application, they can be responsible for it.
#DannySmurf:
It's not debuggable
Depending on your server, yes, they are debuggable. This provides an example for SQL Server 2000. I'm guessing the newer ones also have this. However, the free MySQL server does not have this (as far as I know).
It's not subject to source control
Yes, it is. Kind of. Database backups should include stored procedures. Those backup files might or might not be in your version control repository. But either way, you have backups of your stored procedures.
My personal preference is to try and keep as much logic and configuration out of the database as possible. I am heavily dependent on Spring and Hibernate these days so that makes it a lot easier. I tend to use Hibernate named queries instead of stored procedures and the static configuration information in Spring application context XML files. Anything that needs to go into the database has to be loaded using a script and I keep those scripts in version control.
#Thomas Owens: (re source control) Yes, but that's not source control in the same sense that I can check in a .cs file (or .cpp file or whatever) and go and pick out any revision I want. To do that with database code requires a potentially-significant amount of effort to either retrieve the procedure from the database and transfer it to somewhere in the source tree, or to do a database backup every time a minor change is made. In either case (and regardless of the amount of effort), it's not intuitive; and for many shops, it's not a good enough solution either. There is also the potential here for developers who may not be as studious at that as others to forget to retrieve and check in a revision. It's technically possible to put ANYTHING in source control; the disconnect here is what I would take issue with.
(re debuggable) Fair enough, though that doesn't provide much integration with the rest of the application (where the majority of the code could live). That may or may not be important.
Well, if you care about the consistency of your data, there are reasons to implement code within the database. As others have said, placing code (and/or RI/constraints) inside the database acts to enforce business logic, close to the data itself. And, it provides a common, encapsulated interface, so that your new developer doesn't accidentally create orphan records or inconsistent data.
Well, this one is difficult. As a programmer, you'll want to avoid TSQL and such "Database languages" as much as possible, because they are horrendous, difficult to debug, not extensible and there's nothing you can do with them that you won't be able to do using code on your application.
The only reasons I see for writing stored procedures are:
Your database isn't great (think how SQL Server doesn't implement LIMIT and you have to work around that using a procedure.
You want to be able to change a behaviour by changing code in just one place without re-deploying your client applications.
The client machines have big calculation-power constraints (think small embedded devices).
For most applications though, you should try to keep your code in the application where you can debug it, keep it under version control and fix it using all the tools provided to you by your language.