I am building a web app that will use an auto-complete/suggestions for the end user as they type their information in. This will be specifically for entering Country, Province, City information.
Do a wild card search on the database on each keystroke:
SELECT CityName
FROM City
WHERE CityName LIKE '%#CityName%'
Return a list of all Cities to a given Province to the client and have the client do the matching:
SELECT CityName
FROM City
WHERE ProvinceID = #ProvinceID
These would be returned to the client as a JSON string via an ajax call to a web service. My thoughts are that javascript would be able to handle the list of 100+ entries via JSON faster than the database would be able to do a wildcard search, but I'd like the communities input.
In the past, I have used both techniques. If you are talking about 100 or so entries, and assuming each entry is very small, it will likely be faster to do the autocomplete filter on the client side. That will provide you with better response time (although probably negligible) and will reduce the load on your server.
Google actually does a live search while the user is typing, and it seems to be pretty responsive from the user's point of view. This is an example where the query must be executed server-side because the dataset is far too large to transfer to the client.
One thing you might do is wait until the user types two keystrokes before fetching the list from the server, thus narrowing down the results initially. Of course, that adds complexity - you would then need to refresh the list if the user changes either of the first two keystrokes.
We have implemented same functionality using ajax auto complete control we wait the user type three keystroke before fetching the list from server we have not done any coding at client side we just assigned web services method which return list to ajax control and its start working
In the end user's interest, it is always better to handle this client-side.
The Telerik Autocomplete controller allows for both ways.
Of course under load client-side autocomplete is likely to make the application crawl.
Related
We are a using a micro-service based pattern for our project where we have Users and their Orders. Users personal information (name, email, mobile) is stored in User table in relational database while we are storing Orders data of users in Orders collection in NoSql database. We want to develop an API to get a paginated list of all the orders placed with order details along with finer details of user associated like - user name, mobile, email along with each order. We are storing userId in Orders collection.
The problem is how do we get User details for each order in this list since both the resources are in different databases. We also thought of storing user name, email and mobile in Orders collection only but what if a user updates their profile, the Orders collection will have stale user data.
What is the best approach to address this issue?
You can use API gateway pattern, UI will call to API gateway endpoint and the Endpoint will call the both the API/services to get the result and aggregate it then returns aggregated response to the UI (caller)
https://microservices.io/patterns/apigateway.html
Well it mostly depends on scalability needs in terms of data size and number of requests. You may go with the API gateway if you don't have too much data and you don't get many requests to that service.
Otherwise if you really need something scalable then you should implement your own thought with an event based communication.
I already provided an answer for a similar situation you can take a look
https://stackoverflow.com/a/63957775/3719412
You have two services Orders and Users. You are requesting Orders service to get all Orders. It will return a response data which will contains ID of Users (each Order contains ID of User). Then, you will make a request to a Users service to get an information regarding User by ID which you got before. And finally, you can aggregate those results (if it is needed).
As guys mention, good solution will be to implement API Gateway here. As a client, you will send a request to a single port with endpoint (to a Gateway) and Gateway should create logic which I have described before.
I am writing a web app and I am trying to improve the performance of search/displaying results. I am relatively new to programming this sort of thing, so I apologize in advance if these are simple questions/concepts.
Right now I have a database of ~20,000 sites, each with properties, and I have a search form that (for now) just asks the database to pull all sites within a set distance (for this example, say 50km). I have put the data into an index and use the Search API to find sites.
I am noticing that the database search takes ~2-3 seconds to:
1) Search the index
2) Get a list of key names (this is stored in the search index)
3) Using key names, pull from datastore (in a loop) and extract data properties to be displayed to the user
4) Transmit data to the user via jinja template variables
This is also only getting 20 results (the default maximum for a Search API query.. I haven't implemented cursors here yet, although I will have to).
For whatever reason, it feels quite slow.. I am wondering what websites do to make the process seem faster. Do they implement some kind of "asynchronous" search, where a page loads while in the background the search/data pulls are processed, and then subsequently shown to the user...?
Are there "standard" ways of performing searches here where the processing/loading feels seamless to the user?
Thanks.
edit
Would doing something like just passing a "query ID" via the page work, and then using AJAX to get data from the datastore via JSON work? Like... can app engine redirect the user to the final page, pass in only a "query ID", and then search in the meantime, and then once the data is ready, pass the information the user via JSON?
Make sure you are getting entities from the datastore in parallel. Since you already have the key names, you just have to pass your list of keys to the appropriate method.
For db:
MyModel.get_by_key_name(key_names)
For ndb:
ndb.get_multi([ndb.Key.from_path('MyModel', key_name) for key_name in key_names])
If you needed to do datastore queries, you could enable parallel fetches with the query.run (db) and query.fetch_async (ndb) methods.
In the single page webapp I've recently built I'm getting data for my models using Restangular module. I'd like to add real-time updates to the app so whenever any model has been changed or added on the server I can update my model list.
I've seen this working very well in webapps like Trello where you can see the updates without refreshing the web page. I'm sure Trello webclient uses REST API.
What is a proper way to architect both server and client to archive this?
First of all, your question is too general and can have a lot of solutions that depend
on your needs and conditions.
I'll give you a brief overview for a single case when you want to leave REST APIs
and add some realtime with web sockets.
Get all data from the REST -- Sokets for notifications only.
Pros: Easy to implement both server side and client side. You only need to emit events on the server with
info about modified resource (like resource name and ID), and catch these events on the client side and fetch
data with REST APIs.
Cons: One more request to the server on every notification. That can increase traffic dramaticaly when you have a lot of active clients for a single resource (they will generate a lot of reverse requests to the server).
Get initial load from the REST -- Sockets for notifications with data payload.
Pros: All info comes with the notification and will not cause new requests to the server, so we have less traffic.
Cons: Harder to implement both server side and client side. You will need to add data to all the events on the server. You will need to fetch data from all the events on the client side.
Updated according to the comment
As for handling different types of models (just a way to go).
Client side.
Keep a factory for each model.
Keep in mind that you need realtime updates only for displayed data (in most cases), so you can easily
use memory caching (so you can find any entity by its ID).
Add listener for every type of changes (Created, Updated, Deleted).
In any listener you should call some initObject function, that will find entity in the cache by ID and extend it, if there is no entity with such ID, just create a new one and add it to cache.
Any Delete just removes an entity from the cache.
Any time you need this resource, you should return the link to cache object in order to keep two way databinding (that is why I use extend and not =). Of course, you need to handle the cases like: "User is editing the resource while notification about deleting comes".
Server side.
It is easier to send all the model then just modified fields (in both cases you must send the ID of resource).
For any Create, Update, Delete event push event to all engaged users.
Event name should contain action name (like New, Update, Delete) and the name of resource (like User, Task etc.). So, you will have NewTask, UpdateTask events.
Event payload should contain the model or just modified fields with the ID.
Collection changes can be handled in two ways: with add/update/remove items in collection or changing all the collection as a whole.
All modifications like PUT, POST, DELETE are made with REST of course.
I've made a super simple pseudo gist for the case 1). https://gist.github.com/gpstmp/9868760 but it can be updated for case 2) like so https://gist.github.com/gpstmp/9900454
Hope this helps.
In a web application, we often come across a form submission process that spans across several pages, for ex: In first form we capture basic information, next page capture some other information and so on. I have a scenario where I've 7 screens to capture all the details about user and "Submit" button appears on 7th page.
Usually we store all the intermediate values in HttpSession and when its time to submit we retrieve all the values from Session and create an entry in database.
With this approach, by the time user completes all the form entries (i.e. from Page 1 to Page 7), everything resides in Session.
I would like to know, is there any alternative apart from HttpSession for storing the intermediate values?
I'm actually trying to find the ways to make my HttpSession less bulky.
You can also store just the reference in a session which then maps to a cache like e.g. Memcached. Or if it is important that you don't lose the data while the user walks through the steps, you can also persist the data in a database and just refer via a key from a session to it. To store too much data in the session is sometimes not the best choice, so I would just store a reference there.
You can try caching technology of .Net, this might be useful instead of using session for all the data, also you can just use the id of the session for the cache id.
Second option I think is configuring your Session-State mode to use SQLServer mode for the storage.
I'd like to set up a coldfusion page that will pull the status updates from my own facebook account and twitter accounts and put them in a SQL database along with their timestamps. Whenever I run this page it should only grab information after the most recent time stamp it already has within the database.
I'm hoping this won't be too bad because all I'm interested in is just status updates and their time stamps. Eventually I'd like to pull other things like images and such, but for a first test just status updates is fine. Does anyone have sample code and/or pointers that could assist me in this endeavor?
I'd like it if any information relates to the current version of the apis (twitter with oAuth and facebook open graph) if they are necessary. Some solutions I've seen involve the creation of a twitter application and facebook application to interact with the APIs; is that necessary if all I want to do is access a subset of my own account information? Thanks in advance!
I would read the max(insertDate) from the database and if the API allows you, only request updates since that date. Then insert those updates. The next time you run you'll just need to get the max() of the last bunch of updates before calling for the next bunch.
You could run it every 5 minutes using a ColdFusion scheduled task.
How you communicate with the API is usually using <cfhttp />. One thing I always do is log every request and response, either in a text file, or in a database. That's can be invaluable when troubleshooting.
Hope that helps.
Use the cffeed tag to pull RSS feeds from Twitter and Facebook. Retain the date of the last feed scan somewhere (application variable or database) and loop over the feed entries. Any entry older than last scan is ignored, everything else gets committed. Make sure to wrap cffeed in a try/catch, as it will throw errors if the service is down (ahem, twitter) As mentioned in other answers, set it up as a scheduled task.
<cffeed action="read" properties="feedMetadata" query="feedQuery"
source="http://search.twitter.com/search.atom?q=+from:mytwitteraccount" />
Different approach than what you're suggesting, but it worked for us. We had two live events, where we asked people to post to a bespoke Facebook fan page, or to Twitter with a hashtag we endorsed for the event in realtime. Then we just fetched and parsed the RSS feeds of the FB page, and the Twitter search results, extracting what was new, on a short interval... I think it was approximately every three minutes. CFFEED was a little error-prone and wonky, just doing a CFHTTP get of the RSS feeds, and then processing the CFHTTP.filecontent struct item as XML worked fine
.LAG