I am working on a music game website and I am using the Spotify play button. This takes spotify track numbers/urls and I am looking for a huge list of them with which to populate my database. Does anyone know where I could find such a list? Or do I have to write a script to try and scrape them?
Well, there are many millions of tracks in Spotify so that'd be a pretty unmanageable list.
More importantly, scraping the Spotify catalog like this is very much against Spotify's API Terms of Service, and if you try it you'll find yourself rate limited pretty fast.
Instead, consider asking users of their game to search for something - searching for a user-supplied string and caching the results of the search doesn't break the Terms of Service.
Related
I am building a web application using React on the front end and Ruby on Rails on the backend. I am a newbie developer but am working on a real world project to help local pharmacies better respond to patient medication requests. I was hoping to get some insight on a search box feature I am working to implement.
I want to create this feature so that a patient can search for a medication name as they type into a search box that generates autocomplete suggestions. We are using a dataset of 10,000 medications, so the challenge here is how to deal with this large dataset while making the user experience pleasant. Last thing I want to do is send a request every time a user types in a word to fetch relevant medication names and have it be laggy due to round trip response.
One approach I have thought of was to seed the medications to a Postgres SQL database, then when the patient loads the page with the search box, fetch all the records and maybe use some sort of cache to store on the browser. Rails documentation seems to have some good information on this.
I was also looking into elasticsearch, since I hear its good for large dataset retrievals.
Does anyone have any suggestions on any other concepts or technologies I could explore for the particular feature I want to build without sacrificing user experience?
I am trying to create a forum app using Django as a backend and React as a front end. I want to find out how many people have visited a post created by a user so that I can store as views and list the posts according to popularity.
I am just a student and I have no experience with live websites, so I'm wondering if it is okay to just save a user at componentdidmount life cycle? But I'm afraid it'd make the same user be counted as many as he visits and the post creator will be able to increase his post's popularity by just spamming his website.
I would suggest you implement this on the backend, not the front end. I don't know Django well, but there should be some way to know whether that particular post is getting requested. At that point, you'll want to increment the counter for that post.
The problem of course is determining which "views" count as real views. Was it the poster? Was it a robot? Was it a spider? Was it a scraper? Was it the same person who is not the poster viewing it many times?
I wouldn't say this last part is not an easy thing to implement, and would probably take some trial and error before finding the right conditions to get your metrics "right".
As #Mike suggests above, there are many analytics packages which use sophisticated algorithms to determine "realness", and you may be able to use this data. My understanding is that you want to actually apply the data to sorting and UI for your app, not just view it on the dashboard of your analytics tool. I've never tried to look for one that supports an API to discover what you're interested in programmatically, but they all probably allow you do download structured data about your traffic. The problem with the latter approach is that creates a delay and a manual step (always something to try to avoid imo).
I have seen so many e-commerce websites that provides search box to search products. In that search features most of the search fields are auto-complete. If we enter a letter on field, then it will show the data which is including that letter as suggestions from database. As I know basics on developing that functionality.
But what if database contains huge amount of data?
For example, e-commerce websites like flipkart and amazon had a lot of products in their database. so, if user enter a letter in search field, it have to search for data including that letter among all the data in database and match data including that letter and display data as suggestions. The websites are processing it within nano seconds of time. I wonder how they achieved that functionality? I can't understand what are the technologies they are using.
As a learner I wanna know the functional flow and if possible demo for that feature.
I think your question can be divided into two parts. 1) how to design the database for the search technology. 2) how to implement an effect search... They belong to the field of search engine technology.
About the Q1, you can create a table to save the keywords for search, and in the table, you'd better to design a column or similar method to describe the "search-weight". As well known, a view is a practical solution to accelerate the access of the data.
About the Q2, the search engine technique is No longer mysterious, some open source projects can simulate the feature of search engine, such as Apache Lucene, visit please Apache Lucene.
more discuss:
And specially, in your front system for example, the ASP/JSP or even simple HTML page, you should use some scripts e.g. Ajax, to popup, drawdown, of caurse, simple DOM Javascript+DIV can reach it too, but with jQuery or other libarary can make it easily. Here is an example.
Here is the backend system example
To reduce the burden on the host and reduce the requirement of network's bandwidth, the front javascript should active the autocomplete feature with more than three characters.
Please pay attention in your actual application, that your server has calculation's limitation, and the client page has usually many elements, all will reduce user friendliness. Please do not make the request and response too complex.
An alternative simulation can be: make a FIFO logic, save some usual search keyword in the "cache" or temp-table|view, and the amount of data will be reduced.
There are too many solutions, I can only think of these tricks at this moment.
regards
One of my customer requested some changes on shopify site. She sell pictures and she would like to start offer the frames.
But the whole administration of the frames will be so complicated that I know now, that I will need to somehow extend the data model, because I will need to store some additional relations.
So my question is: Is it somehow possible to store any kind of data by shopify API? Like create a new entities with custom attributes etc. I was searching through the API documentation but I was not able to find any solution.
Second question is: Would it be possible to solve this problem by Embedded App? Thats mean, that I will develop the whole administration part as a small application and then embed it to the shopify? Will it be possible then join data from shopify storage and my database through Shopify API?
Is there some example for this scenario?
Thanks you for your help.
I built a Shopify store selling frames when Shopify first came out. That shop still sells many many frames. There was not even a Shopify API back then. Still, it was possible to do it with 100% client-side code. You simply price each frame in units. For example, $1 per inch. Client-side you collect the frame size as length x width, and come up with the total inches needed, and there is your quantity. Use mm, cm whatever units work for you. You can even get fancy if you're good, and work in mats, backing, and types of glass, all with one add to cart click.
I'm with a company that is building a venue / artist database for live music and recently came across Freebase. It looks very compelling, even if the data isn't there for new, up-and-coming bands. For those of you who have worked with Freebase, I have a couple questions:
Are there downsides to integrating all of the data entry with Freebase? We are not looking to sell or privatize this information.
What are the weaknesses of Freebase, with regards to usability?
Disclosure: I work on Freebase at Google.
The music data in Freebase is one of our strongest areas and is going to continue to get broader and richer as we continue to load more datasets. For example, we import data from MusicBrainz, clean it up and match the topics against existing topics in Freebase to avoid duplicates.
In terms of downsides, you should be prepared to work with a lot of data. For example, Freebase currently has 4 musical artists named "John Smith" which may or may not be useful for your application but you'll still need to figure out which one(s) map to the John Smith that your users are interested in. We call this "reconciliation" and its necessary so that your app knows precisely which topics to query the API for.
Since you mentioned music venues I should also point out that while Freebase has a lot of data about places, we don't yet have a geosearch API so you'd need to roll your own if that's something you need.
Since anyone can edit Freebase, you should also consider using as_of_time to protect your site against vandalism.
Freebase is great for developers because you can easily jump in and clean up bad data or add missing topics. However, one area that has always been a challenge is loading large amounts of data from outside of Google. We've built the OpenRefine which allows folks to upload datasets, but these datasets must pass a QA process that takes some time to complete. Its necessary to have these QA processes to maintain the level of quality in Freebase, but it does slow down the process of loading large datasets.
I really hope that you choose to make use of Freebase music data to build your company. I know that there are already a number of music startups happily using our data.