I'm building a service where people can basically find listings based on location. The service is offered in Germany and locations there seem to be extra tricky.
So how is the current setup?
To every listing I'm storing an address like that using nominatim data. The address data is based on zipCodes. So Im receiving zipCodes and out of this storing this address object:
address: {
suburb: "Neukölln",
city_district: "Neukölln",
city: "Berlin",
state: "Berlin",
postcode: "12053",
country: "Deutschland",
country_code: "de",
lat: "52.4775104",
lon: "13.4322809163976",
displayName: "Neukölln, Berlin, 12053, Deutschland"
},
The user in the frontend then uses autosuggest based on komoot api which gives me something in following format:
city: "Berlin"
country: "Deutschland"
name: "Berlin"
osm_id: 240109189
osm_key: "place"
osm_type: "N"
osm_value: "city"
postcode: "10117"
state: "Berlin"
As you can see I can search now with osm_value and name. So I would search city:Berlin.
Theoretically this works quite well. But for bigger cities like Berlin nominatim and komoot are giving very bad results. Often times with nomnatim city is missing even tough there is a city_district. (See this issue: https://github.com/openstreetmap/Nominatim/issues/1487)
My question would be: How could a proper data structure look like and do you have a better idea to build this? Paid services are fine - I just haven't found one proper (besides extremely expensive gmaps).
Requirements are:
Search for zipCode, city, city_district, village, town, county
Having roughly 300k geocodings per month
Related
Every order has a shipping info and every user has several shipping addresses. This is my design:
User:
...
defaultAddress: ObjectId
...
Address:
...
...
...
Order:
...
shipping: ObjectId
...
Here are my questions:
Is it reasonable to use a reference for shipping in Order Model, or just embed the entire Address object.
Every time add a new address from App, Is there any better way to check if it is already exist in mongodb?
Should I keep a defaultAddress key in User Model, or should I just use a array to keep all the address object?
Thanks very much.
Addresses bind closely to Users, so make Addresses an array inside Users.
Depending on your overall audit / revisioning data architecture, you may wish to copy the address into the Order doc upon creation to create an unambiguous record of where it went. Having Order reference back into User and address implies you cannot change the address; you would have to mark it as no longer active but not deleted.
Note you do not have to laborious copy all the fields; you can treat the selected address as an object, something like:
var addr = get addr; // from DB or wherever; any shape with any number of subfields is fine.
db.order.insert({orderId:"someID", date:new ISODate(), shippingAddress: addr});
To prevent dupes, it is a matter of looking up the info in the array -- but of course caveats apply such as making sure you canonicalize case, strip white space, remove unnecessary punctuation, etc. etc. Let's assume you have street,city, and state as the matchable fields:
{
"user":"U1",
"addrs": [
{street: "20 Pine Ln", city: "Smallville", state: 'NM'},
{street: "16 Elm Ln", city: "Denver", state: 'CO'},
{street: "77 Sunshine Pkway", city: "Denver", state: 'CO'},
]
}
After grooming the input from app, assume we have a target lookup doc like this. Note: We do NOT have to create the doc with fields in a particular order.
var target = {city: "Denver", state: 'CO', street: '16 Elm Ln'};
The lookup is simple. Since addrs is an array, the match will dive into and iterate over the array. The $elemMatch operator ensures that ALL fields in the candidate match those in the target.
db.foo.aggregate([
{$match: {user:'U1', "addrs":{$elemMatch: target} }}
]);
If no records are found, then this is a new address.
I'm using use-places-autocomplete fro getting results from the google api. On my hook I have the following configuration
usePlacesAutocomplete({
requestOptions: {
types: ["address"],
componentRestrictions: {
country: "us",
},
}
But when typing Holly I'm getting some results that don't have specific address as the one that are on the image, because they are cities.
What I'm doing wrong in this case? I want to get only the suggestions from USA with a specific address. Does the hook enabled this ?
Since I am new to React Js, I wanted to use google address autocomplete service. So i came to find npm package "react-places-autocomplete" package which is great to implement. But one this i came to found that it return address but no postal_code. Since i needed post_code, has anybody ever done customization to get postal_code using this package ?
id: "0a69230df738495807b4d2a21e399895416cf7ae", description: "8 Burnley Street, Richmond VIC,
Australia", placeId: "ChIJ-RqIcFND1moR6pgdVLp_7Ss", active: false, index: 4, …}
id: "0a69230df738495807b4d2a21e399895416cf7ae"
description: "8 Burnley Street, Richmond VIC, Australia"
placeId: "ChIJ-RqIcFND1moR6pgdVLp_7Ss"
active: false
index: 4
formattedSuggestion: {mainText: "8 Burnley Street", secondaryText: "Richmond VIC, Australia"}
matchedSubstrings: (2) [{…}, {…}]
terms: (5) [{…}, {…}, {…}, {…}, {…}]
types: (2) ["street_address", "geocode"]
__proto__: Object
Postal Code is missing in the response. I hope to get atleast some hints how to do it.
react-places-autocomplete will probably only be using the Places Autocomplete Service, in which case I don't believe that it returns a postcode at all.
So the simple answer would be no, you can't directly get the postcode from react-places-autocomplete or the google places autocomplete service.
The long answer would be, you can make use of other google and free services to get the postcode.
In your prediction you will have a property called place_id, you can then use that with googles getDetails, for example, to get a PlaceResult where you can then get address details.
If at this point you still can't get what you need from google you can then use a free service like https://postcodes.io/ to get the postcode from the lat and lng coordinates.
I have a blog post about using react with google places autocomplete service here that you might find useful.
And a post here about how you can use postcodes with the JavaScript Google Places Api
Sources:
https://developers.google.com/maps/documentation/javascript/reference/places-autocomplete-service
https://postcodes.io/
https://developers.google.com/maps/documentation/javascript/reference/places-service#PlacesService.getDetails
https://developers.google.com/maps/documentation/javascript/reference/places-service#PlaceResult
I am a fairly new web developer and would need your help with a project I am currently working on. I have worked in the past on a very simple realtime database example and have little to none experience in firestore or NoSql in general.
I want to create a system which allows end-users to get an email once a week that contains a list of special offers from bars the end-user has subscribed to. The offers change each day of the week. Bar owners can fill out a form in a vue.js web application every week with their weekly special offers.
Every Monday morning a cron job has to look up which end user has subscribed to which bars and then aggregate the data and send it via email.
The question is how would you structure the data so that I can easily compose the email and send it via a cloud function?
My approach would be to have three main collections: RestaurantOwner, EndUser, SpecialOfferings
Please see the graphic for an example process:
BarOwner and EndUser are pretty straight forward. However, the difficult part is how to structure the SpecialOffers in order to be queried the right way.
My idea would be to structure it based on the calendar week and link it to the uid from the barOwner:
specialOffers: {
2019_CW27: {
barUID001: {
mon: {
title: 'Banana Daiquir',
price: 4.99,
},
tue: {
title: 'After Five',
price: 2.99,
},
wed: {
title: 'Cool Colada',
price: 6.99
},
thu: {
title: 'Crantini',
price: 5.99
},
fri: {
title: 'French Martini',
price: 4.99
}
},
barUID002: {
mon: {
title: 'Gin & Tonic',
price: 8.99,
},
tue: {
title: 'Cratini',
price: 4.99,
},
wed: {
title: 'French Martini',
price: 4.99
},
thu: {
title: 'After Five',
price: 3.99
},
fri: {
title: 'Cool Colada',
price: 6.99
}
}
},
2019_CW28: {
barUID01: {~~~},
barUID02: {~~~}
}
}
The disadvantage of this approach is that it creates a deeply nested object when you imagine that there are 52 calendar weeks, f.e 100 signed up bars à 5 special offers per week and I am not sure if I am able to query it the way I need to.
Is this approach reasonable or what would you do differently?
Thank you so much for your help! I highly appreciate it.
I'm assuming the following scenarios:
1) The bar owners make modifications to their offers very often.
2) The bar owners should be the only ones allowed to modify each bar's offers.
If you have these two scenarios, I would recommend a sub-collections approach here.
When to use sub-collections:
1) When there are lot of fields in a document. Cloud Firestore has 20,000 field limit. (If the number of Bars can exceed more than 20,000 fields)
2) When updating the parent collection is a common operation. Firestore only lets you update the document at rate of 1 write/second. (If the SpecialOffers information of each bar is modified very often. If two bar owners modify their offers, only 1 write is successful and the second write operation waits until the first is completed. This can delay the updation offers particularly at the end of a week when almost all the bars update the offers.)
3) When you want to limit the access to particular fields of a document. (If you want to restrict the access to a Bar's Offers to the barOwner alone. You can restrict the access to each document in the Bars sub-collection according to its owner using Firestore Security Rules)
So I would recommend a sub-collection Bars under the main collection SpecialOffers. This way the design becomes scalable and you can add restaurants and super-markets as other similar sub-collections in the future without heavily altering your design.
Another advantage is that sub-collections are basically collections and they don't have a limit for number of documents it can hold. So even if the number of bars registered is above 20,000 which is the limit of number of fields for a fire-store document, your sub-collection wont be having a problem but your document will run out of fields to save the offers for a new bar.
Ultimately the choice depends on your use cases.
Hope this helps.
Let's say I'm building Twitter.
One of the tasks is to track, which tweets are read by particular user and store this data on server. When user requests somebody's feed, server should return:
[
{
id: 1,
tweet: "Hey there!",
isRead: false
},
{
id: 2,
tweet: "Here's my cat, look",
isRead: true
},
{
id: 2,
tweet: "Blue or yellow? Thats the question",
isRead: true
},
...
]
Which is the most efficient way to store data for which tweets are read by which user, and retrieving this data when returning somebody's feed for particular user?
Any ideas about data storing architecture are highly appreciated. My current stack is PostgreSQL for storing users and "tweets". Redis, MongoDB and neo4j are also used in the project, so available.
The first guess was to use Redis, like:
user_id: tweet_id
-----------------
user_id: tweet_id
-----------------
....
But I think, there may be better variants, more suitable for persistent data storage.
Thank you in advance.
Have a look at this Twitter-clone that Redis' author, antirez (a.k.a Salvatore Sanfilippo), had made: http://redis.io/topics/twitter-clone