What is the best way to find out if a address/postalcode is within 60 miles of an address in Snowflake? In Google BigQuery there is the bigquery-public-data.geo_us_boundaries.zip_codes that allows you to this.
The Snowflake Marketplace has multiple data sets available for geocoding such as Worldwide Address Data. After geocoding your addresses, you can use Snowflake geospatial function ST_DISTANCE to calculate distances.
It would help a lot if you can provide us a new question with a sample query that you run in BQ. That would help us make sure that we are getting what you want, and we would be able to test it.
In the meantime, let me give you a query with the Worldwide Address Data dataset that Dave suggests:
select st_distance(st_point(lon, lat), xy), postcode
from worldwide_address_data.address.openaddress
join data
where country='us'
and postcode != ''
and lon between -180 and 180
and lat between -90 and 90
and postcode like '95%'
and length(postcode)>=51
order by 1 desc
limit 100
This could/needs to be optimized, but we can get there as you give us more specifics.
Related
I looked at several other SO questions that seem somewhat related, but not quite what i need (or i'm just not smart enough to connect the dots).
Working on an app for a client. Their database holds the origin and destination of people that are traveling, limited (i believe) to just places in US and Canada, and a date when the trip will take place. The records are updated regularly. Call these "trips."
Users come to the site, and enter an origin and destination city, and a radius for each, indicating how far away from their desired origin/destination cities they are willing to travel in order to make their trip.
The job of the app is to find any/all trips that are already in the database, that are closest to the origin and destination that the user needs to travel.
My original thought was to find all origin cities in the database that are within the radius of the user's desired origin, then use that recordset to search the destination cities in the database for any/all cities within the radius of the user's desired destination.
I also need a decent (preferably free... low budget project here) API that can help look up the city geographic location and perform the actual radius calculation... I think.
Is what I'm looking to do even close to the best options? It looks like the hardest part will be finding all the existing cities in the database that are within the radius of the user's desired cities - which is a bit of a twist on a more simple query of just "find all cities in the radius of X city".
So, this is KINDA like an Uber situation, except the Uber driver is deciding what the trip parameters are, and the user just needs to know which Uber drivers are going from/to the places nearest those of the user (on the specified date, to boot).
Right now, users are just looking things up at a state level - BC to NY, and reading down rows of data looking at rides to find the ones that seem closest to what they need.
Thanks in advance, for any clever insights you smart folks might have!
Declare #DriverLat float = 41.744068
Declare #DriverLng float = -71.315024
Declare #Within int = 20
Select *
From (
Select Distinct
A.ZipCode
,A.CityName
,A.StateCode
,Miles = [dbo].[udf-Geo-Calc-Miles] (#DriverLat,#DriverLng,A.Lat,A.Lng)
From [dbo].[ZipCodes] A
Where CityType = 'D'
and ZipType = 'S'
) A
Where Miles <= #Within
Order By Miles
Returns
The UDF
CREATE Function [dbo].[udf-geo-Calc-Miles] (#Lat1 float,#Lng1 float,#Lat2 Float,#Lng2 float)
Returns Float as
Begin
Declare #Miles Float = (Sin(Radians(#Lat1)) * Sin(Radians(#Lat2))) + (Cos(Radians(#Lat1)) * Cos(Radians(#Lat2)) * Cos(Radians(#Lng2) - Radians(#Lng1)))
Return Case When #Miles is null then 0 else abs((3958.75 * Atan(Sqrt(1 - power(#Miles, 2)) / #Miles))) end
End
I intend to analyse multiple data sets on the same time series (daily EOD). I will need to use computed columns. Use column A + B to create column C (store net result of calculation in column C). Is this functionality available using the MongoDB / Arctic database?
I would also intend to search the data... for example: What happens when the advance decline thrust pushes over 70 when the cumulative TICK was below -100,000 in the past 'n days'
Two data sets: Cumulative TICK and the Advance Decline Thrust (Uses advancers / decliners data). So they would be stored in the database, then I would want to have the capability to search for the above condition. This is achievable with the mongoDB / Arctic database structure?
Just looking for some general information before I move to a DB format. Currently everything I had created is on excel / VBA now its alrady out grown!
Any information greatly appreciated.
Note: I will use the same database for weekly, monthly, yearly and 1 minute, 3 minute, 5 minute 60 minute TICK/TIME based bars - not feeding live but updated EOD
yes, this can be done with arctic. Arctic can store pandas dataframes, and an operation like you have mentioned is trivial in pandas. Arctic is just a store, so you'd want to read the data out of arctic (data is stored in symbols in arctic) and then perform your transform, and then write the data back. Any of the storage engines (VersionStore, TickStore, or ChunkStore) should work for this.
I am analysing app data which contains lat value and lon value of a user visited places. I was able to export the data to tableau and plot it on the map but I want to find the name of place for each pair of lat and lon.
One solution could be, if I get a table of three columns (Lat, Lon, Place) then I can join it with my user data table to find the name of a place at a given Lat and Lon.
My question is, do we have a ready made table with the above three columns which I can import in my SQL-Server? I am interested in places of UK or London. Is there any other approach to achieve it?
You can get this from the Ordinance Survey which should get you lat, long, postcode;
https://www.ordnancesurvey.co.uk/business-and-government/products/code-point-open.html
You'll then need another data source to map the postcode to location name (e.g. town, county etc). See the similar post below;
Where can I find a list of all UK _full_ postcodes including street name and their precise coordinates?
It might take a little fiddling about, and you're always going to have the issue with data being a little out of date but it should be good enough.
I wrote an API wrapper in R for postcodes.io, which is a free UK postcode database. Check the original documentation so that you could create an API wrapper in your language of choice. Wrappers in languages other than R are also available.
If you use R, then type you can get the place names in the following way:
if (!require("devtools")) install.packages("devtools")
devtools::install_github("erzk/PostcodesioR")
library(PostcodesioR)
rev_geo <- reverse_geocoding(0.127, 51.507)
It will return a list with extensive information about the latitude and longitude, e.g. wards, NUTS, administrative district, county, parish, consituency, CCG and many more.
There is also a bulk_reverse_geocoding() function which takes several lat and lon inputs.
I am working on a web app with Google Map that I’d like to display a “coverage area”/’ geographical area” by creating a polygon overlay of a given set of geo coordinates/points.
The “coverage area” can consist of thousands of the geo coordinates (Longitude and Latitude data stored in a table in sql server). Ideally, I’d like to calculate the Convex Hull points from the sql server database (2008 R2) so I can pass the results (points) to the Google Map to create the polygon overlay.
The sample here (http://www.geocodezip.com/v3_map-markers_ConvexHull.asp) is exactly what I am looking for, except that I’d like to get the hull points on the right-panel straight from the SQL server if possible. The reason is that I may have to process thousands of the geo coordinates. I’d rather not to retrieve a huge amount of data from the database and then send to the client using JavaScript to calculate the convex hull points.
Any help will be very much appreciated!!!
Thank you.
You didn't mention what version you're on, but there's a built in ConvexHullAggregate starting in SQL 2012 that should do exactly what you're looking for.
Here's an extension of the example in the linked to documentation that gets the coordinates of the corners of the convex hull. It assumes that you have a table of numbers laying around (a pretty useful thing in my experience).
with cte as (
SELECT City, geography::ConvexHullAggregate(SpatialLocation) AS Hull
FROM Person.Address
WHERE City in ('Ottawa', 'Burnaby')
group by City
)
select City, Number, Edge.Long as Long, Edge.Lat as Lat
from cte
cross apply (
select Number, Hull.STPointN(Number) as Edge
from dbadmin.dbo.Numbers
where Number < Hull.STNumPoints()
) as HullEdges
Hey, so I'm trying to figure out the best way of storing movement paths and then afterwards how they might be queried.
Let me try to explain a bit more. Say I have many cars moving around on a map and I want to determine if and when they're in a convoy. If I store just the paths then I can see that they travelled along the same road, but not if they were there at the same time. I can store the start and end times but that will not take into account the changes in speed of the two vehicles. I can't think of any obvious way to store and achieve this so I thought I'd put the question out there in case there's something I'm missing before trying to implement a solution. So does anyone know anything I don't?
Thanks,
Andrew
Well it depends on what type of movement information you have.
If you have some tables setup like:
Vehicle (Id, Type, Capacity, ...)
MovementPoint(VehicleId, Latitude, Longitude, DateTime, AverageSpeed)
This would allow you to query if two cars going to the same point plus or minus 5 minutes like so:
Select * from Vehicle v INNER JOIN MovementPoint mp on mp.VehicleId = v.Id
WHERE v.Id = #FirstCarID
AND EXISTS
(
SELECT 1 FROM Vehicle v2 INNER JOIN MovementPoint mp2 on mp2.VehicleId = v2.Id
WHERE v2.Id = #SecondCarId
AND mp2.Latitude = mp.Latitude AND mp2.Longitude = mp.Longitude
AND mp2.DateTime BETWEEN DATEADD(minute,-5,mp.DateTime) AND DATEADD(minute,5,mp.DateTime)
)
You could also query for multiple points in common between multiple vehicles with specific time windows.
Also you could make the query check latitude and longitude values are within a certain radius of each other.