I started exploring Overpass Turbo and Mapbox with hopes of building my travel app. I can query some data in OT and get towns or islands, no problem, I understand the whole process of querying and exporting as Geojson.
But for learning purposes, I always do queries within a small area so I don't get too much data back.
Also, various resources mention that OSM data for the whole planet is huge, like here: https://wiki.openstreetmap.org/wiki/Downloading_data it says: The entire planet is a huge amount of data. Start with a regional extract to make sure your setup works properly. Common tools like Osmosis or various import tools for database imports and converters take hours or days to import data, depending largely on disk speed.
But when I go to apps like AllTrails, Maps.me or Mapbox, they seem to be showing a huge amount of data, definitely the main POIs.
here's an example screenshot from All Trails
Can someone briefly explain how is this done then? Do they actually download all of data? Or little by little depending on the current bounding box. Any info I can research further, I'd appreciate it!
Thanks
P.S. I am hoping to build my app with Node.js, if that makes a difference.
Several reasons:
They don't always display everything. You will always only see a limited region, never the whole world in full detail. If you zoom in, you will see a smaller region but with more details. If you zoom out, you will see a larger region but with reduced details (less or no POIs, smaller roads and waterways disappear etc.).
They don't contain all the available data. OSM data is very diverse. OSM contains roads, buildings, landuse, addresses, POI information and much more. For each of the mentioned elements, there is additional information available. Roads for instance have maxspeed information, lane count, surface information, whether they are lit and if they have sidewalks or cycleways etc. Buildings may have information about the number of building levels, the building color, roof shape and color and so on. Not all of these information are required for the apps you listed and thus can be removed from the data.
They perform simplifications. It isn't always necessary to show roads, buildings, waterways and landuse in full detail. Instead, special algorithms reduce the polygon count so that the data becomes smaller while keeping sufficient details. This is often coupled with the zoom level, i.e. roads and lakes will become less detailed if zoomed out.
They never ship the whole world offline. Depending on the app, the map is either online or offline available, or both. If online, the server has to store the huge amount of data, not the client device. If offline, the map is split into smaller regions that can be handled by the client. This usually means that a certain map only covers a state, a certain region or sometimes a city but rarely a whole country except for smaller countries. If you want to store whole countries offline you will need a significant amount of data storage.
They never access OSM directly. All apps and websites that display OSM maps don't obtain this information live from OSM. Instead, they either already have a local database containing the required data. This database is periodically updated from the main OSM database via planet dumps. Or they use a third-party map provider (such as MapBox from your screenshot) to display a base map with layers on top. In this case they don't have to store much information on their server, just the things they want to show on top of OSM.
None of the above is specifically for OSM. You will find similar mechanisms in other map apps and for other map sources.
I have been working on a project using Azure Indoor Maps. I started to use the Azure Maps Web SDK. I have looked for a way to loop to all features that are loaded automatically by the SDK, without making a request to WFS API https://learn.microsoft.com/en-us/rest/api/maps/v2/wfs/get-feature.
As I see the map loaded, I think that this information should be accessible directly by SDK, and I do not need to create another request. But maybe I am wrong.
I have found a method that does something similar to what I need getRenderedShapes but it only returns the features that are visible when the method is called, and I need all the features in the indoor map or in one floor.
Does anybody know if this is possible? On one side I think should be something similar to getRenderedShapes, but on the other side, I think that the front-end only has the visual information and that azure indoor maps use the Vector tile source and are optimized in the back-end and only serve to the front-end the required information.
https://learn.microsoft.com/en-us/azure/azure-maps/web-sdk-best-practices#optimize-data-sources
The Web SDK has two data sources,
GeoJSON source: Known as the DataSource class, manages raw location
data in GeoJSON format locally. Good for small to medium data sets
(upwards of hundreds of thousands of features). Vector tile source:
Known at the VectorTileSource class, loads data formatted as vector
tiles for the current map view, based on the maps tiling system. Ideal
for large to massive data sets (millions or billions of features).
Vector tile source: Known at the VectorTileSource class, loads data
formatted as vector tiles for the current map view, based on the maps
tiling system. Ideal for large to massive data sets (millions or
billions of features).
As you noted, the map SDK only loads the indoor maps via vector tiles which are condensed set of the data set clipped to areas of the view port. This only loads a small subset of the data. This makes it possible to create a large scalable indoor map platform that in theory could support every building in the world in real time. As you noted, the getRenderedShapes function can retrieve data from the vector tiles, but only those that are in the current viewport (plus a small buffer). I believe the only way to get the data as GeoJSON if via the WFS GetFeatures service: https://learn.microsoft.com/en-us/rest/api/maps/v2/wfs/get-features
I'm building a face recognition web app using Flask (verification task, i.e., compare to face embeddings, one from webcam and another from storage. Then, verify if those are the same person). Now, I face the problem that "Should I store registered face embeddings in file system or database system?".
It will always depend on the size of this embeds. If it's represented by a Gaussian distribution of thousands of samples I wouldn't store it in a database since its size can grow too much.
A common solution to this problem, if you want to have a reference to the embeds in the database, is using a mixed approach: Store the results in a file, and save the path to this file in the database.
As a tip, if you proceed to implement this approach, don't store the absolute path of the images on the database, but a relative one, since it will give you flexibility on moving the files around the file-system without messing the data.
You can refer to this complete answer to help you decide.
So what I want to do is store images inside a database and not on the file system. And after storing them, i want to detect whether the image is coloured or black& white, crop the image to a size i specify later, etc.
Also, I don't want to get the image manipulation done via the overlying application like Java, C# or PhP as my database may be indirectly accessed without using any overlying application.
For storing images, I checked the following DBMS's:
Oracle Multimedia(formerly Intermedia) but couldn't really find an image manipulation documentation, also the installation of different modules is rather confusing.
IBM DB2 and even though the image extenders have been discontinued, I have found a suitable third party image processing library, ImageMagick. From whatever I have read this is the most suitable database with the extension for my requirement, but i am a bit skeptical about its usefulness as its pretty old (http://www.ibm.com/developerworks/data/library/techarticle/dm-0504stolze/ <- This was written in 2005)
SQL server with Filestream where I can store images as BLOB, but i am again not sure about how can I fulfill my requirements.
MongoDB with GridFS, and this also helps in storing images but not exactly manipulating.
I am now lost and require a direction where I can proceed so that I am not stuck anymore at choosing a suitable DB. Any alternative DBMS or method to do the things I want are more than welcome.
In software world (and not only there) people are making things that does one thing, but they try to make it in such a way, that this one thing will be done perfectly. It does not make sense much sense to create a database that no only store information, but also be an image editor, video clips enhancer and a text reader.
It is better to make it just a database and if it is so much in need, create 3 other programs for each of this task.
Therefore you will not find a database which will fulfill your requirements, store images in your database if you want so and then calculate your color, resize them on the application layer. If you need to do this often - then store them in database, precalculate and preresize things you need and store them in database as well. Then query based on precalculated values.
Now, I have read these questions which may have a relation with this question: Scalable Image Storage, Large scale image storage, https://serverfault.com/q/95444.
The following things i have found out, before i ask my question:
1. Facebook uses Haystack (something CLOSED-SOURCE to the open-source world) which is very efficient. Its a form of File system storage, engineered for speed and large metadata management.2. Any Operating System has a file limit in directories and may start to perform extremely poorly when this limit is being exceeded.3. Most NoSQL developers, have found it easy to use CouchDB / CouchBase Server to handle images as it handles it as an attachment, glued to a document (record in the database). However, still, this is file system storage.4. HDFS, NFS, ZFS, are all File systems that may make it easy to handle large distributed data. However, at applications like facebook, they could not help5. Any proper form of caching is very essential to highly Image dependent applications6. Some PHP developers (mostly) have used MySQL to keep image meta-data while creating folders and sub-folders (matching the meta-info) on the file system. Each image will have a random hash name in relation to the meta-data in the database to enable fast location on the file system
After understanding these statements and many more others, i have come to realise that its very expensive to keep billions of constantly growing number of Images on the file system. If any one were to use Cloud storage like Amazon S3, it would kill the business because of the high image traffic as well as storage from your application. I have evaluated the use of CouchBase Server, managing images as attachments. However, for an image growing application, this is also a file system storage and i wonder how Couch base would behave if, hundreds/thousands of people are accessing images at the same time. I could use Cloudant/Big Couch which has auto-sharding/load balancing. The main point remains that the NoSQL solution would as well be keeping images on the file system and when the images are being requested for at a high concurrent rate, this might bring the whole service down (images can be heavy).My Thinking I am thinking of managing my images as SVG format. This is because, i think that i can treat this SVG data as text in my storage. Now, most NoSQL databases have a size limit on the document (record) size atleast not greater than 4MB (not sure). This presents a problem, because SVG file can even reach 6-10MB depending on the image. So, i think i cannot use Couch base server for SVG storage. Besides, the nature of the application is such that, the image data keeps growing and never archived/ never removed: and couch base is not good for such data (highly persistent and unchanging data).This brings me back to RDBMS (especially Oracle) which are known for good text compression. If i get SVG data plus its meta data and store it as a BLOB in an Oracle Database, i have a feeling that this could work. I have heard that an Oracle Table can even grow to terabytes, probably with partitioning or some-kind of fragmentation. But the whole point is that, for an oracle table to reach 20GB, containing text, i think this would be a lot of data. Now, my questions arise from all the above findings:1. Why do developers keep choosing File System storage of images as opposed to SVG, which in my (probably naive) thinking, is that SVG can be handled as Text, hence can be compressed, encrypted, digested, split, easilly stored e.t.c. ? 2. What complexities are there when an application works with images entirely as SVG, serving SVG to browsers instead of actually image files ? 3. Which is technically more memory disturbing to a Webserver: Serving images read from file system (.png, .jpg, .gif) and serving images as SVG (probably from a Database, or from a middle tier) especially under heavy loads, an example scenario of Facebook ? 4. SVG seems to not loose quality when rendered under different "Zooms" or Resolutions, why still, haven't developers worked with SVG alot in image dynamic applications ? i mean, is there any known loss of quality in converting from PNG, JPG or GIF to SVG ? 5. Is my view of using RDBMS like Oracle/MySQL Cluster very naive, for storing highly persistent meta-data as well as the persistent SVG data ?
Please highlight, and give your suggestions about large image storage formats. Thanks
EDIT / UPDATE
There are tools like Image Magick which offer command line option for manipulating images. The most important idea i need probably is this: Can CouchBase Server (whether single server or version 2.0 capable of serving Images at "user-experience acceptable performance" or at a "Social Network Scale" ?)
On databases
What is file but a data and what is file system but a database? Records in database, file on file systems, keys and values in your KV-stores - those are all fruits of the same tree.
Plain file systems were developed over decades to serve purposes of delivering files locally - on top of that you can build a distribution model.
Things like HDFS include distribution as part of file system itself but force an unnecessary overhead when you try to work with files locally.
Things like relational databases or KV-stores might help you laying out your diagrams or storing painlessly more bits of metadata but unless they were specifically designed to work as file storage systems - they gonna fail at it.
Picking storage system is all about tradeoffs and it's up to you to figure out what is best solution to your problem. And chances are that your problems are not even close to facebook's problems. Few servers with cdn on top of them and you gonna be fine.
On file format
SVGs won't work for regular pictures, don't even dream about it.
On a large scale you want to do minimum amount of transformations when you accept files: rescale/compress/crop image if it's not fitting your requirements and store it. Unless you're doing some magic on those images you don't want to convert them into different formats or compress them without real need for it.
On a large scale you want you file to be(ordered by priority):
served from client's cache
served from OS cache / memory
served from file system directly
First, I want to mention that your understanding of image file formats may be naive, since you don't provide a lot of details. How do you intend to store (for example) PNG images "as SVG format"?
I can't answer all of your questions, but I'll make the attempt.
"file system or SVG" is a false dichotomy, it's easily possible to store JPG blobs in a database, or SVG files on file-system storage. You can handle any of the bitmap image formats as text too. If you want an example, try opening up a PostScript file with embedded bitmap data. Your question of "why not" implies that the two are interchangeable, and they're typically not. As an example, my company has evaluated a bunch of different file formats for document storage, and we've gone with PDF (shudder) and PS, depending on the situation. We didn't go with SVG for two reasons; firstly while multi-page documents are in the official standard, SVG editors and viewers seem to have choppy support for them. Secondly, SVG presents some complications when being printed in an automated fashion (to demonstrate, try this experiment: whip up an SVG file and an equivalent PostScript file, then try to print both using lp).
I mentioned two already (though if you're dealing with a web-app, neither should bite you since your clients will presumably be using the browsers' rendering engine, and you may not need more than one page). The only other one is browser support, which is, as always, choppy on older editions of IE. You also have to be aware of the font situation; either make sure any fancy typography is treated as a path, or make sure to only use fonts that you know viewers will have access to (for web-apps, CSS3 helps a bit there).
SVGs and other vector/procedural representations tend to be smaller, so I'm inclined to say they'll be easier for a server to handle. This isn't based on any testing, so take it with a grain of salt. Keep in mind that they do tend to consume more resources over at the client end, but that shouldn't be a very big deal in a web situation.
If your image can be expressed as an SVG, yes, very good idea. However, converting arbitrary bitmaps to vector representations is AFAIK an open problem. Some things don't convert well, even manually, and some things are actually larger when expressed as SVGs than as JPGs. For things like business documents, flowcharts or typography, vectors are strictly better (barring the font problem I mention above). Certain types of illustrations do better as vectors, and some do better as rasters. Finally, if you're starting out with a bitmap (say, a photograph), converting it to SVG will either noticeably drop quality, or take a lot of manual time (if it can be done well at all).
This is the one I can't really answer, since I've never built anything to the scale you seem to be aiming at.
I'd suggest storing your images in S3 -- don't worry about rolling your own until the economics force you to. It's much better to worry about things your users care about, than how your blobs are stored.
As far as Couchbase (I'm a cofounder) we see people using it in similar use cases: typically for metadata and image tracking (who owns it, timestamps, tags, basically anything you want to store or query on.) The Couchbase record would then just contain a URL to the actual image stored on S3.
"SVGs won't work for regular pictures, don't even dream about it."
"However, converting arbitrary bitmaps to vector representations is AFAIK an open problem. Some things don't convert well, even manually, and some things are actually larger when expressed as SVGs than as JPGs."
I think both these statements are wrong.
https://sites.google.com/site/jcdsvg/svg_paradoxes.svg
See example three and four. The cat image is saved as a medium resolution png file, which allows the zooming of the image to be high resolution. It is a higher file size then a regular web image, but that is on purpose.
Storing bit-mapped images as SVG is as simple as putting them in a SVG container.