I run an Android app which locates OBJECTS with attributes like ID, Name, Owner, Type, Place_ID which are linked to PLACES, on a map. PLACES have attributes like ID, Latitude, Longitude, Opening Hour, Closing Hour,... The data is stored in a MongoDB on Back4App and I want to keep that way. I have one class for OBJECTS and one class for PLACES. The relation between OBJECTS and PLACES is not "a MongoDB relation", it is just a common String field in the OBJECTS and PLACES classes.
In order to allow offline access to the data and to minimize DB server requests, the app synchronizes a local SQLITE database on the device with the MongoDB online database. In the Android App, the queries are passed to the SQLITE DB.
I'm trying to make a website which does the same job as the app, which is displaying filtered data from the MongoDB.
I started with a simple html and javascript website using the Parse SDK, but I'm facing a few difficulties.
A simple query is to list all the OBJECTS in a 50km radius, i.e. I need the OBJECTS and the PLACE where they are located. However, where I could get this easilty with a SELECT...JOIN in SQLITE, I cannot get this information through a simple Parse query because I want to know the OBJECTS too. And I cannot run 2 asynchronous queries in a for loop.
What website architecture and/or languages would you recommend for this type of website ?
How would you recommend to proceed ?
Thanks in advance for your help.
EDIT: ZeekHuge opened my eyes on the bad design of not using pointers. After implementing pointers in my MongoDB, here's the lines of codes which did it for me :
Parse.initialize("", "");
Parse.serverURL = '';
var eiffel = new Parse.GeoPoint(48.858093, 2.294694);
var myScores = '';
var Enseigne = Parse.Object.extend("ENSEIGNE");
var Flipper = Parse.Object.extend("FLIPPER");
var query = new Parse.Query(Flipper);
var innerquery = new Parse.Query(Enseigne);
innerquery.withinKilometers("ENS_GEO",eiffel,500);
query.equalTo("FLIP_ACTIF", true);
query.include("FLIP_ENSPOINT");
query.include("FLIP_MODPOINT");
query.matchesQuery("FLIP_ENSPOINT", innerquery);
query.find({
success: function(results) {
for (var i = 0; i < results.length; i++) {
var object = results[i];
myScores += '<tr><td>' + object.get('FLIP_MODPOINT').get('MOFL_NOM')
+ '</td><td>' + object.get('FLIP_ENSPOINT').get('ENS_NOM')
+ '</td><td>' + object.get('FLIP_ENSPOINT').get('ENS_GEO').latitude
+ '</td><td>' + object.get('FLIP_ENSPOINT').get('ENS_GEO').longitude
+ '</td></tr>';
}
(function($) {
$('#results-table').append(myScores);
})(jQuery);
},
error: function(error) {
alert("Error: " + error.code + " " + error.message);
}
});
Solved by replacing the database keys by pointers and using the Innerquery and include functions. See exemple mentionned in question.
Related
I got the following AngularJS sample code from the document regarding how to add a new record to Firebase. It works fine and the new record can be created successfully.
While I also noticed that from the callback we can get all the siblings with the list variable, which is interesting.
My question is that would there be any performance latency when the list is large, say 100,000 records. Can anyone shed light on this?
var list = $firebaseArray(ref);
list.$add({ foo: "bar" }).then(function(ref) {
var id = ref.key();
console.log("added record with id " + id);
list.$indexFor(id); // returns location in the array
});
$firebaseArray synchronizes a collection from the Firebase Database to your client, for easy display in your Angular views. You should never synchronize more data than the user needs to see, and 100,000 records seems very much beyond what a user should ever see.
If you just want to add a new item, you're better off using the Firebase JavaScript SDK directly:
ref.push({ foo: "bar" }).then(function(ref) {
var id = ref.key();
console.log("added record with id " + id);
});
Since AngularFire is built on top of the Firebase JavaScript SDK, it will pick up this change automatically.
EDIT 1: I just discovered postgis has an ST_AsMVT function built in, which does exactly what I want (I think), so I'm not going to use mapnik at all!
EDIT 2: unfortunately that function isn't yet in a released version of PostGIS, but hopefully it will be within the next few weeks.
Originally posted as a github issue.
When I do something like the following, with logging turned on for my DB, I see that some "metadata" requests are made to postgis, however, no actual data query is ever made.
The requests for metadata are (presumably) needed for the logic relating to field names/types and extent (when not explicitly provided).
var postgis = new mapnik.Datasource({
type: 'postgis',
host: ... etc,
table: 'some_geometry_table',
geometry_field: 'geom',
srid: 4326,
extent: "-180,-85.0511,180,85.0511",
estimate_extent: false,
row_limit: 10 // !! this doesn't seem to do anything
});
var map = new mapnik.Map(256, 256);
var layer = new mapnik.Layer('some_layer');
layer.datasource = postgis;
map.add_layer(layer);
map.render(new mapnik.VectorTile(z, x, y), {}, (err, vtile) => {
if (err) next(err);
var data = vtile.getDataSync({});
var file = path + z + "," + x + "," + y + ".pbf"
console.log(data);
console.log("written: " + file);
fs.writeFileSync(file, data);
next(null);
});
});
Metadata query as seen in postgres logs:
SELECT * FROM some_geometry_table LIMIT 0
What am I doing wrong?
So I have this web-app using angularJS and nodeJS. I don't want to just use localhost to demo my project because it doesn't looks cool at all when I type "node server.js" and then go to localhost.....
Since I intend to use Firebase for the data, I have noticed that Firebase provides hosting. I tried it, but it seems to only host the index.html and not through/using server.js. I have customized files for the server to use/update. So, how can I tell Firebase Hosting to use my server and related files when hosting?
Is it possible to tell Firebase, hey, run "node server.js" to host my index.html?
I'm guessing by the way you are wording the question you want to see this site from "the internet".
Two routes you could go here.
a) Serve your index through Firebase hosting. Firebase only hosts assets. If your Angular app is being served through Node then you will need to change your architecture to be more SPA-ish
SPA-ish would be like an index bootstrap that interacts with the backend purely through API's.
You would host the API server on something more appropriate like through Nodejitsu.
b) Serve the whole thing through something like Nodejitsu (hosting platform) or your very own VM managed by a different kind of hosting company like BuyVM.net.
Another idea, is if your nodejs app is independent of the angularjs app (however they use shared data, and perform operations on that data model) you could separate the two and connect them only via firebase.
Firebase hosting -> index.html and necessary angularjs files.
Locally (your PC) -> server.js which just connects to firebase and trigger on changed data.
I have done this for a few projects and it's a handy way to access the outside world (internet) while maintaining some semblence of security by not opening ports blindly.
I was able to do this to control a chromecast at my house while at a friends house
Here's an example from my most recent project (I'm trying to make a DVR).
https://github.com/onaclov2000/webdvr/blob/master/app.js
var FB_URL = '';
var Firebase = require('firebase');
var os = require('os')
var myRootRef = new Firebase(FB_URL);
var interfaces = os.networkInterfaces();
var addresses = [];
for (k in interfaces) {
for (k2 in interfaces[k]) {
var address = interfaces[k][k2];
if (address.family == 'IPv4' && !address.internal) {
addresses.push(address.address)
}
}
}
// Push my IP to firebase
// Perhaps a common "devices" location would be handy
var ipRef = myRootRef.push({
"type": "local",
"ip": addresses[0]
});
myRootRef.on('child_changed', function(childSnapshot, prevChildName) {
// code to handle child data changes.
var data = childSnapshot.val();
var localref = childSnapshot.ref();
if (data["commanded"] == "new") {
console.log("New Schedule Added");
var schedule = require('node-schedule');
var date = new Date(data["year"], data["month"], data["day"], data["hh"], data["mm"], 0);
console.log(date);
var j = schedule.scheduleJob(date, function(channel, program, length){
console.log("Recording Channel " + channel + " and program " + program + " for " + length + "ms");
}.bind(null, data["channel"], data["program"], data["length"]));
localref.update({"commanded" : "waiting"});
}
});
When I change my "commanded" data at the FB_URL, to "new" (which can be accomplished by angularjs VERY Simply, using an ng-click operation for example) it'll schedule a recording for a particular date and time (not all actually functional at the moment).
I might be late but since 3 years have passed there is an solution available now from Firebase in the form of cloud functions
Its not straight forward but looks promising if one can refactor their code a bit
Here is the javascript object I'm trying to convert to a query string
{$and: [{topic: categoryIds} , {$or :[ {'groups 1': {$ne: ''}}, {groups: $scope.myGroups}]}]};
Basically I'm looking to match a topic that equals a categoryIds and grab documents that have an empty groups array or that the groups array has values and matches one in the array $scope.mygroups
My question is what would be best practice to convert this in an easily parseable format so I can append it to a GET request, and how would you go about parsing it on the express server.
I am using this code in production for querying against a server with a MongoDB backend (using angular and lodash):
.factory('mongoQuery', function() {
return {
fromJson: function(json) {
return JSON.parse(json, fromJsonReviver);
},
toJson: function(object) {
return JSON.stringify(object, toJsonReplacer);
}
};
function fromJsonReviver(key, value) {
var val = value;
if (_.isPlainObject(value)) {
if (_.isNumber(value.$date)) {
val = new Date(0);
val.setUTCMilliseconds(value.$date * 1000);
} else if (_.isString(value.$regexp)) {
var match = /^\/(.*)\/([gimy]*)$/.exec(value.$regexp);
val = new RegExp(match[1], match[2]);
}
}
return val;
}
function toJsonReplacer(key, value) {
var val = value;
if (_.isPlainObject(value)) {
val = _.extend({}, value);
for (var k in value) {
val[k] = toJsonReplacer(k, val[k]);
}
} else if (_.isDate(value)) {
val = {$date: (new Date(value)).valueOf() / 1000};
} else if (_.isRegExp(value)) {
val = {$regexp: value.toString()};
}
return val;
}
})
It includes many of the suggestions mentioned by others in the comments and supports dates and regular expressions.
Other than that, if you need to send the query with a GET request, just use encodeURIComponent like others have mentioned.
Here is a working example: http://plnkr.co/edit/b9wJiUkrHMrDKWFC1Sdd?p=preview
What you're thinking here is basically to redevelop an API server, coupled with some mongodb database, and querying it with some format.
REST is the "best practise" you're looking for. It's a standard that encapsulates some common actions to http ressources.
You should know that you don't need to redevelop an ecosystem based on such a standard. Full-featured REST API servers exist, some are even based on express.js. Loopback and sails.js. These one provide some extra features like
Model abstraction through ORM's, and database-engine agnostism
Automatic REST actions, from database schema or model finition
Advanced querying through "extended REST" ("where", "limit", "order", ...)
Realtime with REST-like websockets
Client side libraries that help you query the server
Some standalone exernal libraries, such as js-data or Restangular can deal with REST server pretty well, and act as a frontend connector to backend
Now, if I had to purely answer your question, and if you realy wanted to go with your solution, I'd just add the mongo query to a where query param on the http call with encodeURIComponent as stated before.
I see jslinq, and see tds js libraries for using node and SQL together... So has anyone ever used those technologies together?
I want to be able to write linq to sql queries in a nodejs app...
I've started with JS variant of LINQ to Entities this week. Check UniMapperJS
Example
var comments = await Comment.getAll()
.where(e => e.author.endsWith("something") || e.author.startsWith("somethingElse"))
.orderByDescending("created")
.limit(10)
.skip(0)
.exec();
But cuz of limitation of JS, it was hard to figure out, how to accept variables in where because that arrow functions are parsed as string. I've solved it by "virtual" variable ($) and list of args as last param.
var comments = await Comment.getAll()
.where(e => e.author.endsWith($) || e.author.startsWith($), filter.endsWith, filter.name)
.orderByDescending("created")
.limit(10)
.skip(0)
.select(e => e.author)
.exec();
Today, JayData http://jaydata.org/ do it.
You can use the new ES6 Arrow Function syntax that looks very similar to C# (with a browser that supports it, like lastest Firefox, or a transpiler, like TypeScript or Traceur):
todoDB.Todos
.filter(todo => todo.Completed == true)
.map(todo => todo.Task )
.forEach(taskName => $('#list')
.append('Task: ' + taskName + ' completed'));
The query will be converted to a SQL Query (select Task from Todos where Completed = true) or a $filter URL parameter (http://.../?$filter=Completed%20eq%201&$select=Task), depending of the data source...
You should checkout the edge.js framework, it connects node.js with .Net. One way would be to use the built in T-SQL support in edge, and then use something like linq.js to manipulate the results.
var getTop10Products = edge.func('sql', function () {/*
select top 10 * from Products
*/});
getTop10Product(null, function (error, products) {
if (error) throw error;
console.log(products);
});
Otherwise, you could setup an EF datacontext in a .Net library and call out to that using Linq
var getTop10Product = edge.func(function () {/*
async (input) => {
using (var db = new ProductContext())
{
//Linq to EF query
var query = from b in db.Products
orderby b.Name
select b;
return query.Take(10).ToList();
}
}
*/});
I came across this question today as I was wondering the same thing, and after a little more searching I came across Squel.
Quoting directly from their website:
//this code
squel.select()
.from("students")
.field("name")
.field("MIN(test_score)")
.field("MAX(test_score)")
.field("GROUP_CONCAT(DISTINCT test_score ORDER BY test_score DESC SEPARATOR ' ')")
.group("name")
);
/* will return this SQL query as a string:
SELECT
name,
MIN(test_score),
MAX(test_score),
GROUP_CONCAT(DISTINCT test_score ORDER BY test_score DESC SEPARATOR ' ')
FROM
students
GROUP BY
name
*/
Hopefully this will help anyone who comes across this question while searching.
Good luck!
PS: <Insert standard disclaimer about not generating SQL queries in the browser...>
If you are creating client-side javascript that will send a SQL query to your database you open a whole can of worms - if the browser can dictate the SQL query, and a user can manipulate what comes from the browser, then you are vulnerable to SQL injection attacks.
Bottom line: don't use this in the browser!
This is not possible, at least if you want to create ad-hoc queries in Javascript. Linq (of any flavor) is a compiler technology. A query using Linq syntax or Linq expressions is handled by the C# or VB compiler, not directly interpreted by the database.
A conventional way to do this would be through a web service in C#, using Linq to fetch and store data, and presenting a clean API to clients. Then the client could consume the service through AJAX calls.