couchdb update design doc - database

I have a nodejs application where i connect to my couchdb using nano with the following script:
const { connectionString } = require('../config');
const nano = require('nano')(connectionString);
// creates database or fails silent if exists
nano.db.create('foo');
module.exports = {
foo: nano.db.use('foo')
}
This script is running on every server start, so it tries to create the database 'foo' every time the server (re)starts and just fails silently if the database already exists.
I like this idea a lot because this way I'm actually maintaining the database at the application level and don't have to create databases manually when I decide to add a new database.
Taking this approach one step further I also tried to maintain my design docs from application level.
...
nano.db.create('foo');
const foo = nano.db.use('foo');
const design = {
_id: "_design/foo",
views: {
by_name: {
map: function(doc) {
emit(doc.name, null);
}
}
}
}
foo.insert(design, (err) => {
if(err)
console.log('design insert failed');
})
module.exports = {
foo
}
Obviously this will only insert the design doc if it doesn't exist. But what if I updated my design doc and want to update it?
I tried:
foo.get("_design/foo", (err, doc) => {
if(err)
return foo.insert(design);
design._rev = doc._rev
foo.insert(design);
})
The problem now is that the design document is updated every time the server restarts (e.g it gets a new _rev on every restart).
Now... my question(s) :)
1: Is this a bad approach for bootstrapping my CouchDB with databases and designs? Should I consider some migration steps as part of my deployment process?
2: Is it a problem that my design doc gets many _revs, basically for every deployment and server restart? Even if the document itself hasn't changed? And if so, is there a way to only update the document if it changed? (I thought of manually setting the _rev to some value in my application but very unsure that would be a good idea).

Your approach seems quite reasonable. If the checks happen only at restarts, this won't even be a performance issue.
Too many _revs can become a problem. The history of _revs is kept as _revs_info and stored with the document itself (see the CouchDB docs for details). Depending on your setup, it might be a bad decision to create unnecessary revisions.
We had a similar challenge with some server-side scripts that required certain views. Our solution was to calculate a hash over the old and new design document and compare them. You can use any hashing function for this job, such as sha1 or md5.
Just remember to remove the _rev from the old document before hashing it, or otherwise you will get different hash values every time.

I tried the md5 comparison like #Bernhard Gschwantner suggested. But I ran into some difficulties because im my case I'd like to write the map/reduce functions in the design documents in pure javascript in my code.
const design = {
_id: "_design/foo",
views: {
by_name: {
map: function(doc) {
emit(doc.name, null);
}
}
}
}
while getting the design doc from CouchDb returns the map/reduce functions converted as strings:
...
"by_name": {
"map": "function (doc) {\n emit(doc.name, null);\n }"
},
...
Obviously md5 comparing does not really work here.
I ended up with the very simple solution by just putting a version number on the design doc:
const design = {
_id: "_design/foo",
version: 1,
views: {
by_name: {
map: function(doc) {
emit(doc.name, null);
}
}
}
}
When I update the design doc, I simply increment the version number and compare it with the version number in database:
const fooDesign = {...}
foo.get('_design/foo', (err, design) => {
if(err)
return foo.insert(fooDesign);
console.log('comparing foo design version', design.version, fooDesign.version);
if(design.version !== fooDisign.version) {
fooDesign._rev = design._rev;
foo.insert(fooDesign, (err) => {
if(err)
return console.log('error updating foo design', err);
console.log('foo design updated to version', fooDesign.version)
});
}
});

Revisiting your question again: In a recent project I used the great couchdb-push module by Johannes Schmidt. You get conditional updates for free, alongside with many other benefits inherited from its dependency couchdb-compile.
That library turned out to be a hidden gem for me. HIGHLY recommended!

Related

Firebase realtime database - filtering query not on client side Web/React

Quite new to Firebase and I'm facing some issue on the logic on querying/filtering the needed requests.
I have my users stored in the /users and they have a list of projects such as :
users : {
userA : {
projects: {
projectId1: true,
projectId2: true
},
...
}
...
}
And obviously I have the projects as such:
projects: {
projectId1: {
name: "bla"
}
...
}
I want for a user to query all the projects that are in his projects list based on their Ids.
Right now I only succeed to query every single projects of the database and their filter on the client side but obviously this has some serious security implication and loading time as well as I don't want anyone to query all the projects and get them. I can add security rules but then I have access to nothing as I can't query /projects/ anymore but need to be specific.
I'm using https://github.com/CSFrequency/react-firebase-hooks/tree/master/database
and getting the data as such:
const [projects, loading, error] = useListVals(firebase.db.ref("projects"), {
keyField: "uid",
});
And so would like to be able to add an array of projected in this request like where({ id is included in [projectsId]})
You'll need to load each individual project for the user separately, pretty much like a client-side join operation. This is not nearly as slow as you may think, as Firebase pipelines the operations over a single connection.
I don't see anything built into the library you use for such client-side joins, but in regular JavaScript it's something like this:
let userRef = firebase.database().ref('users').child(firebase.auth().currentUser.uid);
userRef.once('value').then((projectKeys) => {
let promises = [];
projectSnapshot.forEach((projectKey) => {
let key = projectKey.key;
let projectRef = firebase.database().ref('projects').child(key);
promises.push(projectRef.once('value');
});
Promise.all(promises).then((snapshots) => {
console.log(snapshots.map(snapshot => snapshot.val()));
});
});

Ways to access firebase storage (photos) via web app

I'm confused as to the appropriate way to access a bunch of images stored in Firebase storage with a react redux firebase web app. In short, I'd love to get a walkthrough of, once a photo has been uploaded to firebase storage, how you'd go about linking it to a firebase db (like what exactly from the snapshot returned you'd store), then access it (if it's not just <img src={data.downloadURL} />), and also how you'd handle (if necessary) updating that link when the photo gets overwritten. If you can answer that, feel free to skip the rest of this...
Two options I came across are either
store the full URL in my firebase DB, or
store something less, like the path within the bucket, then call downloadURL() for every photo... which seems like a lot of unnecessary traffic, no?
My db structure at the moment is like so:
{
<someProjectId>: {
imgs: {
<someAutoGenId>: {
"name":"photo1.jpg",
"url":"https://<bucket, path, etc>token=<token>"
},
...
},
<otherProjectDetails>: "",
...
},
...
}
Going forward with that structure and the first idea listed, I ran into trouble when a photo was overwritten, so I would need to go through the list of images and remove the db record that matches the name (or find it and update its URL). I could do this (at most, there would be two refs with the old token that I would need to replace), but then I saw people doing it via option 2, though not necessarily with my exact situation.
The last thing I did see a few times, were similar questions with generic responses pointing to Cloud Functions, which I will look into right after posting, but I wasn't sure if that was overcomplicating things in my case, so I figured it couldn't hurt too much to ask. I initially saw/read about Cloud Functions and the fact that Firebase's db is "live," but wasn't sure if that played well in a React/Redux environment. Regardless, I'd appreciate any insight, and thank you.
In researching Cloud Functions, I realized that the use of Cloud Functions wasn't an entirely separate option, but rather a way to accomplish the first option I listed above (and probably the second as well). I really tried to make this clear, but I'm pretty confident I failed... so my apologies. Here's my (2-Part) working solution to syncing references in Firebase DB to Firebase Storage urls (in a React Redux Web App, though I think Part One should be applicable regardless):
PART ONE
Follow along here https://firebase.google.com/docs/functions/get-started to get cloud functions enabled.
The part of my database with the info I was storing relating to the images was at /projects/detail/{projectKey}/imgs and had this structure:
{
<autoGenKey1>: {
name: 'image1.jpg',
url: <longURLWithToken>
},
<moreAutoGenKeys>: {
...
}, ...}
My cloud function looked like this:
exports.updateURLToken = functions.database.ref(`/projects/detail/{projectKey}/imgs`)
.onWrite(event => {
const projectKey = event.params.projectKey
const newObjectSet = event.data.val()
const newKeys = Object.keys(newObjectSet)
const oldObjectSet = event.data.previous.val()
const oldKeys = Object.keys(oldObjectSet)
let newObjectKey = null
// If something was removed, none of this is necessary - return
if (oldKeys.length > newKeys.length) {
return null
}
for (let i = 0; i < newKeys.length; ++i) {// Looking for the new object -> will be missing in oldObjectSet
const key = newKeys[i]
if (oldKeys.indexOf(key) === -1) {// Found new object
newObjectKey = key
break
}
}
if (newObjectKey !== null) {// Checking if new object overwrote an existing object (same name)
const newObject = newObjectSet[newObjectKey]
let duplicateKey = null
for (let i = 0; i < oldKeys.length; ++i) {
const oldObject = oldObjectSet[oldKeys[i]]
if (newObject.name === oldObject.name) {// Duplicate found
duplicateKey = oldKeys[i]
break
}
}
if (duplicateKey !== null) {// Remove duplicate
return event.data.ref.child(duplicateKey).remove((error) => error ? 'Error removing duplicate project detail image' : true)
}
}
return null
})
After loading this function, it would run every time anything changed at that location (projects/detail/{projectKey}/imgs). So I uploaded the images, added a new object to my db with the name and url, then this would find the new object that was created, and if it had a duplicate name, that old object with the same name was removed from the db.
PART TWO
So now my database had the correct info, but unless I refreshed the page after every time images were uploaded, adding the new object to my database resulted (locally) in me having all the duplicate refs still, and this is where the realtime database came in to play.
Inside my container, I have:
function mapDispatchToProps (dispatch) {
syncProjectDetailImages(dispatch) // the relavant line -> imported from api.js
return bindActionCreators({
...projectsContentActionCreators,
...themeActionCreators,
...userActionCreators,
}, dispatch)
}
Then my api.js holds that syncProjectDetailImages function:
const SAVING_PROJECT_SUCCESS = 'SAVING_PROJECT_SUCCESS'
export function syncProjectDetailImages (dispatch) {
ref.child(`projects/detail`).on('child_changed', (snapshot) => {
dispatch(projectDetailImagesUpdated(snapshot.key, snapshot.val()))
})
}
function projectDetailImagesUpdated (key, updatedProject) {
return {
type: SAVING_PROJECT_SUCCESS,
group: 'detail',
key,
updatedProject
}
}
And finally, dispatch is figured out in my modules folder (I used the same function I would when saving any part of an updated project with redux - no new code was necessary)

Meteor with query on publication is not reactive

I have a problem with a meteor publication not being reactive when using a query inside it.
Let's say I have many files, and each file has many projects, so I can go to the route:
http://localhost:3000/file/:file_id/projects
And I would like to both display the projects of the selected file and add new projects to it.
I am currently using angularjs, so the controller would look something like this:
class ProjectsCtrl {
//some setup
constructor($scope, $reactive, $stateParams){
'ngInject'
$reactive(this).attach($scope)
let ctrl = this
//retrieve current file id
ctrl.file_id = Number($stateParams.file)
//get info from DB and save it in a property of the controller
ctrl.subscribe('projects', function(){return [ctrl.file_id]}, function(){
ctrl.projects = Projects.find({file_id: ctrl.file_id}).fetch()
})
//function to add a new project
ctrl.addProject = function(){
if(ctrl.projectName){
Meteor.call('projects.insert', {name: ctrl.projectName, file_id: ctrl.file_id }, function(error, result){
if(error){
console.log(error)
}else{
console.log(result)
}
})
}
}
}
}
The publication looks something like this:
Meteor.publish('projects', function(file_id){
return Projects.find({file_id: file_id})
})
The problem is that, if I insert a new project to the DB the subscription doesn't run again, I mean the array stays the same instead of displaying the new projects I am adding.
I got many problems with this as I thought that meteor would work something like: "Oh there is a new project, let's re run the query and see if the publication change, if it does, let's return the new matching documents"... but no.
I have not found a problem similar to mine as every question regardind querys inside the publication is about how to reactively change the query (the file_id in this case) but that is not the problem here as I don't change the file_id unless I go to another route, and that triggers a new subscription.
My current solution is to expose the complete collection of projects and make the query using minimongo, but I don't know if it is a good workaround (many projects exposed uses too much memory of the browser, minimongo is not as fast as mongo... etc, I don't really know).
Your issue is that the Meteor.subscribe call doesn't know that file_id has changed. There's no reactive relationship between that argument and executing the subscription.
To fix this, whenever you are passing criteria in publish-subscribe, you must write a subscription of Collection inside a tracker.
To know more about trackers, Click here.
While I'm unsure how to do this in Angular, consider this simple Blaze template as an example:
Template.Name.onCreated(function(){
this.autorun(() => {
Meteor.subscribe('projects', file_id);
});
});
Whenever file_id changes, a new subscription is triggered, giving you the desired effect of auto pub-sub utility.
I hope this will give you some insight. It could be easily achieved via Angular JS as well.

Correct way to check if DocumentDB object exists

In Microsoft examples I saw two ways to check if DocumentDb object like Database, DocumentCollection, Document etc. exists :
First is by creating a query:
Database db = client.CreateDatabaseQuery().Where(x => x.Id == DatabaseId).AsEnumerable().FirstOrDefault();
if (db == null)
{
await client.CreateDatabaseAsync(new Database { Id = DatabaseId });
}
The second one is by using "try catch" block:
try
{
await this.client.ReadDatabaseAsync(UriFactory.CreateDatabaseUri(databaseName));
}
catch (DocumentClientException de)
{
if (de.StatusCode == HttpStatusCode.NotFound)
{
await this.client.CreateDatabaseAsync(new Database { Id = databaseName });
}
else
{
throw;
}
}
What is the correct way to do this procedure in terms of performance?
You should use the new CreateDatabaseIfNotExistsAsync in the DocumentDB SDK instead of both these approaches, if that's what you're trying to do.
In terms of server resources (request units), a ReadDocumentAsync is slightly more lightweight than CreateDatabaseQuery, so you should use that when possible.
I've just seen the try/catch example in one of the Microsoft provided sample project and it got me baffled, as it is plain wrong: you don't use try/catch for control flow.
Never.
This is just bad code. The new SDK provides CreateDatabaseIfNotExistsAsync which I can only hope doesn't just hide this shit. In older lib just use the query approach, unless you want to get shouted at by whoever is going to review the code.

Removing created records in e2e tests after all tests have run

I have been looking around for suitable ways to 'clean up' created records after tests are run using Protractor.
As an example, I have a test suite that currently runs tests on create and update screens, but there is currently no delete feature, however there is a delete end point I can hit against the backend API.
So the approach I have taken is to record the id of the created record so that in an afterAll I can then issue a request to perform a delete operation on the record.
For example:
beforeAll(function() {
loginView.login();
page.customerNav.click();
page.customerAddBtn.click();
page.createCustomer();
});
afterAll(function() {
helper.buildRequestOptions('DELETE', 'customers/'+createdCustomerId).then(function(options){
request(options, function(err, response){
if(response.statusCode === 200) {
console.log('Successfully deleted customer ID: '+ createdCustomerId);
loginView.logout();
} else {
console.log('A problem occurred when attempting to delete customer ID: '+ createdCustomerId);
console.log('status code - ' + response.statusCode);
console.log(err);
}
});
});
});
//it statements below...
Whilst this works, I am unsure whether this is a good or bad approach, and if the latter, what are the alternatives.
I'm doing this in order to prevent a whole load of dummy test records being added over time. I know you could just clear down the database between test runs, e.g. through a script or similar on say a CI server, but it's not something I\we have looked into further. Plus this approach seems on the face of it simpler, but again I am unsure about the practicalities of such an approach directly inside the test spec files.
Can anyone out there provide further comments\solutions?
Thanks
Well, for what it's worth I basically use that exact same approach. We have an endpoint that can reset data for a specific user based on ID, and I hit that in a beforeAll() block as well to reset the data to an expected state before every run (I could have done it afterAll as well, but sometimes people mess with the test accounts so I do beforeAll). So I simply grab the users ID and send the http request.
I can't really speak to the practicality of it, as it was simply a task that I accomplished and it worked perfectly for me so I saw no need for an alternative. Just wanted to let you know you are not alone in that approach :)
I'm curious if other people have alternative solutions.
The more robust solution is to mock your server with $httpBackend so you don't have to do actual calls to your API.
You can then configure server responses from your e2e test specs.
here's a fake server example :
angular.module('myModule')
.config(function($provide,$logProvider) {
$logProvider.debugEnabled(true);
})
.run(function($httpBackend,$log) {
var request = new RegExp('\/api\/route\\?some_query_param=([^&]*)');
$httpBackend.whenGET(request).respond(function(method, url, data) {
$log.debug(url);
// see http://stackoverflow.com/questions/24542465/angularjs-how-uri-components-are-encoded/29728653#29728653
function decode_param(param) {
return decodeURIComponent(param.
replace('#', '%40').
replace(':', '%3A').
replace('$', '%24').
replace(',', '%2C').
replace(';', '%3B').
replace('+', '%20'));
}
var params = url.match(request);
var some_query_param = decodeURIComponent(params[1]);
return [200,
{
someResponse...
}, {}];
});
});
Then load this script in your test environnement and your done.

Resources