MongoDB sensors storage strategy - database

I know that this is a broad questions so I'll try to set some boundaries to the domain of this question:
Let's assume that the data can arrive at any time from a source and the period is not fixed between data acquisition.
The data is represented using a generic index (time-based, length-based or index based)
The data payload could miss some "columns" (if we imagine a data chunk as a table)
An example of this scenario could be the following:
{ │ {
"timestamp": t3, │ "timestamp": t1, # t1 < t3
"temperature": 40, │ "temperature": 40,
┌───────────┐ "pressure": 1220 │ "color": "ffffff" ┌───────────┐
│ │ } │ } │ │
│ │ │ │
│ Source 1 ├─────────────────────────────────────────────────────────►│ │
│ │ │ │
│ │ │ │
└───────────┘ │ │
│ │ │
{ │ { │ │
"length_meters": l1,│ "length_meters": l2, │ │
┌───────────┐ "resistivity": 40 │ "resistivity": 40 │ │
│ │ } │ } │ │
│ │ │ MongoDB │
│ Source 2 ├─────────────────────────────────────────────────────────►│ │
│ │ │ │
│ │ │ │
└───────────┘ │ │
│ │
│ │
{ │ │
"index": 1, │ │
┌───────────┐ "description: "lorem" │ │
│ │ } │ │
│ │ │ │
│ Source 3 ├─────────────────────────────────────────────────────────►│ │
│ │ │ │
│ │ └───────────┘
└───────────┘
I've read the official documentation on the Bucket pattern in which they describe this technique which is, in my opinion, far from reality since in the example they suppose to store a single sensor data (temperature, in that case). Furthermore I don't like the solution since is repeating the column (temperature) for every sample of the sensors.
Is there any better and generic way to design the collection(s) to tackle such requirements which uses contained storage size and can enable good performance in terms of data querying?
Samples queries could be the following:
Return pressure and temperature between tx and ty.
Return the maximum temperature between tx and ty.
Return the minimum of resistivity between lx and ly.

Based on the sample data, the buckets could be also like this:
{
sensor_id: 12345,
start_date: ISODate("2019-01-31T10:00:00.000Z"),
end_date: ISODate("2019-01-31T10:59:59.000Z"),
timestamp: [
ISODate("2019-01-31T10:00:00.000Z"),
ISODate("2019-01-31T10:01:00.000Z"),
ISODate("2019-01-31T10:42:00.000Z")
],
temperature: [
40,
40,
42,
]
}
What do you mean by "good performance in terms of data querying"? It heavily depends ont the queries you like to run.

Related

How to make a unique layout for the main page?

I have multiple pages on my web-site and only main page should have a different layout? How can I achieve this?
src/
- page1/ <- LayoutB
- page2/ <- LayoutB
- page3/ <- LayoutB
- index.svelte <- LayoutA
If I use __layout.svelte in the src/ folder it will apply for every page.
Update November 2022
This approach was tested in
npm -v #sveltejs/kit
8.19.2
To make a unique layout for a specific page (or group of pages) you need to use group.
src/
routes/
(unauthenticated)/
welcome/
register/
login/
+page.svelte
+layout.svelte (this layout applies to welcome/, register/ and login/
(authenicated)/
home/
settings/
+page.svelte
+layout.svelte (this layout applies to home/, settings/ and dashboard/
+page.svelte
+layout.svelte (this layout applies to all pages in unauthenticated/ and authenticated/)
I was able to find a solution by using named layouts from SvelteKit
The solution to my problem will look like this
src/
- page1/ <- LayoutB
- page2/ <- LayoutB
- page3/ <- LayoutB
- __layout.svelte (LayoutB)
- __layout-main.svelte (LayoutA)
- index#main.svelte <- LayoutA
I created a layout file with -main suffix and used this name as a reference to the layout in my index#main.svelte file

PlantUML not finding custom stylesheets

I attempted to replicate the directory structure in this tutorial by jerieljan. Unfortunately I seem to be getting the default PlantUML stylings. I have included the expected results vs. the actual results in screenshots below. I am using the Visual Studio Code extension by jebbs with preview. While the style sheets are too long to include here they can be found in the aforementioned linked tutorial. The diagram file is as follows:
# Sequence Diagram Example
#startuml
'General Defaults
!define BASEPATH ..
!include BASEPATH/globals/stylesheet.iuml
'Actor Definitions
autonumber 1 "0:"
title Sequence Diagram
A -> B: Perform Action
B --> A: (Response)
#enduml
The directory structure is the same as in the tutorial and is as follows:
$ tree
.
├── globals
│ ├── style-presets.iuml
│ └── stylesheet.iuml
└── diagrams
├── example-sequence.puml
├── etc...
Expected Diagram
Actual Diagram

Blocks of type “azure_blob_fs_location” are not expected here

I am currently deploying Azure Data Factory IaC with Terraform and DevOps Pipelines. When trying to deploy a new Delimited Text Dataset, I run into the following error:
│ Error: Unsupported block type
│
│ on ds_test.tf line 7, in resource “azurerm_data_factory_dataset_delimited_text” “test_dataset”:
│ 7: azure_blob_fs_location {
│
│ Blocks of type “azure_blob_fs_location” are not expected here.
│
##[error]Bash exited with code ‘1’.
This is my .tf file:
resource "azurerm_data_factory_dataset_delimited_text" "test_dataset" {
name = "test_dataset"
resource_group_name = "test-rsg"
data_factory_name = "test-adf"
linked_service_name = "AzureDataLakeStorage1"
azure_blob_fs_location {
file_system = "csv-dump-demo"
path = ""
filename = "personal_customer_data.csv"
}
column_delimiter = ","
row_delimiter = "\r\n"
encoding = "UTF-8"
quote_character = "\""
escape_character = "\\"
first_row_as_header = true
null_value = "NULL"
}
The Terraform documentation for Delimited Text Dataset states, that exactly one of the following location blocks need to be defined, in order to make the Dataset work:
azure_blob_fs_location
azure_blob_storage_location
http_server_location
Why is Terraform plan telling me that it is a unsupported block type? Am I missing something?
Seems like the Terraform documentation is deprecated because when I removed the block and tried to deploy the Data Set again, Terraform apply gave me the following output:
│ Error: One of http_server_location, azure_blob_storage_location must be specified to create a DataFactory Delimited Text Dataset
After I tried it with the use of azure_blob_storage_location instead of azure_blob_fs_location it worked. Maybe there are only two location blocks available anymore and the documentation is not up to date.

How to get the image resource permalink in shortcode?

I'm trying to create a shortcode where I need to read a resource property. This is my shortcode lbimg.html:
{{ $img := $.Page.Resources.GetMatch (.Get "name")}}
{{$img.RelPermalink}}
This is how I use it:
{{< lbimg name="/images/test.png" >}}
This is what I'm getting when building the site:
failed to render shortcode "lbimg": failed to process shortcode: "path_to_blog/layouts/shortcodes/lbimg.html:2:6": execute of template failed: template: shortcodes/lbimg.html:2:6: executing "shortcodes/lbimg.html" at : nil pointer evaluating resource.Resource.RelPermalink
I don't understand what does that mean, why the resource is nil?
I'm using Hugo 0.59.1.
The content structure is as follows:
blog
├── content
│ └── post
│ └── test_post
│ ├── images
│ │ └── test.png
│ └── index.md
└── ...
Can you try this:
{{< lbimg name="images/test.png" >}}
My error was similar but my problem was not naming the page file index.md

AngularJs App Structure: minify javascript order and loading of partials/templates

I'm trying to use the app structure recommended by Angular folks: https://docs.google.com/document/d/1XXMvReO8-Awi1EZXAXS4PzDzdNvV6pGcuaF4Q9821Es/pub
It works great in development when all partials/templates are being loaded by relative path from a component's folder.
When I try to prepare my app for production and minify/concat all .js into one file I get 2 issues:
Output file still has relative path to my templates and partials, which are obviously not correct anymore.
Hot to control order of components/modules concatenation to guarantee that all component will be combined in correct order. (Can I achive this without tools like AMD/CommonJs)
some thoughts from me.
I have the following structure for my components:
├── src/scripts/components
│ ├── example
│ │ ├── example.js
│ │ ├── example.controllers.js
│ │ ├── example.directives.js
│ │ ├── example.filters.js
│ │ └── example.services.js
│ ├── address
│ │ ├── address.js
│ │ ├── address.controllers.js
│ │ └── address.filters.js
│ ├── costs
…
I use gulp to build the following structure:
├── inc/scripts/components
│ ├── example.js
│ ├── address.js
│ ├── costs.js
…
To get this structure I use gulp-flatten and gulp-tap. Here is my task:
// Components
gulp.task('scripts-components', function() {
return gulp.src(['src/scripts/components/**/*', '!**/*.md'])
.pipe(jshint('.jshintrc'))
.pipe(jshint.reporter('jshint-stylish'))
.pipe(flatten())
.pipe(tap(function (file) {
if (file.stat.isDirectory()) {
var name = file.relative + '.js';
return gulp.src([file.path + '/' + name, file.path + '/' + file.relative + '.*.js'])
.pipe(concat(name))
.pipe(gulp.dest('inc/scripts/components/'))
.pipe(uglify())
.pipe(rename({ suffix: '.min'}))
.pipe(gulp.dest('inc/scripts/components/'));
}
}));
});
May it's helpful for you or anyone other.
Ciao
Ralf
I found uglifying/minifying my AngularJS code caused a few issues, ng-min seemed to sort alot of these: https://www.npmjs.org/package/grunt-ngmin
Here's an snippet of my scripts tasks that build all my angular related code.
gulp.task('scritps', function () {
gulp.src(config.src.js + '/app.js')
.pipe(ngmin())
.pipe(uglify({ mangle: false }) : gutil.noop())
.pipe(gulp.dest(config.dest.js));
});

Resources