AWS SageMaker Canvas Model usage on Edge device in Python - amazon-sagemaker

This way I wanted to ask a question about AWS Sagemaker. I must confess that I'm quite a newbee to the subject and therefor I was very happy with the SageMaker Canvas app. It works really easy and gives me some nice results.
First of all my model. I try to predict solar power production based on the time (dt), the AWS IoT Thingname (thingname), clouds percentage (clouds) and temperature (temp). I have a csv filled with data measured by IoT things
clouds + temp + dt + thingname => import
2022-08-30 07:45:00+02:00,1.0,0.1577,0.03,***
2022-08-30 08:00:00+02:00,1.0,0.159,0.05,***
2022-08-30 08:15:00+02:00,1.0,0.1603,0.06,***
2022-08-30 08:30:00+02:00,1.0,0.16440000000000002,0.08,***
2022-08-30 08:45:00+02:00,,,0.09,***
2022-08-30 09:00:00+02:00,1.0,0.17,0.12,***
2022-08-30 09:15:00+02:00,1.0,0.1747,0.13,***
2022-08-30 09:30:00+02:00,1.0,0.1766,0.15,***
2022-08-30 09:45:00+02:00,0.75,0.1809,0.18,***
2022-08-30 10:00:00+02:00,1.0,0.1858,0.2,***
2022-08-30 10:15:00+02:00,1.0,0.1888,0.21,***
2022-08-30 10:30:00+02:00,0.75,0.1955,0.24,***
In AWS SageMaker canvas I upload the csv and build the model. All is very easy and when I use the predict tab I upload a CSV where the import column is missing and containing API weather data for some future moment:
2022-09-21 10:15:00+02:00,***,0.1235,1.0
2022-09-21 10:30:00+02:00,***,0.1235,1.0
2022-09-21 10:45:00+02:00,***,0.1235,1.0
2022-09-21 11:00:00+02:00,***,0.1235,1.0
2022-09-21 11:15:00+02:00,***,0.12689999999999999,0.86
2022-09-21 11:30:00+02:00,***,0.12689999999999999,0.86
2022-09-21 11:45:00+02:00,***,0.12689999999999999,0.86
2022-09-21 12:00:00+02:00,***,0.12689999999999999,0.86
2022-09-21 12:15:00+02:00,***,0.1351,0.69
2022-09-21 12:30:00+02:00,***,0.1351,0.69
2022-09-21 12:45:00+02:00,***,0.1351,0.69
From this data SageMaker Canvas predicts some real realistic numbers, from which I assume the model is nicely build. So I want to move this model to my Greengrass Core Device to do predictions on site. I found the best model location using the sharing link to the Junyper notebook.
From reading in the AWS docs I seem to have a few options to run the model on an edge device:
Run the Greengrass SageMaker Edge component and run the model as a component and write an inference component
Run the SageMaker Edge Agent yourself
Just download the model yourself and do your thing with it on the device
Now it seems that SageMaker used XGBoost to create the model and I found the xgboost-model file and downloaded it to the device.
But here is where the trouble started:
SageMaker Canvas never gives any info on what it does with the CSV to format it, so I have really no clue on how to make a prediction using the model.
I get some results when I try to open the same csv file I used for the Canvas prediction, but the data is completely different and not realistic at all
# pip install xgboost==1.6.2
import xgboost as xgb
filename = f'solar-prediction-data.csv'
dpredict = xgb.DMatrix(f'{filename}?format=csv')
model = xgb.Booster()
result = model.predict(dpredict)
print('Prediction result::')
I read that the column order matters, the CSV may not contain a header. But it does not get close to the SageMaker Canvas result.
I also tried using pandas:
# pip install xgboost==1.6.2
import xgboost as xgb
import pandas as pd
filename = f'solar-prediction-data.csv'
df = pd.read_csv(filename, index_col=None, header=None)
dpredict = xgb.DMatrix(df, enable_categorical=True)
model = xgb.Booster()
result = model.predict(dpredict, pred_interactions=True)
print('Prediction result::')
But this last one always gives me following error:
ValueError: DataFrame.dtypes for data must be int, float, bool or category. When
categorical type is supplied, DMatrix parameter `enable_categorical` must
be set to `True`. Invalid columns:dt, thingname
To be honest, I'm completely stuck and hope someone around here can give me some advice or clue on how I can proceed.
Kind regards

Hacor, Canvas autoML creates artifacts, including python feature engineering code and the feature engineering model. You can access them for the best model, under the artifact tab.
Canvas artifacts
Canvas feature engineering python code (.py file) example


Exporting geojson mapbox shapes to mango db?

I am working on a project where a user can outline an area in a map provided by mapbox. I would like to make it so that once the person saves their progress they can log out and log back in to find the area they outlined still there.
I would like to know how I can export a geojson file to my server. If it matters, I am using mango db.
Also, I am using React JS.
You can serialize the geojson feature to string and then store it in a string type column of your DB
JSON.stringify(feature, null, 2);

v2 to v3 Transition for Form Recognizer

Because FR v3.0 is still Preview mode, so I went v2.1 Quickstarts, "Analyze using a Prebuilt model", Navigate to the Form Recognizer Sample Tool.
Using Form Type = "Invoice" to test many size and text including handwriting, very happy with the results, especially returned JSON file structure:
For large/complex image/doc, use pageResults.tables[0].cells based on rowIndex and columnIndex, I can easily piece each row text restoring the whole doc. For small/simple image/doc or when pageResults.tables.length==0, use readResults.lines achieve the same OCR outcome, like one-size-fits-all, perfect!
Next is my own hands-on for the same images, Samples, JavaScript. Because I've been using Invoice only, so I picked recognizeInvoice.js, great sample, easy and simple to follow. Even it's v3 and missing readResults and pageResutls, I'm still able to use invoice.pages[0].tables[0].cells achieve the same result for large/complex image/doc. For small/simple image found 2 issues:
invoice.pages[0].tables.length = 0, so no text values.
only text value is NRT LLC. of invoice.fields.VendorName.value, all other printed text and handwriting returned by v2.1 are gone!
I believe there must be some reasons at MS side for the above changes, for us it means v3 is not backward compatible. And more importantly we wouldn't be able to know if the image fits a model and/or will return something before submitting, even we provide a list of choices of models users may frustrate by extra manual work. At the moment all we can do is switching back to Google. So,
where is the v2.x sample code and when will MS discontinue v2.x?
how does v2.x transit to v3?
Below is my navigation route. Thank you and really appreciate the great work!
It is a bit confusing, but the versions of the #azure/ai-form-recognizer package on NPM are one major version ahead of the Form Recognizer API versions. The preview API version "2021-09-30-preview" (REST API "v3") can be used with Form Recognizer SDK version 4.0.0-beta.2. REST API version v2.1 (GA) is used with SDK version 3.2.0. On the README for #azure/ai-form-recognizer 3.2.0, it explains this:
Note: This package targets Azure Form Recognizer service API version 2.x.
I'm guessing based on what you've said that you are using the latest stable version 3.2.0 of the SDK. When extracting data using a prebuilt or custom model in this version, tables are attached to pages, and pages are attached to Forms, so you can access a table by looking through the forms:
const poller = await client.beginRecognizeInvoices(inputs);
const invoices = await poller.pollUntilDone();
const table = invoices[0].pages[0].tables[0];
If a table appears on a page that isn't associated with any form (no form appears on that page), it can't be accessed using this method. That feature is present in the new beta SDK for the new preview API, but in the current SDK to get all pages (regardless of whether or not they contain a form), you could consider using the beginRecognizeContent method.

where should I store images of my e-commerce react app?

I have an e-commerce react app, so as you know every product has at least three or four images, so to show the image of product in my website i created a folder with a name of " images " inside the public folder so everytime i want to show images of a specific product i can get and show them very simply and for Now this is very very awesome.
The Problem:
as we know each e-commerce website should have an admin where he can publish new products and upload new images, so by the time may be i will have thousands of images in my website.
what is the best practice to store images of my react app ?
do i need to use third party like AWS or Firebase ?
Thank you.
Storing images in the code-base assets folder is not the best option for large number of images. Handling updates, inserts is a big problem. So you have the following options.
Options: Cloud/On Premises
You may store in the cloud like AWS S3
If you want to store on premises, you may store in MongoDB Grids
or even on the File System with file-path stored in the database.
Step Ahead
But going ahead you might need responsive images according to the image placeholder. Example for an image you might need thumbnails of different sizes lets say for listing pages, android apps, iOS apps.You might also need to compress the images in case they are heavy for web purpose.
In that case you may choose to store the images in the desired
resolutions.For this you'll have to store multiple versions on an image. For example product1/original.jpg, product1/compressed.jpg, product1/300x300.jpg etc
You may resize/crop the images on the fly. If you want to write your resizing systems you may write ImageMagick/vips/PIL etc based code.You may try to search for readymade nginx based plugins to server responsive images.
If you do not want to do this resizing stuff yourself you may use image storage services like imgix,cloudinary,akamai that provides CDN + image manipulations.Some of these provide storage+manipulation while some only manipulation.

How do I pull the live weather radar from the new (2020) NWS Weather Radar API?

NWS has depricated their old (pre 2020) weather radar site here.
I am trying to find their new API endpoint for pulling the live weather radar image for a lat/long point. All I've been able to find about it is just JSON feeds for forecast.
There is no way (that I know of) to do this anymore. If there is, it is well hidden. The best I can do is use their endpoint with lat/lon and it will show the closest weather forecast office - which will get you close... but not the actual radar needed.
For my answer I would suggest using MRMS or a national level-2 mosaic. I am not sure how your maps are setup so this may not be ideal. However, you can find them here :
Level-2 Mosaic Files :
MRMS (Multi-Radar-Multi-Sensor) :

Where to store image if the image is used in two different websites

I want to upload the image in my website. The same image will be shown in another website. So this is the scenario where image is uploaded in one website and displayed in both website. These two websites are hosted in two different servers as well as they both have their own database.
I am using Angular JS, Entity Framework, Web API and SQL Server 2014 as backend for both of the website. Currently I am using ngFileUpload to upload the images. Please answer me on below questions:-
Should I upload the image in database(as nvarchar-max) or filesystem(FTP or local web server file system)? I read many articles and get to know that Database retrieval of image has affect on performance but it is more secured. However File System is easy in performance but complex on maintenance like back ups. So I am just not able to decide which to choose among these two as both have pros and cons. Which option will be more suitable to my requirement where same image will be displayed in both website. Please note that there can big images like upto 5 MB uploaded in the application but the number of images will not be huge as compare to any social networking or online shopping site.
How to create different size of images(thumbnail, medium, large etc) automatically upon uploading of image in website? Is there any tool or directive already available in Angular JS to achieve this?
I know my question is broad but I need suggestion to start with my requirements.
Please help.
I use the file system to host my images. If the image is displayed on someone's computer screen, they can use an image capturing software to copy it anyway. Also, while storing them to a database may be more secure, I don't need the extra overhead in my code where a simple url to retrieve the image will suffice.
As for resizing an image using Angular, check out these links: