I am working on a desktop application using nim's webgui package, which sort of works like electron in that it renders a gui using HTML + CSS + JS. However, instead of bundling its own browser and having a backend in node, it uses the browser supplied by the OS (Epiphany under Linux/GNOME, Edge under Windows, Safari under iOS) and allows writing the backend in nim.
In that context I am basically writing an SPA in Angular and need to load in the HTML, JS and CSS files at compile-time into my binary.
Reading from a known absolute filepath is not an issue, you can use nim's staticRead method for that.
However, I would like to avoid having to adjust the filenames in my application code all the time, e.g. when a new build of the SPA changes a file name from main.a72efbfe86fbcbc6.js to main.b72efbfe86fbcbc6.js.
There is an iterator in std/os that you can use at runtime called walkFiles and walkPattern, but these fail when used at compileTime!
import std/[os, sequtils, strformat, strutils]
const resourceFolder = "/home/philipp/dev/imagestable/html" # Put into config file
const applicationFiles = toSeq(walkFiles(fmt"{resourceFolder}/*"))
/home/philipp/.choosenim/toolchains/nim-#devel/lib/pure/os.nim(2121, 11) Error: cannot 'importc' variable at compile time; glob
How do I get around this?
Thanks to enthus1ast from nim's discord server I arrived at an answer: using the collect macro with the walkDir iterator.
The walkDir iterator does not make use of things that are only available at runtime and thus can be safely used at compiletime. With the collect macro you can iterate over all your files in a specific directory and collect their paths into a compile-time seq!
Basically you start writing collect-block, which is a simple for-loop that at its end evaluates to some form of value. The collect macro will put them all into a seq at the end.
The end result looks pretty much like this:
import std/[sequtils, sugar, strutils, strformat, os]
import webgui
const resourceFolder = "/home/philipp/dev/imagestable/html"
proc getFilesWithEnding(folder: string, fileEnding: string): seq[string] {.compileTime.} =
result = collect:
for path in walkDir(folder):
if path.path.endswith(fmt".{fileEnding}"): path.path
proc readFilesWithEnding(folder: string, fileEnding: string): seq[string] {.compileTime.} =
result = getFilesWithEnding(folder, fileEnding).mapIt(staticRead(it))
Related
I have a tensorflow.js script/app that runs in Node.js using tfjs-node and Universal Sentence Encoder (USE).
Each Time the script runs, it downloads a 525 MegaByte File (the USE model file).
Is there any way to load the Universal Sentence Encoder Model File from the local file system to avoid downloading such a large file every time I need to run the node.js tensorflow script?
I've noted several similar model loading examples but none that work with Universal Sentence Encoder as it does not appear to have the same type functionality. Below is a stripped down example of a functioning script that downloads the 525 MB file every time it executes.
Any help or recommendations would be appreciated.
const tf = require('#tensorflow/tfjs-node');
const use = require('#tensorflow-models/universal-sentence-encoder');
// No Form of Universal Sentence Encoder loader appears to be present
let model = tf.loadGraphModel('file:///Users/ray/Documents/tf_js_model_save_load2/models/model.json');
use.load().then(model => {
const sentences = [
'Hello.',
'How are you?'
];
model.embed(sentences).then(embeddings => {
embeddings.print(true /* verbose */);
});
});
I've tried several recommendations that appear to work for other models but not Universal Sentence Encoder such as:
const tf = require('#tensorflow/tfjs');
const tfnode = require('#tensorflow/tfjs-node');
async function loadModel(){
const handler = tfnode.io.fileSystem('tfjs_model/model.json');
const model = await tf.loadLayersModel(handler);
console.log("Model loaded")
}
loadModel();
its not a model issue per-say, its a module issue.
model can be loaded any way you want, but the module #tensorflow-models/universal-sentence-encoder implements only a specific internal way on how it loads actual model data.
specifically, it internally uses tf.util.fetch.
solution? use some library (or write your own) to register a global fetch handler that knows how to handle file:// prefixes - if global fetch handler exists, tf.util.fetch will simply just use it.
hint: https://gist.github.com/joshua-gould/58e1b114a67127273eef239ec0af8989
I've found uses of the following, but no documentation for other possible actions using the browserstack_executor:
fileExists
getFileContent
getFileProperties
setSessionStatus
I'm looking for a removeFile or unlinkFile or deleteFile to remove a file that was downloaded by the browser and is now in the way when the next file downloads and gets a (1) added to the filename.
In my selenium test I'm doing something like this:
if driver._is_remote:
action = {"action": "fileExists", "arguments": {"fileName": os.path.basename(self.filepath)}}
if driver.execute_script(f'browserstack_executor:{json.dumps(action)}'):
action = {"action": "getFileContent", "arguments": {"fileName": os.path.basename(self.filepath)}}
content = driver.execute_script(f'browserstack_executor:{json.dumps(action)}')
with open(self.filepath, "wb") as f:
f.write(base64.b64decode(content))
action = {"action": "deleteFile", "arguments": {"fileName": os.path.basename(self.filepath)}}
delete_status = driver.execute_script(f'browserstack_executor:{json.dumps(action)}')
I keep getting invalid action with all of the 3 I've tried so there must be something else to get rid of a file on the machine at browserstack.
I believe 'browserstack_executor' is a custom executor specific to BrowserStack and has a limited set of operations that it can perform.
The supported operations are available in their documentation:
https://www.browserstack.com/docs/automate/selenium/test-file-upload
https://www.browserstack.com/docs/automate/selenium/test-file-download
Hence, operations like removeFile or unlinkFile or deleteFile, cannot be performed, as they are not supported currently and are also not mentioned in the links shared above.
Per the companies support staff, there is no list and unlink is not supported. In order to work around it I've modified the FileExists ExpectedCondition I was using to auto increment the filename after one is pulled from the test system and to use the "next available" name so that my tests can be the same running locally or on browser stack.
I'm in the early stages of developing an app with react-native, and I need a DB implementation for for testing and development. I thought that the obvious choice would be to use simple JSON files included with the source, but the only way I see to load JSON files requires that you know the file name ahead of time. This means that the following does not work:
getTable = (tableName) => require('./table-' + tableName + '.json') // ERROR!
I cannot find a simple way to load files at runtime.
What is the proper way to add test data to a react-native app?
I cannot find a simple way to load files at runtime.
In node you can use import() though I'm not sure if this is available in react-native. The syntax would be something like:
async function getTable(tableName){
const fileName = `./table-${tableName}.json`
try {
const file = await import(fileName)
} catch(err){
console.log(err
}
}
though like I said I do not know if this is available in react-natives javascript environment so ymmv
Unfortunately dynamic import not supported by react-native but there is a way so to do this
import tableName1 from './table/tableName1.json';
import tableName2 from './table/tableName2.json';
then create own object like
const tables = {
tableName1,
tableName2,
};
after that, you can access the table through bracket notation like
getTable = (tableName) => tables[tableName];
I want to parse Swagger data from the JSON I get from {service}/swagger/docs/v1 into dynamically generated .NET class.
The problem I am facing is that different APIs can have different number of parameters and operations. How do I dynamically parse Swagger JSON data for different services?
My end result should be list of all APIs and it's operations in a variable on which I can perform search easily.
Did you ever find an answer for this? Today I wanted to do the same thing, so I used the AutoRest open source project from MSFT, https://github.com/Azure/autorest. While it looks like it's designed for generating client code (code to consume the API documented by your swagger document), at some point on the way producing this code it had to of done exactly what you asked in your question - parse the Swagger file and understand the operations, inputs and outputs the API supports.
In fact we can get at this information - AutoRest publically exposes this information.
So use nuget to install AutoRest. Then add a reference to AutoRest.core and AutoRest.Model.Swagger. So far I've just simply gone for:
using Microsoft.Rest.Generator;
using Microsoft.Rest.Generator.Utilities;
using System.IO;
...
var settings = new Settings();
settings.Modeler = "Swagger";
var mfs = new MemoryFileSystem();
mfs.WriteFile("AutoRest.json", File.ReadAllText("AutoRest.json"));
mfs.WriteFile("Swagger.json", File.ReadAllText("Swagger.json"));
settings.FileSystem = mfs;
var b = System.IO.File.Exists("AutoRest.json");
settings.Input = "Swagger.json";
Modeler modeler = Microsoft.Rest.Generator.Extensibility.ExtensionsLoader.GetModeler(settings);
Microsoft.Rest.Generator.ClientModel.ServiceClient serviceClient;
try
{
serviceClient = modeler.Build();
}
catch (Exception exception)
{
throw new Exception(String.Format("Something nasty hit the fan: {0}", exception.Message));
}
The swagger document you want to parse is called Swagger.json and is in your bin directory. The AutoRest.json file you can grab from their GitHub (https://github.com/Azure/autorest/tree/master/AutoRest/AutoRest.Core.Tests/Resource). I'm not 100% sure how it's used, but it seems it's needed to inform the tool about what is supports. Both JSON files need to be in your bin.
The serviceClient object is what you want. It will contain information about the methods, model types, method groups
Let me know if this works. You can try it with their resource files. I used their ExtensionLoaderTests for reference when I was playing around(https://github.com/Azure/autorest/blob/master/AutoRest/AutoRest.Core.Tests/ExtensionsLoaderTests.cs).
(Also thank you to the Denis, an author of AutoRest)
If still a question you can use Swagger Parser library:
https://github.com/swagger-api/swagger-parser
as simple as:
// parse a swagger description from the petstore and get the result
SwaggerParseResult result = new OpenAPIParser().readLocation("https://petstore3.swagger.io/api/v3/openapi.json", null, null);
I'm trying to generate combined JavaScript and CSS resources into a single file using gulp-concat using something like this:
var concatjs = gulp
.src(['app/js/app.js','app/js/*Controller.js', 'app/js/*Service.js'])
.pipe(concat('app.js'))
.pipe(gulp.dest('build'));
I get a concatted file with this, but the order of the javascript files embedded in the combined output file is random - in this case the controllers are showing up before the initial app.js file, which causes problems when trying to load the Angular app that expects app.js before any of the related resources are loaded. Likewise for CSS resources that get combined end up in random order, and again the order is somewhat important - ie. bootstrap needs to load before the theme and any custom style sheets.
How can I set up the concatenation process so that the order remains intact?
Update
So it turns out the ordering above DOES actually work by explicitly specifying the file order in the array of file specs. So in this case the crucial thing is to list app/js/app.js first, then let the rest of the scripts where order doesn't matter in in any order.
The reason I failed to see this behavior (Duh!) is that Gulp Watch was running and the gulpfile.js update wasn't actually reflected in the output. Restarting gulp did update the script. Neophyte error...
Other Thoughts:
Still wondering though - is this the right place to specify build order? It seems you're now stuffing application logic (load order) into the build script, which doesn't feel right. Are there other approaches to address this?
For an angular application like the one in your example (and it's dependency management), I normally use this kind of syntax: gulp.src(['app\js\app.js', 'app\js\**\*.js']).
You can also use just gulp.src('app\js\**\*.js') if your app.js file is the first one in alphabetic order.
I see your point about moving the load file order into the build script: I had the same feeling till I started using gulp-inject for injecting the unminified files references in my index.html at development time and injecting the bundled, minified and versioned ones in the production index file. Using that glob ordering solution across all my development cycle made so sense to me that i don't think to it anymore.
Finally, a possible solution for this 'ordering smell' can be using browserify but to me it is just complicating the architecture for an angular application: in the end, as you said, you just need that one specific file is called before all the other ones.
For my js i use a particular structure/naming convention which helps. I split it up into directories by feature, where each 'feature' is then treated as a separate encapsulated module.
So for my projects i have,
app/js/
- app.js
- app.routes.js
- app.config.js
/core/
- core.js
- core.controllers.js
- core.services.js
/test/
- .spec.js test files for module here
/feature1/
- feature1.js
- feature1.controllers.js
/feature2/
- feature2.js
- feature2.controllers.js
...
So each directory has a file of the same name that simply has the initial module definition in it, which is all that app.js has in it for the whole app. So for feature1.js
angular.module('feature1', [])
and then subsequent files in the module retrieve the module and add things (controllers/services/factories etc) to it.
angular.module('feature1')
.controller(....)
Anyway, i'll get to the point...
As i have a predefined structure and know that a specific file has to go first for each module, i'm able to use the function below to sort everything into order before it gets processed by gulp.
This function depends on npm install file and npm install path
function getModules(src, app, ignore) {
var modules = [];
file.walkSync(src, function(dirPath, dirs, files) {
if(files.length < 1)
return;
var dir = path.basename(dirPath)
module;
if(ignore.indexOf(dir) === -1) {
module = dirPath === src ? app : dir;
files = files.sort(function(a, b) {
return path.basename(a, '.js') === module ? -1 : 1;
})
.filter(function(value) {
return value.indexOf('.') !== 0;
})
.map(function(value) {
return path.join(dirPath, value);
})
modules = modules.concat(files);
}
})
return modules;
}
It walks the directory structure passed to it, takes the files from each directory (or module) and sorts them into the correct order, ensuring that the module definition file is always first. It also ignores any directories that appear in the 'ignore' array and removes any hidden files that begin with '.'
Usage would be,
getModules(src, appName, ignoreDirs);
src is the dir you want to recurse from
appName is the name of your app.js file - so 'app'
ignoreDirs is an array of directory names you'd like to ignore
so
getModules('app/js', 'app', ['test']);
And it returns an array of all the files in your app in the correct order, which you could then use like:
gulp.task('scripts', function() {
var modules = getModules('app/js', 'app', ['test']);
return gulp.src(modules)
.pipe(concat('app.js'))
.pipe(gulp.dest('build'));
});