I have two test pages where the conversion pages are the same for both tests.
Should I just be able to do something like this on the conversion page to include both tests as a goal (notice the 2 '_trackPageview' on the same 'gwo' object):
var _gaq = _gaq || [];
_gaq.push(['gwo._setAccount', 'UA-123456789-2']);
_gaq.push(['gwo._setDomainName', '.mydomain.com']);
_gaq.push(['gwo._trackPageview', '/xxxxxxxx/goal']);
_gaq.push(['gwo._trackPageview', '/yyyyyyyy/goal']);
Related
I've trained a model(object detection) using Azure Custom Vision, and export the model as ONNX,
then import the model to my WPF(.net core) project.
I use ML.net to get prediction from my model, And I found the result has HUGE different compared with the prediction I saw on Custom Vision.
I've tried different order of extraction (ABGR, ARGB...etc), but the result is very disappointed, can any one give me some advice as there are not so much document online about Using Custom Vision's ONNX model with WPF to do object detection.
Here's some snippet:
// Model creation and pipeline definition for images needs to run just once, so calling it from the constructor:
var pipeline = mlContext.Transforms
.ResizeImages(
resizing: ImageResizingEstimator.ResizingKind.Fill,
outputColumnName: MLObjectDetectionSettings.InputTensorName,
imageWidth: MLObjectDetectionSettings.ImageWidth,
imageHeight: MLObjectDetectionSettings.ImageHeight,
inputColumnName: nameof(MLObjectDetectionInputData.Image))
.Append(mlContext.Transforms.ExtractPixels(
colorsToExtract: ImagePixelExtractingEstimator.ColorBits.Rgb,
orderOfExtraction: ImagePixelExtractingEstimator.ColorsOrder.ABGR,
outputColumnName: MLObjectDetectionSettings.InputTensorName))
.Append(mlContext.Transforms.ApplyOnnxModel(modelFile: modelPath, outputColumnName: MLObjectDetectionSettings.OutputTensorName, inputColumnName: MLObjectDetectionSettings.InputTensorName));
//Create empty DataView. We just need the schema to call fit()
var emptyData = new List<MLObjectDetectionInputData>();
var dataView = mlContext.Data.LoadFromEnumerable(emptyData);
//Generate a model.
var model = pipeline.Fit(dataView);
Then I use the model to create context.
//Create prediction engine.
var predictionEngine = _mlObjectDetectionContext.Model.CreatePredictionEngine<MLObjectDetectionInputData, MLObjectDetectionPrediction>(_mlObjectDetectionModel);
//Load tag labels.
var labels = File.ReadAllLines(LABELS_OBJECT_DETECTION_FILE_PATH);
//Create input data.
var imageInput = new MLObjectDetectionInputData { Image = this.originalImage };
//Predict.
var prediction = predictionEngine.Predict(imageInput);
Can you check on the image input (imageInput) is resized with the same size as in the model requirements when you prepare the pipeline for both Resize parameters:
imageWidth: MLObjectDetectionSettings.ImageWidth,
imageHeight: MLObjectDetectionSettings.ImageHeight.
Also for the ExtractPixels parameters especially on the ColorBits and ColorsOrder should follow the model requirements.
Hope this help
Arif
Maybe because the aspect ratio is not preserved during the resize.
Try with an image with the size of:
MLObjectDetectionSettings.ImageWidth * MLObjectDetectionSettings.ImageHeight
And you will see much better results.
I think Azure does preliminary processing on the image, maybe Padding (also during training?), or Cropping.
Maybe during the processing it also uses a moving window(the size that the model expects) and then do some aggregation
I am developing a sheet in google sheet that pulls data from coinmarketcap with a script i've been trying to write.
I am a f. noob # coding.
I use the function importxml (i need to refresh the latest price for each coin, like 100 coins) in this script:
function CryptoRefresher() {
var spreadsheet = SpreadsheetApp.getActive();
var queryString = Math.random();
var link1 = "C";
var xpath = "D";
var destination = "E";
var Direction=SpreadsheetApp.Direction;
var NumeroRighe =spreadsheet.getRange("B"+(spreadsheet.getLastRow()+1)).getNextDataCell(Direction.UP).getRow();
for (var i = 2; i <= NumeroRighe; i++) {
var cellFunction1 = '=IMPORTXML("' + SpreadsheetApp.getActiveSheet().getRange(link1+i).getValue() + '?' + queryString + '", "'+ SpreadsheetApp.getActiveSheet().getRange(xpath+i).getValue() + '")';
SpreadsheetApp.getActiveSheet().getRange(destination+i).setValue(cellFunction1);
}
};
Example Data:
Cell B2 = "bitcoin"
cell C2 = "https://coinmarketcap.com/currencies/Bitcoin"
Cell D2 = "//div[#class='priceValue___11gHJ']"
Cell E2 = is the destination and will receive the bitcoin price
The problem is that it's really slow because it calls 1 coin per time.
Question: Is there a way to send ALL THE COINS REQUESTS in 1 single importxml call?
Like I'd like to collect all the coin names in column C (1 cell of column C has 1 different and unique Coin Name) to collect all the coin names that i am watching and ask for 1 single call to speed up the process?
(Is there a way to create an array, a list of the coin names and do 1 single call to coinmarketcap?)
I really can't figure that out and i hope what i'm asking is clear!
Thank you!
Alessandro
Given the structure of the webpage, it is currently not possible to have a single IMPORTXML call to pull multiple arbitrary currencies from the CoinMarketCap site.
However, they have a convenient API that can do that exactly, please see references below:
CoinMarketCap API / Cryptocurrency
And this should get you started in pulling information from the API:
Pulling Currency Data to Google Sheets
I'd suggest using a dedicated service to retrieve the data, for instance this request will give you the data you need without any parsing or signing up to 3rd party services
=IMPORTDATA("https://cryptoprices.cc/BTC")
Trying to parse a complex web page under active development is just prone to fail at some point.
As alternative, go straight to the source, by signing up to the CoinMarketCap api to get more up to date data. (Already mentioned above) You can sign up for the free tier API (333 req/day) at https://pro.coinmarketcap.com/signup/
Wow! Сryptoprices.cc it’s a great service.
But some cryptocurrencies don’t read clearly.
If you change the formula, add / the price is updated and it is correct.
I am playing with Karate to test one resource that accepts a date that cannot be in the past.
Scenario: Schedule one
Given path '/schedules'
And request read('today_at_19h30.json')
When method post
Then status 201
I have created some JSON files (mostly duplicates but with subtle changes) for each scenario. But I cannot seriously imagine changing the date in all of them each time I want to run my tests.
Using a date in the far future is not a good idea because there is some manual verification and that will force us to click (on next) too many times.
Is there a way to include a variable or an expression in a file?
Thanks
There are multiple ways to "clobber" JSON data in Karate. One way is to just use a JS expression. For example:
* def foo = { a: 1 }
* foo.a = 2
* match foo == { a: 2 }
For your specific use case, I suspect embedded expressions will be the more elegant way to do it. The great thing about embedded expressions is that they work in combination with the read() API.
For example where the contents of the file test.json is { "today": "#(today)" }
Background:
* def getToday =
"""
function() {
var SimpleDateFormat = Java.type('java.text.SimpleDateFormat');
var sdf = new SimpleDateFormat('yyyy/MM/dd');
var date = new java.util.Date();
return sdf.format(date);
}
"""
Scenario:
* def today = getToday()
* def foo = read('test.json')
* print foo
Which results in:
Running com.intuit.karate.junit4.dev.TestRunner
20:19:20.957 [main] INFO com.intuit.karate - [print] {
"today": "2020/01/22"
}
By the way if the getToday function has been defined, you can even do this: { "today": "#(getToday())" }. Which may give you more ideas.
In some contexts, entities are common to a group of apps. For example, I use a list of departments in my institution for at least 4 apps (different projects that cannot and should not be merged in a single app). Another example is the type of employes or even the list of employees.
Is it possible to create an entity accessible to every app in an easy and fast way?
I searched it, but can't find any documentation about this.
Is it related to the dotnet external use?
// the app id
var appId = 42;
// create a simple app object to then access datavar appSimple =
ToSic.SexyContent.Environment.Dnn7.Factory.App(appId);
// example getting all data of content type Tagvar tags =
appSimple.Data["Tag"];
If you are working with razor and just want to access the data in code, you can create an AppDataSource and tell it what App you need. Here's some Pseudo-code:
var otherApp = CreateSource<AppDataSource>();
otherApp.ZoneId = 74;
otherApp.AppId = 203;
// do this after setting the values
var categories = otherApp.Data["Categories"];
I've written a CasperJS script that works very well except that it takes a (very very) long time to scrape pages.
In a nutshell, here's the pseudo code:
my functions to scrape the elements
my casper.start() to start the navigation and log in
casper.then() where I loop through an array and store my links
casper.thenOpen() to open each link and call my functions to scrap.
It works perfectly (and fast enough) for scraping a bunch of links. But when it comes to thousands (right now I'm running the script with an array of 100K links), the execution time is endless: the first 10K links have been scrapped in 3h54m10s and the following 10K in 2h18m27s.
I can explain a little bit the difference between the two 10K batches : the first includes the looping & storage of the array with the 100K links. From this point, the scripts only open pages to scrap them. However, I noticed the array was ready to go after roughly 30 minutes so it doesn't explain exactly the time gap.
I've placed my casper.thenOpen() in the for loop hoping that after each new link built and stored in the array, the scrapping will happen. Now, I'm sure I've failed this but will it change anything in terms of performance ?
That's the only lead I have in mind right now and I'd be very thankful if anyone is willing to share his/her best practices to reduce significantly the running time of the script's execution (shouldn't be hard!).
EDIT #1
Here's my code below:
var casper = require('casper').create();
var fs = require('fs');
// This array maintains a list of links to each HOL profile
// Example of a valid URL: https://myurl.com/list/74832
var root = 'https://myurl.com/list/';
var end = 0;
var limit = 100000;
var scrapedRows = [];
// Returns the selector element property if the selector exists but otherwise returns defaultValue
function querySelectorGet(selector, property, defaultValue) {
var item = document.querySelector(selector);
item = item ? item[property] : defaultValue;
return item;
}
// Scraping function
function scrapDetails(querySelectorGet) {
var info1 = querySelectorGet("div.classA h1", 'innerHTML', 'N/A').trim()
var info2 = querySelectorGet("a.classB span", 'innerHTML', 'N/A').trim()
var info3 = querySelectorGet("a.classC span", 'innerHTML', 'N/A').trim()
//For scraping different texts of the same kind (i.e: comments from users)
var commentsTags = document.querySelectorAll('div.classComments');
var comments = Array.prototype.map.call(commentsTags, function(e) {
return e.innerText;
})
// Return all the rest of the information as a JSON string
return {
info1: info1,
info2: info2,
info3: info3,
// There is no fixed number of comments & answers so we join them with a semicolon
comments : comments.join(' ; ')
};
}
casper.start('http://myurl.com/login', function() {
this.sendKeys('#username', 'username', {keepFocus: true});
this.sendKeys('#password', 'password', {keepFocus: true});
this.sendKeys('#password', casper.page.event.key.Enter, {keepFocus: true});
// Logged In
this.wait(3000,function(){
//Verify connection by printing welcome page's title
this.echo( 'Opened main site titled: ' + this.getTitle());
});
});
casper.then( function() {
//Quick summary
this.echo('# of links : ' + limit);
this.echo('scraping links ...')
for (var i = 0; i < limit; i++) {
// Building the urls to visit
var link = root + end;
// Visiting pages...
casper.thenOpen(link).then(function() {
// We pass the querySelectorGet method to use it within the webpage context
var row = this.evaluate(scrapDetails, querySelectorGet);
scrapedRows.push(row);
// Stats display
this.echo('Scraped row ' + scrapedRows.length + ' of ' + limit);
});
end++;
}
});
casper.then(function() {
fs.write('infos.json', JSON.stringify(scrapedRows), 'w')
});
casper.run( function() {
casper.exit();
});
At this point I probably have more questions than answers but let's try.
Is there a particular reason why you're using CasperJS and not Curl for example ? I can understand the need for CasperJS if you are going to scrape a site that uses Javascript for example. Or you want to take screenshots. Otherwise I would probably use Curl along with a scripting language like PHP or Python and take advantage of the built-in DOM parsing functions.
And you can of course use dedicated scraping tools like Scrapy. There are quite a few tools available.
Then the 'obvious' question: do you really need to have arrays that large ? What you are trying to achieve is not clear, I am assuming you will want to store the extracted links to a database or something. Isn't it possible to split the process in small batches ?
One thing that should help is to allocate sufficient memory by declaring a fixed-size array ie:
var theArray = new Array(1000);
Resizing the array constantly is bound to cause performance issues. Every time new items are added to the array, expensive memory allocation operations must take place in the background, and are repeated as the loop is being run.
Since you are not showing any code, so we cannot suggest meaningful improvements, just generalities.