I am wondering why I am unable to show the loss and accuracy curve in Sagemaker Studio, Trial components chart.
I am using tensorflow's keras API for training.
from sagemaker.tensorflow import TensorFlow
estimator = TensorFlow(
entry_point="sm_entrypoint.sh",
source_dir=".",
role=role,
instance_count=1,
instance_type="ml.m5.4xlarge",
framework_version="2.4",
py_version="py37",
metric_definitions=[
{'Name':'train:loss', 'Regex':'loss: ([0-9.]+'},
{'Name':'val:loss', 'Regex':'val_loss: ([0-9.]+'},
{'Name':'train:accuracy', 'Regex':'accuracy: ([0-9.]+'},
{'Name':'val:accuracy', 'Regex':'val_accuracy: ([0-9.]+'}
],
enable_sagemaker_metrics=True
)
estimator.fit(
inputs="s3://xxx",
experiment_config={
"ExperimentName": "urbansounds-20211027",
"TrialName": "tf-classical-NN-20211027",
"TrialComponentDisplayName": "Train"
}
)
Regex is enabled, and appears to be logging them correctly. Since under the metrics tab, it shows 12 counts for each metric, corresponding to 12 epochs cycle which I specified.
However, the chart is empty. The x-axis is in time here, but it is also empty when I switched to epoch.
tldr: in your entry_point source code sm_entrypoint.sh, you need to explicitly inform the experiment tracker which epoch the metric is associated with, using the log_metric() function.
There are two ways tracker work in SageMaker Experiment Tracker: (1) you log the metric in your entry_point code, and use metric_definitions argument in the estimator to teach SM to parse the metric from the logs, as the way you did it, or, you can (2) explicitly create a Tracker instance inside your entry_point, and invoke the log_metric() method. Apparently only method (2) tells SM Tracker what epoch each metric entry is registered to.
I found the answer from a random video https://youtu.be/gMnkfPztIHU?t=141, after days of search :(
Oh there is also a catch: SageMaker images do not have the Tracker package installed, so if you just have from smexperiments.tracker import Tracker in your entry_point source code, your SM Estimator will complain. So you will need to install sagemaker-experiments for your image, by
create a requirements.txt file that has sagemaker-experiments==0.1.35 in it;
specify the source dir by including source_dir="./dir_that_contains_requirements.txt" in your estimator creation.
Related
based on the aws documentation/example provided here , https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-pipelines/tabular/abalone_build_train_deploy/sagemaker-pipelines-preprocess-train-evaluate-batch-transform.html#Define-a-Transform-Step-to-Perform-Batch-Transformation, a model is created and a batch transform inference can be run on the trained model. it works for this example but if we need a custom inference script, How do we include a custom inference script in the model or model package before we run the batch transformation ?
from sagemaker.transformer import Transformer
from sagemaker.inputs import TransformInput
from sagemaker.workflow.steps import TransformStep
transformer = Transformer(
model_name=step_create_model.properties.ModelName,
instance_type="ml.m5.xlarge",
instance_count=1,
output_path=f"s3://{default_bucket}/AbaloneTransform",
)
step_transform = TransformStep(
name="AbaloneTransform", transformer=transformer, inputs=TransformInput(data=batch_data)
)
You need a "model repacking step".
From the Amazon SageMaker Workflows FAQ:
Model repacking happens when the pipeline needs to include a custom
script in the compressed model file (model.tar.gz) to be uploaded to
Amazon S3 and used to deploy a model to a SageMaker endpoint. When
SageMaker pipeline trains a model and registers it to the model
registry, it introduces a repack step if the trained model output from
the training job needs to include a custom inference script. The
repack step uncompresses the model, adds a new script, and
recompresses the model. Running the pipeline adds the repack step as a
training job.
Basically, you can redefine a sagemaker Model by calling the training output as model_data and pass the inference script as entry_point.
Then sequentially, after training the model, redefine the model by changing the entry_point and on the latter you can use the transformer.
This is an example flow taken from a tested code:
my_model = Model(
image_uri=your_img_uri,
model_data=step_train.properties.ModelArtifacts.S3ModelArtifacts,
role=role,
entry_point='inference_script.py',
name="your_inference_step_name"
)
step_create_model = ModelStep(
name="YourInfName",
step_args=my_model.create(instance_type="ml.m5.xlarge")
)
transformer = Transformer(
model_name=step_create_model.properties.ModelName,
instance_count=your_instance_count,
instance_type=your_instance_type,
output_path=your_path
)
Of course, instead of the generic Model, you can directly use the one that best suits your requirements (e.g. PyTorchModel etc...).
I follow this example to simulate streaming in Spark from a source file. At the end of the example, a function named display is used which is supported only in databricks. I run my code in Jupyter notebook. What is the alternative in Jupyter to get the same output obtained from display function?
screenshoot_of_the_Example
Update_1:
The code:
# Source
sourceStream=spark.readStream.format("csv").\
option("header",True).\
schema(schema).option("ignoreLeadingWhiteSpace",True).\
option("mode","dropMalformed").\
option("maxFilesPerTrigger",1).load("D:/PHD Project/Paper_3/Tutorials/HeartTest_1/").\
withColumnRenamed("output","label")
#stream test data to the ML model
streamingHeart=pModel.transform(sourceStream).select('label')
I do the following:
streamingHeart.writeStream.outputMode("append").\
format("csv").option("path", "D:/PHD \
Project/Paper_3/Tutorials/sa1/").option("checkpointLocation",\
"checkpoint/filesink_checkpoint").start()\
The problem is that the generated files (output files) are empty. What might be the reason behind that?
I solved the problem by changing the checkpoint, as follow.
Project/Paper_3/Tutorials/sa1/").option("checkpointLocation",\
"checkpoint/filesink_checkpoint_1")
I'm trying to load a filtered Wikidata dump with Anzograph using LOAD WITH 'global' <file:wdump-749.nt.gz> INTO GRAPH <WD_749>. The file exists; Anzograph gives out this error:
Error - At Turtle production subject=http://www.wikidata.org/entity/Q144> predicate=http://www.wikidata.org/prop/direct/P1319> file=wdump-749.nt.gz line=3229 details: -34000-01-01T00:00:00Z:Datum is not a datetime, use setting 'load_normalize_datetime' to patch bad data
I've set load_normalize_datetime=true in settings.conf and settings_anzograph.conf inside Anzograph's filesystem, restarted the server, but still can't load the dump. I get the exact same error.
load_normalize_datetime does not take a boolean. Change bad datetimes in loads to this value, e.g. 0001-01-01T00:00:00Z
So instead try setting:
load_normalize_datetime=0001-01-01T00:00:00Z
in your settings.conf, which worked for me on that specific file using the command you listed.
WD_749 has 38,131,614 statements, loaded in 372 seconds on my Thinkpad. It was relatively slow (102k triples per second) to load because it is a single file. If you break it up into smaller pieces (you can do this with the COPY command to dump the graph to a dir:/mydir/wdump-749.nt.gz) it will load in parallel (for me 114 seconds, 335k tps).
I am using bizreview as the theme for my Drupal 7 site. I am using the Feeds module to import thousands of records that are in CSV files into the site. I need to use a geofield to store the locations.
For this I created a field 'Coordinates' in my content type, made it a geofield and set the widget type to latitude/longitude. I can add the locations manually and they do show up in the map, but I just can't import the coordinates with Feeds.
This seems to be an ongoing issue with the geofield/feeds interface (see Drupal issue here). I had the same problem but applied the patch in comment #12 from the aforementioned link which worked.
One suggestion: If the current version of geofield is not the same as the one used in the patch, or if you are running WAMP without Cygwin, I would suggest applying the patch manually by following the directions here, making sure to save a safe backup file in the process. If you haven't worked with patches before, basically all that you (or the patch command) will do for this particular case is add the following lines of code after line 143 in the ./sites/all/modules/geofield/geofield.feeds.inc file (I am working with geofield version 7.x-2.3):
foreach ($field[LANGUAGE_NONE] as $delta => $value) {
if (!empty($value['lat']) && !empty($value['lon'])) {
// Build up geom data.
$field[LANGUAGE_NONE][$delta] = geofield_compute_values($value, 'latlon');
}
}
I have a legacy Asterisk application in C which does authentication of users, routing and billing using MySQL. I have kept it with Asterisk 1.4.21 because none of the CDR data is returned in newer versions of Asterisk.
Apparently there have been some changes in 1.4.22 https://issues.asterisk.org/jira/browse/ASTERISK-13064 that have completely changed the way CDR-s are handled. Unfortunately no helpful information was given on how to properly migrate existing code.
They have changed the order of execution, the 'h' extension is called and the CDR data is reset.
My code:
ast_log(LOG_NOTICE,"Dialing string: '%s'\n", dialstr);
app = pbx_findapp("Dial");
if (app)
res = pbx_exec(chan, app, dialstr);
ast_log(LOG_NOTICE,"Return from pbx_exec '%i', Disposition: '%s'\n", res, ast_cdr_disp2str(chan->cdr->disposition));
Other parts of the code handle chan->cdr->billsec etc, but it always gives 0 values.
After a successful call I always get this log from CLI:
Return from pbx_exec '-1', Disposition: 'NO ANSWER' while the same code works fine on 1.4.21
One solution I heard is to use ast_reset() before Dial but I am not sure how to implement it.
Any help on how to adapt this application?
You can just get DIALSTATUS variable,that is enought for you application and will be supported in future releases.
pbx_builtin_getvar_helper(chan, "DIALSTATUS");