I'm running the code cell below from https://github.com/aws-samples/amazon-sagemaker-stock-prediction/blob/master/notebooks/dbg-custom-rnn.ipynb , on SageMaker Notebook instance. For more information, here is the link to the error I posted on GitHub: https://github.com/aws-samples/amazon-sagemaker-stock-prediction/issues/7.
%%time
Instantiate estimator with container image of artifact and backend EC2 instance(s)
rnn = Estimator(image,
role, 1, 'ml.c5.18xlarge',
output_path=output_location,
base_job_name = base_job_name,
sagemaker_session=session)
rnn.set_hyperparameters(**hyperparameters)
Train the model
rnn.fit(data_location)
estimator_job = rnn.latest_training_job.job_name
model_archive = "{}/{}/output/{}/output/model.tar.gz".format(artifactname,interval,estimator_job)
print("Estimator created at completion of training job {}".format(estimator_job))
And I hit the error below:
Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.
's3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.
2020-08-12 15:58:15 Starting - Starting the training job...
2020-08-12 15:58:17 Starting - Launching requested ML instances......
2020-08-12 15:59:38 Starting - Preparing the instances for training...
2020-08-12 16:00:16 Downloading - Downloading input data...
2020-08-12 16:00:22 Training - Downloading the training image...
2020-08-12 16:01:13 Uploading - Uploading generated training model.2020-08-12 16:01:08.786584: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-08-12 16:01:08.786650: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Starting the training.
Hyperparameters file : {"target_stock": "BMW", "lag": "10", "interval": "D", "batch_size": "4096", "covariate_columns": "StartPrice, MinPrice, MaxPrice", "percent_train": "85.0", "covariate_stocks": "CON, DAI, PAH3, VOW3", "dropout_ratio": "0.1", "num_epochs": "1000", "target_column": "EndPrice", "horizon": "5", "num_units": "256"}
Hyperparameters initialized
Loading data from : /opt/ml/input/data/training/resampled_stockdata.csv
Loading data from : /opt/ml/input/data/training/resampled_stockdata.csv
Training data loaded
100 Stock symbols found.
Records for 65 trading days found.
0-CON#0111-DAI#0112-PAH3#0113-VOW3
Exception during training: invalid syntax (core.py, line 314)
Traceback (most recent call last):
File "/opt/program/train", line 255, in train
traindata.to_csv(os.path.join(model_path, trainfile))
File "/usr/local/lib/python3.5/dist-packages/pandas/core/generic.py", line 3020, in to_csv
formatter.save()
File "/usr/local/lib/python3.5/dist-packages/pandas/io/formats/csvs.py", line 157, in save
compression=self.compression)
File "/usr/local/lib/python3.5/dist-packages/pandas/io/common.py", line 344, in _get_handle
from s3fs import S3File
File "/usr/local/lib/python3.5/dist-packages/s3fs/__init__.py", line 1, in <module>
from .core import S3FileSystem, S3File
File "/usr/local/lib/python3.5/dist-packages/s3fs/core.py", line 8, in <module>
from fsspec import AbstractFileSystem
File "/usr/local/lib/python3.5/dist-packages/fsspec/__init__.py", line 10, in <module>
from .mapping import FSMap, get_mapper
File "/usr/local/lib/python3.5/dist-packages/fsspec/mapping.py", line 2, in <module>
from .core import url_to_fs
File "/usr/local/lib/python3.5/dist-packages/fsspec/core.py", line 314
out[0] = (f"{out[0][1]}://", out[0][1], out[0][2])
^
SyntaxError: invalid syntax
2020-08-12 16:01:19 Failed - Training job failed
UnexpectedStatusException Traceback (most recent call last)
<timed exec> in <module>
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/estimator.py in fit(self, inputs, wait, logs, job_name, experiment_config)
497 self.jobs.append(self.latest_training_job)
498 if wait:
--> 499 self.latest_training_job.wait(logs=logs)
500
501 def _compilation_job_name(self):
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/estimator.py in wait(self, logs)
1114 # If logs are requested, call logs_for_jobs.
1115 if logs != "None":
-> 1116 self.sagemaker_session.logs_for_job(self.job_name, wait=True, log_type=logs)
1117 else:
1118 self.sagemaker_session.wait_for_job(self.job_name)
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in logs_for_job(self, job_name, wait, poll, log_type)
3075
3076 if wait:
-> 3077 self._check_job_status(job_name, description, "TrainingJobStatus")
3078 if dot:
3079 print()
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in _check_job_status(self, job, desc, status_key_name)
2669 ),
2670 allowed_statuses=,
-> 2671 actual_status=status,
2672 )
2673
UnexpectedStatusException: Error for Training job dbg-custom-rnn-D-BMW-2020-08-12-15-58-15-812: Failed. Reason: AlgorithmError: Exception during training: invalid syntax (core.py, line 314)
Traceback (most recent call last):
File "/opt/program/train", line 255, in train
traindata.to_csv(os.path.join(model_path, trainfile))
File "/usr/local/lib/python3.5/dist-packages/pandas/core/generic.py", line 3020, in to_csv
formatter.save()
File "/usr/local/lib/python3.5/dist-packages/pandas/io/formats/csvs.py", line 157, in save
compression=self.compression)
File "/usr/local/lib/python3.5/dist-packages/pandas/io/common.py", line 344, in _get_handle
from s3fs import S3File
File "/usr/local/lib/python3.5/dist-packages/s3fs/__init__.py", line 1, in <module>
from .core import S3FileSystem, S3File
File "/usr/local/lib/python3.5/dist-packages/s3fs/core.py", line 8, in <module>
from fsspec import AbstractFileSystem
File "/usr/local/lib/python3.5/dist-packages/fsspec/__init__.py", line 10, in <module>
from .mapping import FSMap, get_mapper
File "/usr/local/lib/python3.5/dist-packages/fsspec/map
Could anyone explain for me how to solve this error.
Thank you so much!
Rename s3_input to TrainingInput and remove warning message.
May be this will help you https://github.com/aws/sagemaker-python-sdk/pull/1680
Related
I created a small program in Python for reading messages from a Pub/Sub subscription. I am using Python 3.7 and google-cloud-pubsub 1.1.0.
My code is very simple:
from google.cloud import pubsub_v1
from google.auth import jwt
import json
service_account_info = json.load(open("service-account-info.json"))
audience_sub = "https://pubsub.googleapis.com/google.pubsub.v1.Subscriber"
credentials_sub = jwt.Credentials.from_service_account_info(
service_account_info, audience=audience_sub
)
subscriber_ring = pubsub_v1.SubscriberClient(credentials=credentials_sub)
def callback1(message):
print("In callback!!")
print(message.data)
message.ack()
sub_path = "projects/my-project/subscriptions/my-sub"
future = subscriber_ring.subscribe(sub_path, callback=callback1)
future.result()
When the code reaches "future.result()", it hangs there forever and times out 10 minutes later with the error
pubsub 503 failed to connect to all addresses
I already verified that:
Pub/Sub is up and running
My service account has all the needed permissions. I even tried with my personal Google Cloud account (I am the project owner) with the same results.
There are unacked messages in the Topic
My network connection is OK
but I cannot make it work. Any ideas?
EDIT: I got some more info from the exception:
om_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/anaconda3/envs/loadtest/lib/python3.7/site-packages/google/cloud/pubsub_v1/publisher/_batch/thread.py", line 219, in _commit
response = self._client.api.publish(self._topic, self._messages)
File "/usr/local/anaconda3/envs/loadtest/lib/python3.7/site-packages/google/cloud/pubsub_v1/gapic/publisher_client.py", line 498, in publish
request, retry=retry, timeout=timeout, metadata=metadata
File "/usr/local/anaconda3/envs/loadtest/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
return wrapped_func(*args, **kwargs)
File "/usr/local/anaconda3/envs/loadtest/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "/usr/local/anaconda3/envs/loadtest/lib/python3.7/site-packages/google/api_core/retry.py", line 206, in retry_target
last_exc,
File "<string>", line 3, in raise_from
google.api_core.exceptions.RetryError: Deadline of 60.0s exceeded while calling functools.partial(<function _wrap_unary_errors.<locals>.error_remapped_callable at 0x7fa030891a70>,
, metadata=[('x-goog-request-params', 'topic=projects/my-project/subscriptions/my-sub'), ('x-goog-api-client', 'gl-python/3.7.6 grpc/1.26.0 gax/1.16.0 gapic/1.2.0')]), last exception: 503 failed to connect to all addresses
It is likely that there is a firewall rule in place or some network configuration that is disallowing/dropping connections to *.googleapis.com (or specifically pubsub.googleapis.com). You can see an example of this with another Google product.
I'm using pydicom (installed with pip3, on python 3.7, using Idle) and I need to access pixel_array values.
I just copy-paste the example provided into the documentation and this leads to two errors:
first is about the get_testdata_files operation, which is not working because
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license()" for more information.
>>>
==================== RESTART: D:\OneDrive\Desktop\test.py ====================
None
Traceback (most recent call last):
File "D:\OneDrive\Desktop\test.py", line 8, in <module>
filename = get_testdata_files("bmode.dcm")[0]
IndexError: list index out of range
I have solved this not using this operation.
second is about the pixel_array and I'm not so able to decode what is wrong, but it seems like the pixel_array is not populated. However I'm able to access other fields in the dataset and the file can be displayed (using ImageJ for example).
==================== RESTART: D:\OneDrive\Desktop\test.py ====================
None
Filename.........: bmode.dcm
Storage type.....: 1.2.840.10008.5.1.4.1.1.3.1
Patient's name...: Femoral trombenarterectomy, Case Report:
Patient id.......: Case Report 1
Modality.........: US
Study Date.......: 20110824
Image size.......: 768 x 1024, 27472108 bytes
Slice location...: (missing)
Traceback (most recent call last):
File "D:\OneDrive\Desktop\test.py", line 38, in <module>
plt.imshow(dataset.pixel_array, cmap=plt.cm.bone)
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\pydicom\dataset.py", line 949, in pixel_array
self.convert_pixel_data()
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\pydicom\dataset.py", line 895, in convert_pixel_data
raise last_exception
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\pydicom\dataset.py", line 863, in convert_pixel_data
arr = handler.get_pixeldata(self)
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\pydicom\pixel_data_handlers\pillow_handler.py", line 188, in get_pixeldata
UncompressedPixelData.extend(decompressed_image.tobytes())
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\PIL\Image.py", line 746, in tobytes
self.load()
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\PIL\ImageFile.py", line 261, in load
raise_ioerror(err_code)
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\PIL\ImageFile.py", line 58, in raise_ioerror
raise IOError(message + " when reading image file")
OSError: broken data stream when reading image file
Here is my code:
import matplotlib.pyplot as plt
import sys
import pydicom
import numpy
from pydicom.data import get_testdata_files
print(__doc__)
#filename = get_testdata_files("bmode.dcm")[0]
filename = "bmode.dcm"
dataset = pydicom.dcmread(filename)
# Normal mode:
print()
print("Filename.........:", filename)
print("Storage type.....:", dataset.SOPClassUID)
print()
pat_name = dataset.PatientName
display_name = pat_name.family_name + ", " + pat_name.given_name
print("Patient's name...:", display_name)
print("Patient id.......:", dataset.PatientID)
print("Modality.........:", dataset.Modality)
print("Study Date.......:", dataset.StudyDate)
if 'PixelData' in dataset:
rows = int(dataset.Rows)
cols = int(dataset.Columns)
print("Image size.......: {rows:d} x {cols:d}, {size:d} bytes".format(
rows=rows, cols=cols, size=len(dataset.PixelData)))
if 'PixelSpacing' in dataset:
print("Pixel spacing....:", dataset.PixelSpacing)
# use .get() if not sure the item exists, and want a default value if missing
print("Slice location...:", dataset.get('SliceLocation', "(missing)"))
# plot the image using matplotlib
plt.imshow(dataset.pixel_array, cmap=plt.cm.bone)
plt.show()
Could you help me to solve these two errors and access pixel_array values?
Don't hesitate to give me some advices /remarks/...
Thanks!
Hi Marc welcome to SO!
Your first error means that the get_testdata_files returns an empty list, so your file is not found. Have a look at the pydicom source, it shows that a search is performed in [DATA_ROOT]/test_files. Is your file located in that path?
Your second error is related to PIL and that can be quite difficult to debug and fix. First try to read the pixel_array from a dataset created from one of the supplied test files. If that works, your problem is probably that PIL cannot handle the specific encoding of your image data. You want to install and use GDCM instead of PIL to see if that solves the problem. Another user has had a similar issue as you, GDCM solved the problem. It can be a bit of a headache to get working unfortunately. Or have a look at this page, it shows some other alternatives on viewing the image data.
Use the example code from app engine will give an attribute error. The more strange thing is,
When the batch_size is 100, the first fetch will give an error while if it were set to 10, the second fetch will give the error, when the batch_size is 1, the 25th fetch will give the error. Is it due to the problem of remote API?
Python version: 2.7
App engine sdk version: 1.9.6
query = MyModel.all()
entities = query.fetch(100)
while entities:
for entity in entities:
# Do something with entity
query.with_cursor(query.cursor())
entities = query.fetch(100)
error message:
Traceback (most recent call last):
File "migrate.py", line 77, in <module>
entities = query.fetch(batch_size)
File "/home/kamel/Library/google_appengine/google/appengine/ext/db/__init__.py", line 2157, in fetch
return list(self.run(limit=limit, offset=offset, **kwargs))
File "/home/kamel/Library/google_appengine/google/appengine/ext/db/__init__.py", line 2326, in next
return self.__model_class.from_entity(self.__iterator.next())
File "/home/kamel/Library/google_appengine/google/appengine/ext/db/__init__.py", line 1435, in from_entity
entity_values = cls._load_entity_values(entity)
File "/home/kamel/Library/google_appengine/google/appengine/ext/db/__init__.py", line 1413, in _load_entity_values
value = prop.make_value_from_datastore(value)
File "/home/kamel/labola/src/model/properties.py", line 295, in make_value_from_datastore
return pickle.loads(value)
File "/usr/lib/python2.7/pickle.py", line 1382, in loads
return Unpickler(file).load()
File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
File "/usr/lib/python2.7/pickle.py", line 1083, in load_newobj
obj = cls.__new__(cls, *args)
AttributeError: class Reference has no attribute '__new__
I encountered the same issue when trying to unpickle python3 pickles under python2. The problem was linked to new-style classes becoming default in python3. (source)
Solution for me was to replace class AClass: by class AClass(object):
My tests are failing when I execute a script using jybot.bat. On the other hand if i use pybot.bat the test runs well. I dont understand why this is happening. If any one has seen this issue please assist.
C:\>jybot.bat "\Automation\robot_framework\ui_testing\unity_portal\unity_smoke_t
est\01_basic_admin_fuctionalities\01 - 10 - Customer Admin - 01 - User Managemen
t - Users - Add User.txt"
[ ERROR ] Error in file 'C:\Automation\robot_framework\ui_testing\unity_portal\l
ib\resources\resource.txt': Importing test library 'Selenium2Library' failed: Sy
ntaxError: future feature unicode_literals is not defined (selenium.py, line 17)
Traceback (most recent call last):
File "C:\Python27\Lib\site-packages\Selenium2Library\__init__.py", line 2, in
<module>
from keywords import *
File "C:\Python27\Lib\site-packages\Selenium2Library\keywords\__init__.py", li
ne 3, in <module>
from _browsermanagement import _BrowserManagementKeywords
File "C:\Python27\Lib\site-packages\Selenium2Library\keywords\_browsermanageme
nt.py", line 4, in <module>
from selenium import webdriver
File "C:\Python27\Lib\site-packages\selenium\__init__.py", line 16, in <module
>
from selenium import selenium
PYTHONPATH:
C:\Python27\Lib\site-packages\robot\libraries
C:\Python27\Lib\site-packages
C:\jython2.5.2\Lib__classpath____pyclasspath__/
C:\jython2.5.2\Lib\site-packages
C:\CLASSPATH:
C:\jython2.5.2\jython.jar
[ ERROR ] Error in file 'C:\Automation\robot_framework\ui_testing\unity_portal\l
ib\resources\resource_int.txt': Importing test library 'Selenium2Library' failed
: SyntaxError: future feature unicode_literals is not defined (selenium.py, line
17)
Traceback (most recent call last):
File "C:\Python27\Lib\site-packages\Selenium2Library\__init__.py", line 2, in
<module>
from keywords import *
File "C:\Python27\Lib\site-packages\Selenium2Library\keywords\__init__.py", line 3, in
<module>
from _browsermanagement import _BrowserManagementKeywordsFile
"C:\Python27\Lib\site-packages\Selenium2Library\keywords\_browsermanageme
nt.py", line 4, in <module>
from selenium import webdriver
File "C:\Python27\Lib\site-packages\selenium\__init__.py", line 16, in <module
>
from selenium import selenium
PYTHONPATH:
C:\Python27\Lib\site-packages\robot\libraries
C:\Python27\Lib\site-packages
C:\jython2.5.2\Lib
__classpath__
__pyclasspath__/
C:\jython2.5.2\Lib\site-packages
C:\CLASSPATH:
C:\jython2.5.2\jython.jar
==============================================================================
01 - 10 - Customer Admin - 01 - User Management - Users - Add User :: 01) U...
==============================================================================
10 - Customer Admin - 01 - User Management - Users - Add User | FAIL |
No keyword with name 'Set Selenium Timeout' found.
Also teardown failed:
No keyword with name 'Close All Browsers' found.
------------------------------------------------------------------------------
01 - 10 - Customer Admin - 01 - User Management - Users - Add User... | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
Output: C:\output.xml
Log: C:\log.html
Report: C:\report.html
C:\>
Selenium2Library does not work with jython. You should try Selenium2library java port
I'm getting this error... "ImportError: Could not find 'input_readers' on path 'map reduce'" when trying to Run my map reduce job via the http://localhost:8080/mapreduce launcher page.
It looks like my problem is similar to this post, AppEngine mapper API import error. Unfortunately, no definitive answers were given.
I've simplified it down to this tiny testmapreduce.py:
from google.appengine.ext import db
class TestEntity(db.Model):
value = db.StringProperty()
def mapperhandler(test):
print test.value
return
And my mapreduce.yaml:
mapreduce:
- name: Simplest MapReduce
mapper:
handler: testmapreduce.mapperhandler
input_reader: mapreduce.input_readers.DatastoreInputReader
params:
- name: entity_kind
default: testmapreduce.TestEntity
One possible clue is the presence of __init__.py has no effect (whether in the project root, the mapreduce directory, or both). I'm sure I'm making a beginner mistake, but over the last couple of days I have read every bit of documentation, and all the examples I can find. Thanks.
UPDATE:
I get the same error trying to invoke it via...
control.start_map(
"Give awards",
"testmapreduce.mapperhandler",
"mapreduce.input_readers.DatastoreInputReader",
{"entity_kind": "testmapreduce.TestEntity"},
shard_count=10)
UPDATE:
As requested, the stack trace -- let me know what else would be helpful...
ERROR 2011-10-16 17:09:27,216 _webapp25.py:464] Could not find 'input_readers' on path 'mapreduce'
Traceback (most recent call last):
File "/Develop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/webapp/_webapp25.py", line 701, in __call__
handler.get(*groups)
File "/Users/lc/PycharmProjects/mrtest/testmapreduce.py", line 22, in get
shard_count=10) # handle web form test interface
File "/Users/lc/PycharmProjects/mrtest/mapreduce/control.py", line 94, in start_map
transactional=transactional)
File "/Users/lc/PycharmProjects/mrtest/mapreduce/handlers.py", line 811, in _start_map
mapper_input_reader_class = mapper_spec.input_reader_class()
File "/Users/lc/PycharmProjects/mrtest/mapreduce/model.py", line 393, in input_reader_class
return util.for_name(self.input_reader_spec)
File "/Users/lc/PycharmProjects/mrtest/mapreduce/util.py", line 94, in for_name
module = for_name(module_name, recursive=True)
File "/Users/lc/PycharmProjects/mrtest/mapreduce/util.py", line 102, in for_name
short_name, module_name))
ImportError: Could not find 'input_readers' on path 'mapreduce'
INFO 2011-10-16 22:09:27,253 dev_appserver.py:4247] "GET /giveawards HTTP/1.1" 500 -
This problem turned out to be that I was using the 2.7 version of the Python Interpreter in my local environment. When I switched to 2.5, it works fine.