Moving files to different directories based on key in dictionary - file

I am new to this, please tell me if the form is not correct.
I am trying to move my images from a file to a new empty files based on their name related to their key in the dictionary I created. Basically their label.
The dictionary is of this form:
print(multivalue_dict)
defaultdict(list,{'cat_a': ['55b85ea15a1536d46b7190ad6fff8ce7.jpg',
'd4684dcdc759dd9cdf41504698d737d8.jpg',
'6325b6870c54cd47be6ebfbffa620ec7.jpg'],
'cat_b': ['55b85ea15a1536d46b7190ad6fff8ce7.jpg',
'd4684dcdc759dd9cdf41504698d737d8.jpg',
'6325b6870c54cd47be6ebfbffa620ec7.jpg'],....}
The desired outcome would be then to have a file cat_a with all the images inside.
I am trying use the path and shutil
import os
import shutil
source_dir = "/Users/Me/Downloads/File/Images"
for key in multivalue_dict:
target_dir = f"/Users/Me/Downloads/File/{key}"
for file in multivalue_dict[key]:
shutil.move(os.path.join(source_dir, file_name), target_dir)
Though, I get the error:
Error Traceback (most recent call last)
/var/folders/gs/0bx6d0ms3jxfgjd6vfkz22l40000gn/T/ipykernel_29606/1312100907.py in
7 if os.path.exists(file):
8 pass
----> 9 shutil.move(os.path.join(source_dir, file_name), target_dir)
/Users/anaconda3/lib/python3.9/shutil.py in move(src, dst, copy_function)
821
822 if os.path.exists(real_dst):
--> 823 raise Error("Destination path '%s' already exists" % real_dst)
824 try:
825 os.rename(src, real_dst)
Error: Destination path '/Users/Me/Downloads/File/cat_a/f54d0c5cbd8e2270742293edf9075cb1.jpg' already exists
It seems that it is able to transfer 1 image to a new file but then stop, because it loops over it again.

Found the solution using iterrows() and pathlib rather than dictionary and shutil.
from pathlib import Path
for (index,row) in df.iterrows():
for label in label_list:
if row.product_cat == label:
p = Path(f'/Users/Me/Downloads/Images/{row.image}')
p.rename(Path(f'/Users/Me/Downloads/Images/{label}/{row.image}'))

Related

Python image_dataset_loader Module Instances are inconsistent

I want to import an image dataset into Numpy arrays with images and labels. I am trying to use the image_dataset_loader to do this and have wrote this so far:
import image_dataset_loader
(x_train, y_train), (x_test, y_test) = image_dataset_loader.load('./data', ['train', 'test'])
I also have my data directory structured as follows:
data
-train
-male
-male_1.jpg
-male_2.jpg
-male_3.jpg
-male_4.jpg
-......
-female
-female_1.jpg
-female_2.jpg
-female_3.jpg
-female_4.jpg
-......
-test
-male
-male_1.jpg
-male_2.jpg
-male_3.jpg
-male_4.jpg
-......
-female
-female_1.jpg
-female_2.jpg
-female_3.jpg
-female_4.jpg
-......
I have formated all my images to be 120x120 and named them exactly as shown above. I have about 56000 files per category. When I run the script above, it throws the following error:
Traceback (most recent call last):
File "main.py", line 33, in <module>
(x_train, y_train), (x_test, y_test) = image_dataset_loader.load('./data', ['train', 'test'])
File "/home/user/anaconda3/envs/AIOS/lib/python3.8/site-packages/image_dataset_loader.py", line 44, in load
raise RuntimeError('Instance shapes are not consistent.')
RuntimeError: Instance shapes are not consistent.
Can someone please help me sort these images into Numpy arrays?
Check the color of your images. Chances are that some of your images may be grayscale.

SavedModel file does not exist when using Tensorflow hub

When trying to use the hub.load function from tensorflow_hub, I get an OSError: SavedModel file does not exist at: error.
The weird thing is that it worked a few days ago, so I don't quite understand why I'm getting this error now.
Code to reproduce:
import tensorflow as tf
import tensorflow_hub as hub
URL = 'https://tfhub.dev/google/universal-sentence-encoder/4'
embed = hub.load(URL)
Specific error received:
OSError Traceback (most recent call last)
<ipython-input-11-dfb80f0299b2> in <module>
1 URL = 'https://tfhub.dev/google/universal-sentence-encoder/4'
----> 2 embed = hub.load(URL)
~/opt/anaconda3/lib/python3.7/site-packages/tensorflow_hub/module_v2.py in load(handle, tags)
100 if tags is None and is_hub_module_v1:
101 tags = []
--> 102 obj = tf_v1.saved_model.load_v2(module_path, tags=tags)
103 obj._is_hub_module_v1 = is_hub_module_v1 # pylint: disable=protected-access
104 return obj
~/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py in load(export_dir, tags)
576 ValueError: If `tags` don't match a MetaGraph in the SavedModel.
577 """
--> 578 return load_internal(export_dir, tags)
579
580
~/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py in load_internal(export_dir, tags, loader_cls)
586 tags = nest.flatten(tags)
587 saved_model_proto, debug_info = (
--> 588 loader_impl.parse_saved_model_with_debug_info(export_dir))
589
590 if (len(saved_model_proto.meta_graphs) == 1 and
~/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/saved_model/loader_impl.py in parse_saved_model_with_debug_info(export_dir)
54 parsed. Missing graph debug info file is fine.
55 """
---> 56 saved_model = _parse_saved_model(export_dir)
57
58 debug_info_path = os.path.join(
~/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/saved_model/loader_impl.py in parse_saved_model(export_dir)
111 (export_dir,
112 constants.SAVED_MODEL_FILENAME_PBTXT,
--> 113 constants.SAVED_MODEL_FILENAME_PB))
114
115
OSError: SavedModel file does not exist at: /var/folders/77/rvfl368x44s51r8dc3b6l2rh0000gn/T/tfhub_modules/063d866c06683311b44b4992fd46003be952409c/{saved_model.pbtxt|saved_model.pb}
So, just deleting that folder and running the hub.load() function again solves the issue
I have tried the above solution, but it didn't work for me...
Here's what worked:
Download the model from tfhub.dev with assets, variables and .pb checkpoint file.
Make sure you import tensorflow_text module!
import tensorflow_text
Specify the downloaded folder path in the hub.load() statement as in:
model =
hub.load("/Users/bilguun/Desktop/universal-sentence-encoder-multilingual-large_3/")
I was using i3d = hub.load(https://tfhub.dev/deepmind/i3d-kinetics-400/1).signatures['default'] as shown here
This method of loading the model worked in that day but after a few days I got the same error: OSError: SavedModel file does not exist at: C:\Users\catal\AppData\Local\Temp\tfhub_modules\092225fb776e28d6d64ac605ab6be03f18dd2027{saved_model.pbtxt|saved_model.pb}
After doing some research, I understood that the saved_model file location (specified in the error) was temporary so even if that folder still exists, there is no more saved_model.pb in it. So I downloaded the model linked here: https://tfhub.dev/deepmind/i3d-kinetics-400/1 and set i3d = hub.load("C:\\absolute_path_to_saved_model_folder").signatures['default'] and it worked.
So using Bilguun's answer really helped in my case.
This kind of error happens because the downloaded model is saved in temporary folder created by the application.
Since it is a temporary folder, the model saved in it may gets deleted or even the folder can be deleted.
When calling the hub.load() command, the programme checks for the temporary folder, if the folder is not found, the program will download the data from internet.
But if the folder is found and the data(or model) is not there, instead of downloading from the internet, it raises an error.
It can be resolved by deleting the temporary folder completely.
In Mac the temporary folder can be accessed by the command "open $TMPDIR" on the terminal( reference : https://osxdaily.com/2018/08/17/where-temp-folder-mac-access/).
The name of the folder can be get from the raised error and can be deleted easily .

Failed to compile

I'm creating a new component called "counters" but I'm getting this error:
./src/components/counters.jsx
Cannot find file: 'Counter.jsx' does not match the corresponding name on disk: '.\src\components\counter.jsx'.
Cannot find file: 'Counter.jsx' does not match the corresponding name on disk: '.\src\components\counter.jsx'
In this error, it is clear that either the file you are trying to import is unavailable or you have misspelled the file name in import. Also paths and files are case-sensitive. So the error is in either of the two Counter.jsx or .\src\components\counter.jsx

Issue with pixel_array using pydicom on python 3.x

I'm using pydicom (installed with pip3, on python 3.7, using Idle) and I need to access pixel_array values.
I just copy-paste the example provided into the documentation and this leads to two errors:
first is about the get_testdata_files operation, which is not working because
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license()" for more information.
>>>
==================== RESTART: D:\OneDrive\Desktop\test.py ====================
None
Traceback (most recent call last):
File "D:\OneDrive\Desktop\test.py", line 8, in <module>
filename = get_testdata_files("bmode.dcm")[0]
IndexError: list index out of range
I have solved this not using this operation.
second is about the pixel_array and I'm not so able to decode what is wrong, but it seems like the pixel_array is not populated. However I'm able to access other fields in the dataset and the file can be displayed (using ImageJ for example).
==================== RESTART: D:\OneDrive\Desktop\test.py ====================
None
Filename.........: bmode.dcm
Storage type.....: 1.2.840.10008.5.1.4.1.1.3.1
Patient's name...: Femoral trombenarterectomy, Case Report:
Patient id.......: Case Report 1
Modality.........: US
Study Date.......: 20110824
Image size.......: 768 x 1024, 27472108 bytes
Slice location...: (missing)
Traceback (most recent call last):
File "D:\OneDrive\Desktop\test.py", line 38, in <module>
plt.imshow(dataset.pixel_array, cmap=plt.cm.bone)
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\pydicom\dataset.py", line 949, in pixel_array
self.convert_pixel_data()
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\pydicom\dataset.py", line 895, in convert_pixel_data
raise last_exception
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\pydicom\dataset.py", line 863, in convert_pixel_data
arr = handler.get_pixeldata(self)
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\pydicom\pixel_data_handlers\pillow_handler.py", line 188, in get_pixeldata
UncompressedPixelData.extend(decompressed_image.tobytes())
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\PIL\Image.py", line 746, in tobytes
self.load()
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\PIL\ImageFile.py", line 261, in load
raise_ioerror(err_code)
File "C:\Users\marcl\AppData\Local\Programs\Python\Python37\lib\site-packages\PIL\ImageFile.py", line 58, in raise_ioerror
raise IOError(message + " when reading image file")
OSError: broken data stream when reading image file
Here is my code:
import matplotlib.pyplot as plt
import sys
import pydicom
import numpy
from pydicom.data import get_testdata_files
print(__doc__)
#filename = get_testdata_files("bmode.dcm")[0]
filename = "bmode.dcm"
dataset = pydicom.dcmread(filename)
# Normal mode:
print()
print("Filename.........:", filename)
print("Storage type.....:", dataset.SOPClassUID)
print()
pat_name = dataset.PatientName
display_name = pat_name.family_name + ", " + pat_name.given_name
print("Patient's name...:", display_name)
print("Patient id.......:", dataset.PatientID)
print("Modality.........:", dataset.Modality)
print("Study Date.......:", dataset.StudyDate)
if 'PixelData' in dataset:
rows = int(dataset.Rows)
cols = int(dataset.Columns)
print("Image size.......: {rows:d} x {cols:d}, {size:d} bytes".format(
rows=rows, cols=cols, size=len(dataset.PixelData)))
if 'PixelSpacing' in dataset:
print("Pixel spacing....:", dataset.PixelSpacing)
# use .get() if not sure the item exists, and want a default value if missing
print("Slice location...:", dataset.get('SliceLocation', "(missing)"))
# plot the image using matplotlib
plt.imshow(dataset.pixel_array, cmap=plt.cm.bone)
plt.show()
Could you help me to solve these two errors and access pixel_array values?
Don't hesitate to give me some advices /remarks/...
Thanks!
Hi Marc welcome to SO!
Your first error means that the get_testdata_files returns an empty list, so your file is not found. Have a look at the pydicom source, it shows that a search is performed in [DATA_ROOT]/test_files. Is your file located in that path?
Your second error is related to PIL and that can be quite difficult to debug and fix. First try to read the pixel_array from a dataset created from one of the supplied test files. If that works, your problem is probably that PIL cannot handle the specific encoding of your image data. You want to install and use GDCM instead of PIL to see if that solves the problem. Another user has had a similar issue as you, GDCM solved the problem. It can be a bit of a headache to get working unfortunately. Or have a look at this page, it shows some other alternatives on viewing the image data.

Getting error "ImportError: Could not find 'input_readers' on path 'map reduce'" trying to start mapReduce job

I'm getting this error... "ImportError: Could not find 'input_readers' on path 'map reduce'" when trying to Run my map reduce job via the http://localhost:8080/mapreduce launcher page.
It looks like my problem is similar to this post, AppEngine mapper API import error. Unfortunately, no definitive answers were given.
I've simplified it down to this tiny testmapreduce.py:
from google.appengine.ext import db
class TestEntity(db.Model):
value = db.StringProperty()
def mapperhandler(test):
print test.value
return
And my mapreduce.yaml:
mapreduce:
- name: Simplest MapReduce
mapper:
handler: testmapreduce.mapperhandler
input_reader: mapreduce.input_readers.DatastoreInputReader
params:
- name: entity_kind
default: testmapreduce.TestEntity
One possible clue is the presence of __init__.py has no effect (whether in the project root, the mapreduce directory, or both). I'm sure I'm making a beginner mistake, but over the last couple of days I have read every bit of documentation, and all the examples I can find. Thanks.
UPDATE:
I get the same error trying to invoke it via...
control.start_map(
"Give awards",
"testmapreduce.mapperhandler",
"mapreduce.input_readers.DatastoreInputReader",
{"entity_kind": "testmapreduce.TestEntity"},
shard_count=10)
UPDATE:
As requested, the stack trace -- let me know what else would be helpful...
ERROR 2011-10-16 17:09:27,216 _webapp25.py:464] Could not find 'input_readers' on path 'mapreduce'
Traceback (most recent call last):
File "/Develop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/webapp/_webapp25.py", line 701, in __call__
handler.get(*groups)
File "/Users/lc/PycharmProjects/mrtest/testmapreduce.py", line 22, in get
shard_count=10) # handle web form test interface
File "/Users/lc/PycharmProjects/mrtest/mapreduce/control.py", line 94, in start_map
transactional=transactional)
File "/Users/lc/PycharmProjects/mrtest/mapreduce/handlers.py", line 811, in _start_map
mapper_input_reader_class = mapper_spec.input_reader_class()
File "/Users/lc/PycharmProjects/mrtest/mapreduce/model.py", line 393, in input_reader_class
return util.for_name(self.input_reader_spec)
File "/Users/lc/PycharmProjects/mrtest/mapreduce/util.py", line 94, in for_name
module = for_name(module_name, recursive=True)
File "/Users/lc/PycharmProjects/mrtest/mapreduce/util.py", line 102, in for_name
short_name, module_name))
ImportError: Could not find 'input_readers' on path 'mapreduce'
INFO 2011-10-16 22:09:27,253 dev_appserver.py:4247] "GET /giveawards HTTP/1.1" 500 -
This problem turned out to be that I was using the 2.7 version of the Python Interpreter in my local environment. When I switched to 2.5, it works fine.

Resources