Looping over with open() thows IOError - file

Question

I am iterating over several DB records and writing data from their respective BLOB fields into files:
def build(self, records):
"""
Builds openimmo.anhang
"""
result = None
anh_records = [r for r in records if type(r) == anhaenge]
if not anh_records:
return result
anhang = []
print('RECORDS: ' + str(len(anh_records)))
for anh_record in anh_records:
if anh_record.daten:
__, path = mkstemp()
with open(path, 'wb') as target:
target.write(anh_record.daten)
anh = openimmo.anhang()
anh.anhangtitel = anh_record.anhangtitel
anh.format = 'image/jpeg' #MIMEUtil.getmime(path)
anh.daten = openimmo.daten()
anh.daten.pfad = path
anh.location = id2location.get(anh_record.location)
anh.gruppe = id2gruppe.get(anh_record.gruppe)
anhang.append(anh)
try:
result.validateBinding()
except:
self.log.err('Could not build "anhang": ' + str(result))
if anhang:
result = openimmo.anhaenge()
result.anhang = anhang
return result
This, however produces the following error:
RECORDS: 5
Message: "[Errno 24] Too many open files: '/tmp/tmpo54qfq'
daemon panic:
Caught unexpected exception in _main() on 2014-08-20 11:53:37.918353
Message: "[Errno 24] Too many open files: '/tmp/tmpo54qfq'" of type "<class 'IOError'>"
Traceback (most recent call last):
File "/usr/local/lib/python3.2/dist-packages/homie_core-1.0-py3.2.egg/homie/serv/daemon.py", line 345, in __run
File "/usr/local/lib/python3.2/dist-packages/homie_core-1.0-py3.2.egg/homie/serv/service.py", line 72, in _main
File "/usr/local/lib/python3.2/dist-packages/homie_core-1.0-py3.2.egg/homie/api/itf.py", line 127, in export
File "/usr/local/lib/python3.2/dist-packages/homie_openimmodb-0.2_indev-py3.2.egg/openimmodb/itf.py", line 51, in _retrieve
File "/usr/local/lib/python3.2/dist-packages/homie_openimmodb-0.2_indev-py3.2.egg/openimmodb/conv.py", line 27, in decode
File "/usr/local/lib/python3.2/dist-packages/homie_openimmodb-0.2_indev-py3.2.egg/openimmodb/factories/openimmo/immobilie.py", line 60, in build
File "/usr/local/lib/python3.2/dist-packages/homie_openimmodb-0.2_indev-py3.2.egg/openimmodb/factories/openimmo/anhaenge.py", line 30, in build
IOError: [Errno 24] Too many open files: '/tmp/tmpo54qfq'
According to lsof the whole process has over 5k open files:
# lsof| grep python3| wc -l
5375
I checked it several times: I am using with open(file) as desc everywhere in the code, when I open a file.
Shouldn't the files be closed automatically at the end of each with block, or am I missing something?

Answer 1

tempfile.mkstemp() opens a file for you:
fd, path = mkstemp()
with open(fd, 'wb') as target:
# os.close(fd) is called automatically
You don't need open(path) that opens another file (with the same name).
You could use tempfile.NamedTemporaryFile(delete=False) instead of tempfile.mkstemp().