I am trying to download files from a ftp server and importing the data to django. So i created a list contain server address,login details,path,file name,and the path where the file to be download and pass to a function which do downloading. it is working file in my sytem when move it to client server it showing error like
" error downloading C_VAR1_31012014_1.DAT - [Errno 20] Not a directory: 'common/VARRate/C_VAR1_31012014_1.DAT"
this is how the list look like
self.fileDetails = {
'NSE FO VAR RATE FILE': ('ftp.xxx.com', username, passwd, 'common/VARRate', 'C_VAR1_\d{4}201[45]_\d{1}.DAT', 'Data/samba/Ftp/Capex10/NSECM/VAR RATE'),
}
for fileType in self.fileDetails:
self.ftpDownloadFiles(fileType)
This details will pass to the function following function
def ftpDownloadFiles(self, fileType):
logging.info('Started ' + str(fileType))
try:
ftpclient = ftplib.FTP(self.fileDetails[fileType][FDTL_SRV_POS],
self.fileDetails[fileType][FDTL_USR_POS],
self.fileDetails[fileType][FDTL_PSWD_POS],
timeout=120)
#ftpclient.set_debuglevel(2)
ftpclient.set_pasv(True)
logging.info('Logged in to ' + self.fileDetails[fileType][FDTL_SRV_POS] +\
time.asctime())
logging.info('\tfor type: '+ fileType)
except BaseException as e:
print e
return
remotepath = self.fileDetails[fileType][FDTL_PATH_POS]
#matched, unmatched, downloaded = 0
try:
ftpclient.cwd(remotepath)
ftpclient.dir(filetimestamps.append)
except BaseException as e:
logging.info('\tchange dir error : ' + remotepath + ' ' +\
e.__str__())
self.walkTree(ftpclient, remotepath, fileType)
#logging.info('\n\tMatched %d, Unmatched %d, Downloaded %d'
# % (matched, unmatched, downloaded))
ftpclient.close()
From here it will call next function here the download process will start
def walkTree(self, ftpclient, remotepath, fileType):
# process files inside remotepath; cwd already done
# remotepath to be created if it doesnt exist locally
copied=matched=downloaded=imported = 0
files = ftpclient.nlst()
localpath = self.fileDetails[fileType][FDTL_DSTPATH_POS]
rexpCompiled = re.compile(self.fileDetails[fileType][FDTL_PATRN_POS])
for eachFile in files:
try:
ftpclient.cwd(remotepath+'/'+eachFile)
self.walkTree(ftpclient, remotepath+'/'+eachFile+'/', fileType)
except ftplib.error_perm: # not a folder, process the file
# every file to be saved in same local folder as on ftp srv
saveFolder = remotepath
saveTo = remotepath + '/' + eachFile
if not os.path.exists(saveFolder):
try:
os.makedirs(saveFolder)
print "directory created"
except BaseException as e:
logging.info('\tcreating %s : %s' % (saveFolder, e.__str__()))
if (not os.path.exists(saveTo)):
try:
ftpclient.retrbinary('RETR ' + eachFile, open(saveTo, 'wb').write)
#logging.info('\tdownloaded ' + saveTo)
downloaded += 1
except BaseException as e:
logging.info('\terror downloading %s - %s' % (eachFile, e.__str__()))
except ftplib.error_perm:
logging.info('\terror downloading %s - %s' % (eachFile, ftplib.error_perm))
elif (fileType == 'NSE CASH CLOSING FILE'): # spl case if file exists
try:
# rename file
yr = int(time.strftime('%Y')) - 1
os.rename(saveTo, saveTo + str(yr))
# download it
ftpclient.retrbinary('RETR ' + eachFile, open(saveTo, 'wb').write)
downloaded += 1
except BaseException as e:
logging.info('\terror rename/ download %s - %s' % (eachFile, e.__str__()))
Can any one help me to resolve this problem
Try to use os.path.join() in stead of the hardcoded slashes as path dividers for the os to download to. / or \ depends of the local os.
e.g. in your code:
saveTo = remotepath + '/' + eachFile
would become:
saveTo = os.path.join(remotepath,eachFile)
see https://docs.python.org/2/library/os.path.html
Related
I used Python 3.4 to create a programm that goes through E-mails and saves specific attachments to a file server.
Each file is saved to a specific destination depending on the sender's E-mail's address.
My problem is that the destination folders and the attachments are both in Hebrew and for a few attachments I get an error that the path does not exsist.
Now that's not possible because It can fail for one attachment but not for the others on the same Mail (the destination folder is decided by the sender's address).
I want to debug the issue but I cannot get python to display the file path it is trying to save correctly. (it's mixed hebrew and english and it always displays the path in a big mess, although it works correctly 95% of the time when the file is being saved to the file server)
So my questions are:
what should I add to this code so that it will proccess Hewbrew correctly?
Should I encode or decode somthing?
Are there characters I should avoid when proccessing the files?
here's the main piece of code that fails:
try:
found_attachments = False
for att in msg.Attachments:
_, extension = split_filename(str(att))
# check if attachment is not inline
if str(att) not in msg.HTMLBody:
if extension in database[sender][TYPES]:
file = create_file(str(att), database[sender][PATH], database[sender][FORMAT], time_stamp)
# This is where the program fails:
att.SaveAsFile(file)
print("Created:", file)
found_attachments = True
if found_attachments:
items_processed.append(msg)
else:
items_no_att.append(msg)
except:
print("Error with attachment: " + str(att) + " , in: " + str(msg))
and the create file function:
def create_file(att, location, format, timestamp):
"""
process an attachment to make it a file
:param att: the name of the attachment
:param location: the path to the file
:param format: the format of the file
:param timestamp: the time and date the attachment was created
:return: return the file created
"""
# create the file by the given format
if format == "":
output_file = location + "\\" + att
else:
# split file to name and type
filename, extension = split_filename(att)
# extract and format the time sent on
time = str(timestamp.time()).replace(":", ".")[:-3]
# extract and format the date sent on
day = str(timestamp.date())
day = day[-2:] + day[4:-2] + day[:4]
# initiate the output file
output_file = format
# add the original file name where needed
output_file = output_file.replace(FILENAME, filename)
# add the sent date where needed
output_file = output_file.replace(DATE, day)
# add the time sent where needed
output_file = output_file.replace(TIME, time)
# add the path and type
output_file = location + "\\" + output_file + "." + extension
print(output_file)
# add an index to the file if necessary and return it
index = get_file_index(output_file)
if index:
filename, extension = split_filename(output_file)
return filename + "(" + str(index) + ")." + extension
else:
return output_file
Thanks in advance, I would be happy to explain more or supply more code if needed.
I found out that the promlem was not using Hebrew. I found that there's a limit on the number of chars that the (path + filename) can hold (255 chars).
The files that failed excided that limit and that caused the problem
For my application on Openshift, I am trying to write a pre_build script that accesses the database. The goal is to have migration scripts between database versions that are executed when the code is deployed. The script would compare the current database version with the version needed by the application code and then run the correct script to migrate the database.
Now the problem is that apparently the pre_build script is executed on Jenkins and not on the destination cartridge and therefore the environment variables with the database connection arguments are not available.
This is the pre_build script that I've written so far:
#!/usr/bin/env python
print "*** Database migration script ***"
# get goal version
import os
homedir = os.environ["OPENSHIFT_HOMEDIR"]
migration_scripts_dir = homedir + "app-root/runtime/repo/.openshift/action_hooks/migration-scripts/"
f = open(migration_scripts_dir + "db-version.txt")
goal = int(f.read())
f.close()
print "I need database version " + str(goal)
# get database connection details
# TODO: find a solution of not hard coding the connection details here!!!
# Maybe by using jenkins environment variables like OPENSHIFT_APP_NAME and JOB_NAME
db_host = "..."
db_port = "..."
db_user = "..."
db_password = "..."
db_name = "..."
import psycopg2
try:
conn = psycopg2.connect("dbname='" + db_name + "' user='" + db_user + "' host='" + db_host + "' password='" + db_password + "' port='" + db_port + "'")
print "Successfully connected to the database"
except:
print "I am unable to connect to the database"
cur = conn.cursor()
def get_current_version(cur):
try:
cur.execute("""SELECT * from db_version""")
except:
conn.set_isolation_level(0)
cur.execute("""CREATE TABLE db_version (db_version bigint NOT NULL)""")
cur.execute("""INSERT INTO db_version VALUES (0)""")
cur.execute("""SELECT * from db_version""")
current_version = cur.fetchone()[0]
print "The current database version is " + str(current_version)
return current_version
def recursive_execute_migration(cursor):
current_version = get_current_version(cursor)
if (current_version == goal):
print "Database is on the correct version"
return
elif (current_version < goal):
sql_filename = "upgrade" + str(current_version) + "-" + str(current_version + 1) + ".sql"
print "Upgrading database with " + sql_filename
cursor.execute(open(migration_scripts_dir + sql_filename, "r").read())
recursive_execute_migration(cursor)
else:
sql_filename = "downgrade" + str(current_version) + "-" + str(current_version - 1) + ".sql"
print "Downgrading database with " + sql_filename
cursor.execute(open(migration_scripts_dir + sql_filename, "r").read())
recursive_execute_migration(cursor)
conn.set_isolation_level(0)
recursive_execute_migration(cur)
cur.close()
conn.close()
Is there another way of doing automatic database migrations?
Thanks for your help.
Whenever I attempt to save using this code:
def openfile(self):
self.filename = askopenfilename(filetypes=(("Extinction Save files", "*.exis"), ("All files", "*.*")))
def save(self):
try:
if not self.filename:
self.filename = asksaveasfile(mode='w', defaultextension=".exis", filetypes=(("Extinction Save files", "*.exis"), ("All files", "*.*")))
# if not '.exis' in self.filename:
# self.filename += ".exis"
self.filename.close()
with open(self.filename, "w") as file:
file.write(self.compiledata)
except Exception as error:
print(str(error))
def saveas(self):
try:
self.filename = asksaveasfile(mode='w', defaultextension=".exis", filetypes=(("Extinction Save files", "*.exis"), ("All files", "*.*")))
# if not '.exis' in self.filename:
# self.filename += ".exis"
self.filename.close()
with open(self.filename, "w") as file:
file.write(self.compiledata)
except Exception as error:
print(str(error))
def compiledata(self):
char = comboBox.currentText()
health = lineEdit_2.text()
level = lineEdit_3.text()
XP = lineEdit_4.text()
inv = []
for index in range(self.listWidget.count()):
inv.append(self.listWidget.item(index))
return char + '\ninv[' + ', '.split(inv) + '\n]' + health + '\n' + level + '\n' + XP
I get the following error message:
invalid file: <_io.TextIOWrapper name='C:/Users/Joan/Desktop/file.exis' mode='w' encoding='cp1252'>
I have seen one instance on this site where someone had a somewhat similar error message, but the solution did not help me. I just don't understand what I am supposed to do at this point.
Because you use asksaveasfile(), self.filename is not a filename, but an opened file. You then use self.filename.close(), so self.filename becomes a closed file, but open(...) expects a filename.
To get a the filename from the file you can use self.filename.name, so:
with open(self.filename.name, "w") as file:
I'm beginner Python. I have many PCs connected with network switch. One is manager PC. others are client PCs. I'm going to make remotely file cCopy/delete batch program such as DOS batch command in Python.
any help how to start ?
DOS Batch command
xcopy E:\Share_main\A*.* \124.122.11.101\A\ /e /h /k /Y
Here is sample code.
#!/usr/bin/env python-3.x
# -*- coding: utf-8 -*-
# based on Carnival http://ask.python.kr/users/6970/carnival/
import os, sys, csv, re, datetime
from multiprocessing import Process
class Server:
def __init__(self, addr, path):
self.addr = addr
self.path = path
def multi_distribute_from_buffers(server, dirpath, filenames, subdir, buffers):
l = re.findall(r"[\w']+",subdir)
m = re.findall(r"[\w']+",dirpath)
cnt_l = len(l)
cnt_m = len(m)
remotepath = "//%s/%s" % (server.addr, server.path)
if(cnt_m > cnt_l):
for j in range(cnt_m - cnt_l):
remotepath += "/%s" % (m[cnt_l + j])
index = 0
for filename in filenames:
remotepathfile = "%s/%s" % (remotepath, filename)
with open(remotepathfile, 'wb') as outFile:
outFile.write(buffers[index])
index = index + 1
def make_dir(server_list, subdir, dirpath):
for server in server_list:
l = re.findall(r"[\w']+",subdir)
m = re.findall(r"[\w']+",dirpath)
cnt_l = len(l)
cnt_m = len(m)
path = "//%s/%s" % (server.addr, server.path)
if(cnt_m > cnt_l):
for j in range(cnt_m - cnt_l):
path += "/%s" % (m[cnt_l + j])
d = os.path.dirname(path)
if not os.path.exists(d):
print ("{}, Make dir {}".format(datetime.datetime.now(), d))
os.makedirs(d)
if not os.path.exists(path):
print ("{}, Make dir {}".format(datetime.datetime.now(), path))
os.makedirs(path)
def dist_mems(server_list, subdir):
filecount = 0
for dirpath, dirnames, filenames in os.walk(subdir):
make_dir(server_list, subdir, dirpath)
buffers = []
for filename in filenames:
pathname = os.path.join(dirpath, filename)
print("{}, {} : Read from {}".format(++filecount, datetime.datetime.now(), pathname))
with open(pathname, 'rb') as inFile:
buffers.append(inFile.read())
for server in server_list:
Process(target = multi_distribute_from_buffers,args=(server, dirpath, filenames, subdir, buffers)).start()
def get_server_list(filename):
mydictionary = []
csvFile = csv.reader(open(filename, "r"))
for row in csvFile:
mydictionary.append(Server(row[0], row[1]))
return mydictionary
if __name__ == '__main__':
start = datetime.datetime.now()
clientListfile = 'C:\\Users\\Public\\client_list.csv'
if(sys.argv[1] != ''):
clientListfile = sys.argv[1]
sourceFolder = 'C:\\Users\\Public\\'
if(sys.argv[2] != ''):
sourceFolder = sys.argv[2]
server_list = get_server_list(clientListfile)
dist_mems(server_list, sourceFolder)
end = datetime.datetime.now()
diff = end - start
print(" XCopying Start to {} clients : {}".format(len(server_list), start))
print(" XCopying finished : {}".format(end))
print(" XCopying Total time span : {}".format(diff))
I'm running a script made in Groovy from Soap UI and the script needs to generate lots of files.
Those files have also in the name two numbers from a list (all the combinations in that list are different), and there are 1303 combinations
available and the script generates just 1235 files.
A part of the code is:
filename = groovyUtils.projectPath + "\\" + "$file"+"_OK.txt";
targetFile = new File(filename);
targetFile.createNewFile();
where $file is actually that part of the file name which include those 2 combinations from that list:
file = "abc" + "-$firstNumer"+"_$secondNumber"
For those file which are not created is a message returned:"The filename, directory name or volume label syntax is incorrect".
I've tried puting another path:
filename = "D:\\rez\\" + "\\" + "$file"+"_OK.txt";
targetFile = new File(filename);
targetFile.createNewFile();
and also:
File parentFolder = new File("D:\\rez\\");
File targetFile = new File(parentFolder, "$file"+"_OK.txt");
targetFile.createNewFile();
(which I've found here: What are possible reasons for java.io.IOException: "The filename, directory name, or volume label syntax is incorrect")
but nothing worked.
I have no ideea where the problem is. Is strange that 1235 files are created ok, and the rest of them, 68 aren't created at all.
Thanks,
My guess is that some of the files have illegal characters in their paths. Exactly which characters are illegal is platform specific, e.g. on Windows they are
\ / : * ? " < > |
Why don't you log the full path of the file before targetFile.createNewFile(); is called and also log whether this method succeeded or not, e.g.
filename = groovyUtils.projectPath + "\\" + "$file"+"_OK.txt";
targetFile = new File(filename);
println "attempting to create file: $targetFile"
if (targetFile.createNewFile()) {
println "Successfully created file $targetFile"
} else {
println "Failed to create file $targetFile"
}
When the process is finished, check the logs and I suspect you'll see a common pattern in the ""Failed to create file...." messages
File.createNewFile() returns false when a file or directory with that name already exists. In all other failure cases (security, I/O) it throws an exception.
Evaluate createNewFile()'s return value or, additionally, use the File.exists() method:
File file = new File("foo")
// works the first time
createNewFile(file)
// prints an error message
createNewFile(file)
void createNewFile(File file) {
if (!file.createNewFile()) {
assert file.exists()
println file.getPath() + " already exists."
}
}