The FM EPS2_GET_DIRECTORY_LISTING has a parameter file_mask which I guess that it should act as a pattern. I need to read from the AS the files containing a word but the file_mask is working faultly. For example if I pass "*ZIP" it returns a file named '.TXT'. Is there a proper way to use that parameter?
The parameters are described in SAP note 1860206 which I will not quote here because I'm not sure about the copyright status. However, wildcards generally do not work as expected in this case - your best bet is to read without the parameter and filter the table afterwards.
I had similar problem but due to poor(eg. * wildcard can be used only at the and of the file mask string :/ ) implementation of standard FILE_MASK-based filtering feature in EPS2_GET_DIRECTORY_LISTING I ended up with the solution where I read entire directory content and then process it with regular expressions to find matching files/directories.
Related
I'm tring to create an SSIS package to import some dataset files, however given that I seem to be hitting a brick
wall everytime I achieve a small part of the task I need to take a step back and perform a sanity check on what I'm
trying to achieve, and if you good people can advise whether SSIS is the way to go about this then I would
appreciate it.
These are my questions from this morning :-
debugging SSIS packages - debug.writeline
Changing an SSIS dts variables
What I'm trying to do is have a For..Each container enumerate over the files in a share on the SQL Server. For each
file it finds a script task runs to check various attributes of the filename, such as looking for a three letter
code, a date in CCYYMM, the name of the data contained therein, and optionally some comments. For example:-
ABC_201007_SalesData_[optional comment goes here].csv
I'm looking to parse the name using a regular expression and put the values of 'ABC', '201007', and
'SalesData' in variables.
I then want to move the file to an error folder if it doesn't meet certain criteria :-
Three character code
Six character date
Dataset name (e.g. SalesData, in this example)
CSV extension
I then want to lookup the Character code, the date (or part thereof), and the Dataset name against a lookup table
to mark off a 'checklist' of received files from each client.
Then, based on the entry in the checklist, I want to kick off another SSIS package.
So, for example I may have a table called 'Checklist' with these columns :-
Client code Dataset SSIS_Package
ABC SalesData NorthSalesData.dtsx
DEF SalesData SouthSalesData.dtsx
If anyone has a better way of achieving this I am interested in hearing about it.
Thanks in advance
That's an interesting scenario, and should be relatively easy to handle.
First, your choice of the Foreach Loop is a good one. You'll be using the Foreach File Enumerator. You can restrict the files you iterate over to be just CSVs so that you don't have to "filter" for those later.
The Foreach File Enumerator puts the filename (full path or just file name) into a variable - let's call that "FileName". There's (at least) two ways you can parse that - expressions or a Script Task. Depends which one you're more comfortable with. Either way, you'll need to create three variables to hold the "parts" of the filename - I'll call them "FileCode", "FileDate", and "FileDataset".
To do this with expressions, you need to set the EvaluateAsExpression property on FileCode, FileDate, and FileDataset to true. Then in the expressions, you need to use FINDSTRING and SUBSTRING to carve up FileName as you see fit. Expressions don't have Regex capability.
To do this in a Script Task, pass the FileName variable in as a ReadOnly variable, and the other three as ReadWrite. You can use the Regex capabilities of .Net, or just manually use IndexOf and Substring to get what you need.
Unfortunately, you have just missed the SQLLunch livemeeting on the ForEach loop: http://www.bidn.com/blogs/BradSchacht/ssis/812/sql-lunch-tomorrow
They are recording the session, however.
I am going to generate XML file based on the data returned from SQL Server, but there are some special characters like and (there may be other characters like these), which will fail the XML.
Is there any way to escape them?
Thanks!
The control characters U+001C (file separator) and U+001F (unit separator) are not legal to include in an XML 1.0 document, whether verbatim or encoded using a &#...; numeric character reference.
They are allowed in XML 1.1 documents only when included as a character reference. However, XML 1.1 is not nearly as widely accepted as 1.0, and you can't have U+0000 (null) even as a character reference, so it's still not possible to put arbitrary binary data in an XML file — not that it was ever a good idea.
If you want to include data bytes in an XML file you should generally be using an ad hoc encoding of your own that is accepted by all consumers of your particular type of document. It is common to use base64 for the purpose of putting binary data into XML. For formats that do not accommodate any such special encoding scheme, you simply cannot insert these control characters.
What is the purpose of the control characters?
The exact same way you're escaping any other user-supplied input prior to insertion into a database; probably one of (from worst to best):
Escaping control characters prior to construction of an SQL statement
Use of parameterised queries
Use of a DAO or ORM which abstracts this problem away from you
Use parametrized queries and you won't have to worry about escaping. Can't really give you more help as to how to use them unless you mention which language you're using.
Well, I just use the pattern matching stuff to replace those special characters manually. Match for '&#.+?;'
As expected, I get an error when entering some characters not included in my database collation:
(1267, "Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '='")
Is there any function I could use to make sure a string only contains characters existing in my database collation?
thanks
You can use a regular expression to only allow certain characters. The following allows only letters, numbers and _(underscore), but you can change to include whatever you want:
import re
exp = '^[A-Za-z0-9_]+$'
re.match(exp, my_string)
If an object is returned a match is found, if no return value, invalid string.
I'd look at Python's unicode.translate() and codec.encode() functions. Both of these would allow more elegant handling of non-legal input characters, and IIRC, translate() has been shown to be faster than a regexp for similar use-cases (should be easy to google the findings).
From Python's docs:
"For Unicode objects, the translate() method does not accept the optional deletechars argument. Instead, it returns a copy of the s where all characters have been mapped through the given translation table which must be a mapping of Unicode ordinals to Unicode ordinals, Unicode strings or None. Unmapped characters are left untouched. Characters mapped to None are deleted. Note, a more flexible approach is to create a custom character mapping codec using the codecs module (see encodings.cp1251 for an example)."
http://docs.python.org/library/stdtypes.html
http://docs.python.org/library/codecs.html
I am allowing the user to choose any username he wants and it can be anything at all such as
AC♀¿!$"Man'#
Now i need to create a directory for him. What function i use to escape the text so i dont a FS problem/exception?
Whether you replace invalid characters or remove them, there's always going to be the possibility of collisions. If I were you, I'd have a separate primary key for the user (a GUID perhaps) and use that for the directory name. That way you can have your user names anything you'd like without worrying about invalid directory names.
Depending on if your characters are ASCII/Unicode, you can use the byte/character values as a replacement and use some character to mark these replaced characters (like an underscore), giving you something like _001_255_200ValidPart_095_253. Note that you have to replace the marking characters, too (_095 in the example, 95 is the ASCII code for the underscore).
Use Path.GetInvalidFileNameChars or Path.GetInvalidPathChars to check for characters to remove.
http://msdn.microsoft.com/en-us/library/system.io.path.getinvalidfilenamechars.aspx
I think your best bet would be to create a Dictionary that maps invalid filesystem characters to a valid replacement string. Then scan the string for invalid characters, replacing with valid strings as you go. This would allow the user to pick anything they want and give you the consistency to translate it back into the user's name if you want.
You're not going to find a way to escape the username that will give a valid non-clashing directory name in every instance.
A more reliable approach will be to create the directory using some arbitary convention, and then store a mapping between the two. This also provides support for the case where your user wants the ability to change name.
Check out Question #663520 for more on this.
does user need to know the exact name of his / her directory ? If not, why not create directories using arbitrary rules and associate each one to his owner in a DB or something ?
I think I saw once that it was possible to use the functionality of windows' search feature(s) in code. That it was possible to search for files using a sql query (something like "select filename from filestore where directory = 'c:\somedir' and extention in ('.doc','.txt','.me') and datemodified >= '2009-01-01 00:00:00'"
Anyway, even if there isn't a way to do it with a query, is there any functionality in the Windows API OR any simple code for doing exactly this type of thing?
It seems such an obvious thing for there not to be an easy way to do it.
I have a function that finds all files in a dir (with subdirs) but it's basically dumb. It can't search with wildcards and it can't search with other criteria such as date newer than a given date.
If possible I don't want to download any 3rd party shareware libraries to do this. I'm using C++ builder in vista, and if there are any vista-only functionality for this then that's ok.
Actually, you can use FindFirstFileEx() to implement your search criteria. Unfortunately, most of the time you have to implement this yourself.. It isn't too hard..
Basically you can check the WIN32_FIND_DATA structure for criteria that you want to search for. For example, if you wanted to find only files with a creation date greater than a certain date, you would check if the WIN32_FIND_DATA structure member ftCreationTime or ftLastWriteTime, depending on your needs..
You can use FindFirstFileEx() to search with wildcards and attributes.
The C code has been posted thousands of times on Google Groups since 90's
( news://comp.os.ms-windows.programmer.win32 and others )