I have set / array of hosts that fall in below three categories i.e
source_hosts (multiple servers)
ansible_host (single server)
destination_hosts. (multiple servers)
Based on our architecture the plan is to do the following Steps.
Verify if the files exists of source_hosts and has copy permissions for the source user. Also, verify if the "path to folder" n the destination exists and has permissions for the files to get copied. Checking if we are not "Running out of space" on the destination should also be considered.
If the above verification is successful the files should get copied from source_host to ansible_server
Note: I plan to use ansible's fetch module for this http://docs.ansible.com/ansible/fetch_module.html
From the ansible server the files should get copied over to the destination server's respective locations.
Note: I plan to use ansible's copy module for this
http://docs.ansible.com/ansible/copy_module.html
If the file already exists on the destination server a backup must be created with a identifier say "tkt432" along with the timestamp.
Note: Again, I am planning to use copy module for backups but i don't know how to append the identifier to the backed-up files. The module does not have any such feature of appending custom identifier to file names as of my limited knowledge.
I have the following concerns.
what would be the ideal ansible module to address Step 1 ?
How do I address the issue highlighted in Step 4 ?
Any other suggestions are welcomed.
Q: "What would be the ideal ansible module to address Step 1 ?"
A: Modules file and stat. Checking "Running out of space" see Using ansible to manage disk space.
Q: "How do I address the issue highlighted in Step 4 ? If the file already exists on the destination server a backup must be created with an identifier say "tkt432" along with the timestamp."
A: Quoting from the parameters of copy module
backup - Create a backup file including the timestamp ...
Neither the extension nor the place of the backup files is optional. See add optional backup_dir for the backup option #16305.
Q: "Any other suggestions are welcomed."
A: Take a look at module synchronize.
Q: "1. Is there any module to check file/folder permissions (rights) for copy-paste operation with that user id?"
A: There are no copy-paste operations in Ansible.
Q: "Requesting more inputs on how we can append identifiers like "tkt432" to backup filenames while using "copy" modules backup option or any other good solution."
A: There is no more input. Ansible does not do that.
Q: "I feel I won't be able to use the copy module and will have to fallback to writing shell scripts for the above-mentioned issues."
A: Yes. Modules shell and command could help with this.
Related
My current scenario is like this:
I need to login to sqlplus from a shell script to call a stored procedure.
After that I need to create a CSV file by SPOOLING data from a table.
Then I need to check whether the CSV file has been created in a particular directory and depending on the result an update query needs to be run.
I know that this can be checked within sqlplus with the help of UTL_FILE package but unfortunately due to Client policies,the access of this package is restricted in the current system.
Another way is to exit from sqlplus and perform the file check in UNIX and then again log in to sqlplus to perform the rest actions. But this I believe would result in slower execution time and performance is an important factor in this implementation as the tables contain huge volumes of data(in millions).
So is there any other way to check this from sqlplus without exiting from the current session?
System Info:
OS - Red Hat Enterprise Linux
Database - Oracle 11g
If the file is on the same machine that you're running SQL*Plus on, you could potentially use the host command.
If the file you're checking is the same one you're spooling to, it must exist anyway, or you would have got an SP error of some kind; but if you do want to check the same file for some reason, and assuming you have a substitution variable with the file name:
define csv_file=/path/to/spool.csv
-- call procedure
spool &csv_file
-- do query
spool off
host ls &csv_file
update your_table
set foo=bar
where &_rc = 0;
If the file exists when the host command is run, the _rc substitution variable will be set to zero. If the file doesn't exist or isn't readable for any reason it will be something else - e.g. 2 if the file just doesn't exist. Adding the check &_rc = 0 to your update will mean no rows are updated if there was an error. (You can of course still have whatever other conditions you need for the update).
You could suppress the display of the file name by adding 1>/dev/null to the host command string; and could also suppress any error messages by also adding 2>/dev/null, though you might want to see those.
The documentation warns against using &_rc as it isn't portable; but it works on RHEL so as long as you don't need your script to be portable to other operating systems this may be good enough for you. What you can't do, though, is do anything with the contents of the file, or interpret anything about it. All you have available is the return code from the command you run. If you need anything more sophisticated you could call a script that generates specific return codes, but that's getting a bit messy.
I have a requirement to create a dynamic file based on the content in hadoop job.properties and then put it in Distributed Cache.
When I create the file I see that it has been created with the path of "/tmp".
I create a symbolic name and refer to this file in the cache. Now, when I try to read the file in the Dis. cache I am not able to access it. I get th error caused by: java.io.FileNotFoundException: Requested file /tmp/myfile6425152127496245866.txt does not exist.
Can you please let me know If should I need to specify the path also while creating the file and also use that path while accessing/reading the file.
I only need the file to be available only till the job is running.
I don't really get your meaning of
I only need the file to be available only till the job is running
But, when I practice to use distributed cache , I use path like this :
final String NAME_NODE = "hdfs://sandbox.hortonworks.com:8020";
job.addCacheFile(new URI(NAME_NODE + "/user/hue/users/users.dat"));
hope this will help you .
I'm working on creating a csv export from a SQL Server database and I've been familiar with a process for doing so that admittedly, I've never completely understood. The process involves creating a "template" file, which defines the columns and structure for the file export. Once the "template" file exists, you can use a Data Flow task to fill it and a File System Task to copy it to the final storage destination with whatever file name you'd like (frequently a date/time stamp).
Is there a reason that you can't simply create a file directly, without the intermediate "template" file? I've looked around for a bit and it seems like all the proposed solutions involve connecting to an existing file. I see that there is a "Create File" Usage type for a "File" connection manager, but you can't use it in any File System Task. The only File System Type connection managers you can use relative to a file are "Copy", "Delete", "Move", "Rename", and "Set Attributes".
Is there a way to create a file at package run time and fill it?
The whole point of SSIS is to create a data flow with metadata so that the data can be manipulated - if you just want to go database direct to CSV you are probably better off using bcp (bulk copy program) from the command line. If you want to include it as part of a SSIS package just add an Execute Process Task and add the command line to that. You can dynamically change the included columns or the output file by adding an expression to the task. You could also call bcp though TSQL using an Excute SQL Task.
One other option is to concatenate all your columns in your query inter-spaced with a comma literal and output to a text file with just one very wide column.
For documentation on bcp look here
I've set up a File System path inside a ForEachFile enumerator in SSIS 2012. I'm iterating over a directory, loading each file, archiving that file, then processing the next file, etc. I've set the Destination folder via an expression that uses a Project Param value, and I get the source file from the variable set in the ForEachFile enumerator. The File System task says it can't find my Destination folder:
Here's the File System Task:
And proof that the destination folder exists:
Why am I getting this error? I'd swear I've used the exact same technique in SSIS 2008 and 2005. This is 2012, but it should work the same way.
I met the same problem and to resolve it i create a variable and i put something like this in the path.Be carefull you must put double "\ \".
So the path must be like this:"\ \ \ \ad1hfdalhp001\ \d$\ \data\ \Archive\ \"
The DestinationConnection field must contain a reference to a flat file connection. The error is saying that you have no Flat file connection manager with that name.
DestinationConnection should not be a free text field. You should be able to open a drop down in the DestinationConnection field and select or create a connection manager.
You will need to configure your output file path as the ConnectionString property on the file connection manager referenced in the DestinationConnection field.
Using a UNC is still an option. Similar to as you've done with the Source, on your Destination, set IsDestinationPathVariable = True and then push \server\path into a Variable User::ArchivePath or similar.
Otherwise, it is as user3922917 indicates: if IsDestinationPathVariable is false, then you need to use a File Connection manager.
In your comments, you indicate that you're building the UNC path based on an Expression in the File System Task. I find I have a better experience when I build my expressions in SSIS Variables and then simply assign that Variable into the Task's Expression. While this step may seem to provide another layer of maintenance, put a break point on the Task and tell me what the expression evaluates to. And you can't. It's only available to the object to use and you are unable to inspect it so you're left high and dry if your formula is off. Which never happens when you're having to deal with escaping a UNC path
Is it possible to open a flatfile when only part of the file name is known?
I have files in a directory that have a timestamp appended to the filename, is it possible to open it by specifiying the known part of the filename (excluding timestamp)?
Is it possible with a PLSQL only approach?
There is a dbms_ package which allows you to get a directory listing for the directory (or you can implement your own in a java stored procedure - google!) This will allow you to find the file you are looking for - if necessary choose which is the relevant file and then process.
See http://notdennis.wordpress.com/2013/07/03/listing-directory-files-plsql/