groovy: parsing files with different extensions and using if/else - file

I have a script:
def tmn_file = ~/.*\.tmn/
def tmc_file = ~/.*\.tmc/
def newTerm = new Properties().with { props ->
new File(inputPath).eachFile(tmn_file) { file ->
file.withReader { reader ->
load( reader )
println "Read data from file $file:"
something read from file...
switch( props.ACTION ) {
case 'NEW':
do something...
}
switch( props.ACTION ) {
case 'CHANGE':
do something...
}
this script looking in directory with path inputPath file with extension tmn_file, which can containt ACTION - NEW or CHANGE.
Script works great, but i want to make another thing:
if file have extension *.tmn (tmn_file) - start only ACTION with NEW case
if file have extension *.tmc (tmc_file) - start only ACTION with CHANGE case
How i can realize decision?

Here is solution:
new Properties().with { props ->
new File(inputPath).eachFile(FileType.FILES) { file ->
file.withReader { reader ->
load(reader)
println "Read data from file $file:"
if (file.name.endsWith('tmn') & props.ACTION == 'NEW' || file.name.endsWith('tmc') & props.ACTION == 'CHANGE') {
// NEW mode
switch( props.ACTION ) {
case 'NEW':
...do someth...
break
// CHANGE mode
case 'CHANGE':
println "***CHANGE mode is on***"
...do someth...
break
default:
throw new RuntimeException("Unknown ACTION $props.ACTION")
}
} else {
if (file.name.endsWith('tmn') || file.name.endsWith('tmc')){
println "$file dont match for action $props.ACTION"
} else {
println "$file have wrong extension "}
}
}
}
}

Related

Get array data and insert into database

I'm trying to get the data in the array that came from another function(that function is extracting the data in the csv file) and when i tried calling the two fields from that array it shows an error that it is unidentified variables.
The $this->csv_process(); as shown on the function action() is the function that extracts the data from the csv file and stores it in an array which is successful since I tried checking it on var_dump();
I also named the two fields as $name and $email as shown below:
Function CSV_process()
public function csv_process()
{
/* variables for openning the csv file */
if (!in_array($extension, $allowed_ext)) {
$this->session->set_flashdata("message", "Sorry, CSV file only.");
} else {
if ($filesize > 0) {
$file = fopen($filename, "r");
$toWrite = array();
$error = false;
$col_size = 2;
$skip = 0;
while ($data = fgetcsv($file, 10000, ","))
{
$skip++;
if ($skip == 1) {
continue;
}
$numofcol = count($data);
if ($numofcol != $col_size ) {
$this->session->set_flashdata("message", "Column count exceeded or missing.");
} else {
$name1 = $data[0];
$name = str_replace("'", "''", $name1);
$email1 = $data[1];
$email = str_replace("'", "''", $email1);
$toWrite[] = [
'name' => $name,
'email' => $email
];
}
}
}
}
return $toWrite;
}
Function Action()
function action(){
$toWrite[] = $this->csv_process();
foreach ($toWrite as $arr) {
list($name, $email) = $arr;
//die(var_dump($arr));
$query = $this->db->query("SELECT * FROM import WHERE name ='$name' AND email = '$email'");
if ($query->num_rows() >= 1) {
} else {
if ($name == "" OR $email == "") {
} else {
if ((filter_var($email, FILTER_VALIDATE_EMAIL)) == FALSE ) {
} else {
$this->db->query("INSERT INTO import(name, email, created_date) VALUES('".$name."', '".$email."', '".date("Y-m-d h-i-s")."')");
$this->session->set_flashdata('message', 'SUCCESS YEAY');
redirect('Clean_csv/index');
}
}
}
$query->free_result();
}
}
Listing arrays doesn't seem to work for here, anyone knows how to extract the data array from $arr?
You don't need to extract the values. You can use each $arr in a bound query. It simplifies the syntax for the select query.
For inserting use CodeIgniter's insert() method. Again, the $arr can be used directly by adding the date to it before the insert is attempted.
I think this will work.
function action()
{
$toWrite[] = $this->csv_process();
foreach($toWrite as $arr)
{
$query = $this->db->query("SELECT * FROM import WHERE name=? AND email=?", $arr);
if($query->num_rows() >= 1)
{}
else
{
if($arr['name'] == "" OR $arr['email'] == "")
{}
else
{
if((filter_var($email, FILTER_VALIDATE_EMAIL)) == FALSE)
{}
else
{
$arr['created_date'] = date("Y-m-d h-i-s");
$this->db->insert("import", $arr);
$this->session->set_flashdata('message', 'SUCCESS YEAY');
//??? redirect('Clean_csv/index');
//Are you sure, you may still have more $arr in $toWrite to process - right?
}
}
}
$query->free_result();
}
}
You need to know what a terrible idea it is to repeatedly run database queries inside a loop. Even though you use free_result() it could be a massive drain on server resources. If your csv file has several thousand items you are severely stressing the database and the server.

Groovy pickle alternative

I'm searching for something similar to Python's pickle. I want to do like this example:
https://www.quora.com/Is-there-a-way-to-keep-the-session-after-login-with-Selenium-Python
saving cookies
import pickle
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('http://www.quora.com')
# login code
pickle.dump(driver.get_cookies() , open("QuoraCookies.pkl","wb"))
loading cookies
import pickle
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('http://www.quora.com')
for cookie in pickle.load(open("QuoraCookies.pkl", "rb")):
driver.add_cookie(cookie)
I've tried many examples from the web without success. I saw the topics here as well.
In groovy case I'll :
Set<Cookie> allCookies = driver.manage().getCookies();
Variable allCookies should be serialized to file and deserialized later.
Java serialization is an option here. With credit to this post, here is an implementation:
import org.openqa.selenium.Cookie
def loadFile = { filename ->
def result = null
try {
def fis = new FileInputStream(filename)
def instream= new ObjectInputStream(fis)
result = (Set<Cookie>) instream.readObject()
instream.close()
} catch (Exception e) {
System.out.println(e)
}
return result
}
def saveFile = { set, filename ->
try {
def fos = new FileOutputStream(filename)
def out = new ObjectOutputStream(fos)
out.writeObject(set)
out.flush()
out.close()
} catch (IOException e) {
System.out.println(e)
}
}
and a sample driver program:
// ----------------------- main
if (args.size() >= 2) {
def action = args[0]
def file = args[1]
if (action == "write") {
def cookies = new HashSet<Cookie>()
cookies << new Cookie("Toronto", "Canada")
cookies << new Cookie("London", "UK")
cookies << new Cookie("Paris", "France")
saveFile(cookies, file)
} else if (action == "read") {
def cookies = loadFile(file)
cookies.each { cookie ->
println "${cookie.name} ${cookie.value}"
}
} else {
println "Unknown action: " + action
}
} else {
println "Usage: groovy CookieManager.groovy [read|write] filename"
}
Example usage (a) with all above code in the same script and (b) assuming that client-combined-3.0.0-beta2-nodeps.jar from here is on the classpath:
bash$ groovy CookieManager.groovy write set.dat
bash$ groovy CookieManager.groovy read set.dat
London UK
Toronto Canada
Paris France

CakePHP 3.x: log as serialized array

I'm writing my own parser log for CakePHP.
I only need one thing: that is not written a log "message" (as a string), but a serialized array with various log information (date, type, line, stack traces, etc.).
But I don't understand what method/class I should rewrite, although I have consulted the APIs. Can you help me?
EDIT:
For now I do the opposite: I read the logs (already written) and I transform them into an array with a regex.
My code:
$logs = array_map(function($log) {
preg_match('/^'.
'([\d\-]+\s[\d:]+)\s(Error: Fatal Error|Error|Notice: Notice|Warning: Warning)(\s\(\d+\))?:\s([^\n]+)\n'.
'(Exception Attributes:\s((.(?!Request|Referer|Stack|Trace))+)\n)?'.
'(Request URL:\s([^\n]+)\n)?'.
'(Referer URL:\s([^\n]+)\n)?'.
'(Stack Trace:\n(.+))?'.
'(Trace:\n(.+))?(.+)?'.
'/si', $log, $matches);
switch($matches[2]) {
case 'Error: Fatal Error':
$type = 'fatal';
break;
case 'Error':
$type = 'error';
break;
case 'Notice: Notice':
$type = 'notice';
break;
case 'Warning: Warning':
$type = 'warning';
break;
default:
$type = 'unknown';
break;
}
return (object) af([
'datetime' => \Cake\I18n\FrozenTime::parse($matches[1]),
'type' => $type,
'error' => $matches[4],
'attributes' => empty($matches[6]) ? NULL : $matches[6],
'url' => empty($matches[9]) ? NULL : $matches[9],
'referer' => empty($matches[11]) ? NULL : $matches[11],
'stack_trace' => empty($matches[13]) ? (empty($matches[16]) ? NULL : $matches[16]) : $matches[13],
'trace' => empty($matches[15]) ? NULL : $matches[15]
]);
}, af(preg_split('/[\r\n]{2,}/', $logs)));
For now I do the opposite: I read the logs (already written) and with a regex I transform them into an array.
The problem is this is terribly expensive. and that it would be better to do the opposite: to write directly to the logs as a serialized array.
I think what you want to do is write your own LogAdapter.
You simply create a class ArrayLog (extends BaseLog) as mentioned in the docs and configure cakePHP to use it. Within the log function you append the information like $level, $message and $context to a file as an array. This will result in a log file with several arrays that then can be split.
That being said, I would suggest to log to the database and read it out instead of parsing.
Ok, that's it!
(note that this code is absolutely experimental, I have yet to test it properly)
One interesting thing that I want to do: for each log, write to the serialized file and also simultaneously in a plan file.
This allows me either to read logs as a plain text file, or they can be manipulated using the serialized file.
use Cake\Log\Engine\FileLog;
class SerializedLog extends FileLog {
protected function _getLogAsArray($level, $message) {
$serialized['level'] = $level;
$serialized['datetime'] = date('Y-m-d H:i:s');
//Sets exception type and message
if(preg_match('/^(\[([^\]]+)\]\s)?(.+)/', $message, $matches)) {
if(!empty($matches[2]))
$serialized['exception'] = $matches[2];
$serialized['message'] = $matches[3];
}
//Sets the exception attributes
if(preg_match('/Exception Attributes:\s((.(?!Request URL|Referer URL|Stack Trace|Trace))+)/is', $message, $matches)) {
$serialized['attributes'] = $matches[1];
}
//Sets the request URL
if(preg_match('/^Request URL:\s(.+)$/mi', $message, $matches)) {
$serialized['request'] = $matches[1];
}
//Sets the referer URL
if(preg_match('/^Referer URL:\s(.+)$/mi', $message, $matches)) {
$serialized['referer'] = $matches[1];
}
//Sets the trace
if(preg_match('/(Stack )?Trace:\n(.+)$/is', $message, $matches)) {
$serialized['trace'] = $matches[2];
}
$serialized['full'] = date('Y-m-d H:i:s').' '.ucfirst($level).': '.$message;
return (object) $serialized;
}
public function log($level, $message, array $context = []) {
$message = $this->_format(trim($message), $context);
$filename = $this->_getFilename($level);
if (!empty($this->_size)) {
$this->_rotateFile($filename);
}
$pathname = $this->_path . $filename;
$mask = $this->_config['mask'];
//Gets the content of the existing logs and unserializes
$logs = #unserialize(#file_get_contents($pathname));
if(empty($logs) || !is_array($logs))
$logs = [];
//Adds the current log
$logs[] = $this->_getLogAsArray($level, $message);
//Serializes logs
$output = serialize($logs);
if (empty($mask)) {
return file_put_contents($pathname, $output);
}
$exists = file_exists($pathname);
$result = file_put_contents($pathname, $output);
static $selfError = false;
if (!$selfError && !$exists && !chmod($pathname, (int)$mask)) {
$selfError = true;
trigger_error(vsprintf(
'Could not apply permission mask "%s" on log file "%s"',
[$mask, $pathname]
), E_USER_WARNING);
$selfError = false;
}
return $result;
}
}

Scala Lift - Save uploaded files to server directory

I'm currently storing images within the webapp folder of my Lift project, which I know will cause problems in future.
val path = "src/main/webapp/files/"
And the code I'm using to save it:
case Full(file) =>
val holder = new File(path, "test.txt")
val output = new FileOutputStream(holder)
try {
output.write(file)
} finally {
output.close()
}
}
What I'm trying to do is save the to the server root in an easily manageable folder called files, so SERVER_ROOT/files outside of the project folder.
Firstly how would I access the path to the root of the server so I can save them there?
Secondly how would I serve these files from my app, so I can display them on a page?
Thanks in advance, any help much appreciated :)
You have to store file to exact place on filesystem according to absolute path. I have written this code and it works, so maybe it helps you:
def storeFile (file : FileParamHolder): Box[File] =
{
getBaseApplicationPath match
{
case Full(appBasePath) =>
{
var uploadDir = new File(appBasePath + "RELATIVE PATH TO YOUR UPLOAD DIR")
val uploadingFile = new File(uploadDir, file.fileName)
println("upload file to: " + uploadingFile.getAbsolutePath)
var output = new FileOutputStream(uploadingFile)
try
{
output.write(file.file)
}
catch
{
case e => println(e)
}
finally
{
output.close
output = null
}
Full(uploadingFile)
}
case _ => Empty
}
}
and this is my getBaseApplicationPath function which finds out absolute path of local machine (server or your devel PC):
def getBaseApplicationPath: Box[String] =
{
LiftRules.context match
{
case context: HTTPServletContext =>
{
var baseApp: String = context.ctx.getRealPath("/")
if(!baseApp.endsWith(File.separator))
baseApp = baseApp + File.separator
Full(baseApp)
}
case _ => Empty
}
}

file upload cakephp

when uploading an image to the server using cakephp
$this->Model->Behaviors->attach('ImageUpload', Configure::read('photo.files'));
photo uploaded successfully, and the database fields also
but shows following error instead of returning to index page.
Notice (8): Undefined index: class [CORE\cake\libs\model\behaviors\upload.php, line 104]
Notice (8): Undefined index: class [CORE\cake\libs\model\behaviors\upload.php, line 107]
Warning (2): Cannot modify header information - headers already sent by (output started at E:\umoorthy_105act10\projects\dev1base\core\cake\basics.php:111) [CORE\cake\libs\controller\controller.php, line 614]
wat to do?
Cake has already wrote where to look for a problem
Configure::read('photo.files')
do following to check if everything is ok
pr(Configure::read('photo.files'))
public function uploadFilesIphone($folder, $formdata, $replace , $itemId = null) {
// setup dir names absolute and relative echo "<pre>"; print_r($formdata); exit;
$folder_url = WWW_ROOT.$folder;
$rel_url = $folder; //echo
// create the folder if it does not exist
if(!is_dir($folder_url)) {
mkdir($folder_url);
}
// if itemId is set create an item folder
if($itemId) {
// set new absolute folder
$folder_url = WWW_ROOT.$folder.'/'.$itemId;
// set new relative folder
$rel_url = $folder.'/'.$itemId;
// create directory
if(!is_dir($folder_url)) {
mkdir($folder_url);
}
}
// list of permitted file types, this is only images but documents can be added
$permitted = array('image/gif','image/jpeg','image/pjpeg','image/png','application/octet-stream');
// loop through and deal with the files;
$key = array();
$value = array();
foreach($formdata as $key => $value)
{
if($key == is_array($value))
{
$filename = str_replace(".", $replace , $value['name']);
}
// replace spaces with underscores
// assume filetype is false
$typeOK = false;
// check filetype is ok
foreach($permitted as $type)
{
if($key == is_array($value))
{
if($type == $value['type'])
{
$typeOK = true;
break;
}
}
}
// if file type ok upload the file
if($typeOK) {
// switch based on error code
if($key == is_array($value))
{
switch($value['error'])
{
case 0:
// check filename already exists
if(!file_exists($folder_url.'/'.$filename))
{
// create full filename
$full_url = $folder_url.'/'.$filename;
$url = $rel_url.'/'.$filename;
// upload the file
if($key == is_array($value))
{
$success = move_uploaded_file($value['tmp_name'], $url);
}
}
else
{
// create unique filename and upload file
// ini_set('date.timezone', 'Europe/London');
$now = date('Y-m-d-His');
$full_url = $folder_url.'/'.$now.$filename;
$url = $rel_url.'/'.$now.$filename;
if($key == is_array($value))
{
$success = move_uploaded_file($value['tmp_name'], $url);
}
}
// if upload was successful
if($success)
{
// save the url of the file
$result['urls'][] = $url;
}
else
{
$result['errors'][] = "Error uploaded $filename. Please try again.";
}
break;
case 3:
// an error occured
$result['errors'][] = "Error uploading $filename. Please try again.";
break;
default:
// an error occured
$result['errors'][] = "System error uploading $filename. Contact webmaster.";
break;
}
}
elseif($value['error'] == 4)
{
// no file was selected for upload
$result['nofiles'][] = "No file Selected";
}
else
{
// unacceptable file type
$result['errors'][] = "$filename cannot be uploaded. Acceptable file types: gif, jpg, png.";
}
}
}
return $result;
}

Resources