I am creating an elixir project to search for patterns in files.
I want to store those patterns a config files to allow for easy changes in the app.
My first idea is storing those files as exs files in the config folder in the mix project.
So, the questions are:
Is there any easy way to store the config in the files a a keyword list?
How would I load it in the app?
I see there are modules like File to read the file, but is there no standard way to parse keyword lists in elixir? I was thinking something similar as the yml files in Rails.
You can read keyword lists stored in a *.exs file, using Mix.Config.read(path). For writing Elixir terms to a *.exs file, you can use Inspect.Algebra.to_doc(%Inspect.Opts{pretty: true}) and write the resulting string content to a file using File.write. It's not as well formatted as if you did it by hand, but it's definitely still readable.
If you don't mind using Erlang terms, you can read and write those easily using :file.consult(path) and :file.write_file(:io_lib.fwrite('~p.\n', [config]), path) respectively.
Using Code.eval_file
Adding another option, is to evaluate the file as a code file, using Code.eval_file and get in return the result as an elixir construct.
Config file config1.ex:
%{configKey1: "configValue1", configKey2: "configValue2"}
Reading the file:
{content, _} = Code.eval_file("config1.ex")
*evaluating a code file has security consideration needs to take in mind.
Regarding using Mix.Config.read! in #bitwalker correct answer
the config file needs to be in a specific format of:
[
appName: [key1: "val1", key2: "val2"]
]
In the Mix.Config.read code, it try to validate the contents and expect a main keyword list [ {}, {}.. ] which includes keys that has value of type keyword list also.
The code is not long:
def validate!(config) do
if is_list(config) do
Enum.all?(config, fn
{app, value} when is_atom(app) ->
if Keyword.keyword?(value) do
true
else
raise ArgumentError,
"expected config for app #{inspect app} to return keyword list, got: #{inspect value}"
end
_ ->
false
end)
else
raise ArgumentError,
"expected config file to return keyword list, got: #{inspect config}"
end
end
We can circumvent and use a first key which is not atom, and then the validate stops but does not throw:
[
{"mockFirstKey", "mockValue"},
myKey1: "myValue1",
myKey2: "myValue2"
]
Related
Classic Snowflake Web UI and the new Snowsight are great at importing sql from a file but neither allows you to export sql to a file. Is there a workaround?
You can use an IDE to connect to snowflake and write queries. Then the scripts can be downloaded using IDE features and can sync with git repo as well.
dbeaver is one such IDE which supports snowflake :
https://hevodata.com/learn/dbeaver-snowflake/
The query pane is interactive so the obvious workaround will be:
CTRL + A (select all)
CTRL + C (copy)
<open_favourite_text_editor>
CTRL + P (paste)
CTRL + S (save)
This tool can help you while the team develops a native feature to export worksheets:
"Snowflake Snowsight Extensions wrap Snowsight features that do not have API or SQL alternatives, such as manipulating Dashboards and Worksheets, and retrieving Query Profile and step timings."
https://github.com/Snowflake-Labs/sfsnowsightextensions
Further explained on this post:
https://medium.com/snowflake/importing-and-exporting-snowsight-dashboards-and-worksheets-3cd8e34d29c8
For example, to save to a file within PowerShell:
PS > $dashboards | foreach {$_.SaveToFolder(“path/to/folder”)}
PS > $dashboards[0].SaveToFile(“path/to/folder/mydashboard.json”)
ETA: I'm adding this edit to the front because this is what actually worked.
Again, BSON was a dead end & punycode is irrelevant. I don't know why punycode is referenced in the metadata file; but my best guess is that they might use punycode to encode the worksheet name itself (though I'm not sure why that would be needed since it shouldn't need to be part of a URL).
After doing terrible things and trying a number of complex ways of dealing with escape character hell, I found that the actual encoding is very simple. It just works as an 8 bit encoding with anything that might cause problems escaped away (null, control codes, double quotes, etc.). To load, treat the file as a text file using an 8-bit encoding; extract the data as a JSON field, then re-encode that extracted data as that same encoding. I just used latin_1 to read; but it may not even matter which encoding you use as long as you are consistent and use the same one to re-encode. The encoded field will then be valid zlib compressed data.
I decided that I wanted to start from scratch so I needed to back the worksheets first and I made a Python script based on my findings above. Be warned that this may return even worksheets that you previously closed for good. After running this and verifying that backups were created, I just ran rm #~/worksheet_data/;, closed the tab & reopened it.
Here's the code (fill in the appropriate base directory location):
import os
from collections import OrderedDict
import configparser
from sqlalchemy import create_engine, exc
from snowflake.sqlalchemy import URL
import pathlib
import json
import zlib
import string
def format_filename(s: str) -> str: # From https://gist.github.com/seanh/93666
"""Take a string and return a valid filename constructed from the string.
Uses a whitelist approach: any characters not present in valid_chars are
removed. Also spaces are replaced with underscores.
Note: this method may produce invalid filenames such as ``, `.` or `..`
When I use this method I prepend a date string like '2009_01_15_19_46_32_'
and append a file extension like '.txt', so I avoid the potential of using
an invalid filename.
"""
valid_chars = "-_.() %s%s" % (string.ascii_letters, string.digits)
filename = ''.join(c for c in s if c in valid_chars)
# filename = filename.replace(' ','_') # I don't like spaces in filenames.
return filename
def trlng_dash(s: str) -> str:
"""Removes trailing character if present."""
return s[:-1] if s[-1] == '-' else s
sso_authenticate = True
# Assumes CLI config file exists.
config = configparser.ConfigParser()
home = pathlib.Path.home()
config_loc = home/'.snowsql/config' # Assumes it's set up from Snowflake CLI.
base_dir = home/r'{your Desired base directory goes here.}'
json_dir = base_dir/'json' # Location for your worksheet stage JSON files.
sql_dir = base_dir/'sql' # Location for your worksheets.
# Assumes CLI config file exists.
config.read(config_loc)
# Add connection parameters here (assumes CLI config exists).
# Using sso so only 2 are needed.
# If there's no config file, etc. enter by hand here (or however you want to do it).
connection_params = {
'account': config['connections']['accountname'],
'user': config['connections']['username'],
}
if sso_authenticate:
connection_params['authenticator'] = 'externalbrowser'
if config['connections'].get('password', None) is not None:
connection_params['password'] = config['connections']['password']
if config['connections'].get('rolename', None) is not None:
connection_params['role'] = config['connections']['rolename']
if locals().get('database', None) is not None:
connection_params['database'] = database
if locals().get('schema', None) is not None:
connection_params['schema'] = schema
sf_engine = create_engine(URL(**connection_params))
if not base_dir.exists():
base_dir.mkdir()
if not json_dir.exists():
json_dir.mkdir()
if not (sql_dir).exists():
sql_dir.mkdir()
with sf_engine.connect() as connection:
connection.execute(f'get #~/worksheet_data/ \'file://{str(json_dir.as_posix())}\';')
for file in [path for path in json_dir.glob('*') if path.is_file()]:
if file.suffix != '.json':
file.replace(file.with_suffix(file.suffix + '.json'))
with open(json_dir/'metadata.json', 'r') as metadata_file:
files_meta = json.load(metadata_file)
# List of files from metadata file will contain some empty worksheets.
files_description_orig = OrderedDict((file_key_value['name'], file_key_value) for file_key_value in sorted(files_meta['activeWorksheets'] + list(files_meta['inactiveWorksheets'].values()), key=lambda x: x['name']) if file_key_value['name'])
# files_description will only track non empty worksheets
files_description = files_description_orig.copy()
# Create updated files description filtering out empty worksheets.
for item in files_description_orig:
json_file = json_dir/f"{files_description_orig[item]['name']}.json"
# If a file didn't make it or was deleted by hand, we should
# remove from the filtered description & continue to the next item.
if not (json_file.exists() and json_file.is_file()):
del files_description[item]
continue
with open(json_file, 'r', encoding='latin_1') as f:
json_dat = json.load(f)
# If the file represents a worksheet with a body field, we want it.
if not json_dat['wsContents'].get('body'):
del files_description[item]
## Delete JSON files corresponsing to empty worksheets.
# f.close()
# try:
# (json_dir/f"{files_description_orig[item]['name']}.json").unlink()
# except:
# pass
# Produce a list of normalized filenames (no illegal or awkward characters).
file_names = set(
format_filename(trlng_dash(files_description[item]['encodedDetails']['scriptName']).strip())
for item in files_description)
# Add useful information to our files_description OrderedDict
for file_name in file_names:
repeats_cnt = 0
file_name_repeats = (
item
for item
in files_description
if file_name == format_filename(trlng_dash(files_description[item]['encodedDetails']['scriptName']).strip())
)
for file_uuid in file_name_repeats:
files_description[file_uuid]['normalizedName'] = file_name
files_description[file_uuid]['stemSuffix'] = '' if repeats_cnt == 0 else f'({repeats_cnt:0>2})'
repeats_cnt += 1
# Now we iterate on non-empty worksheets only.
for item in files_description:
json_file = json_dir/f"{files_description[item]['name']}.json"
with open(json_file, 'r', encoding='latin_1') as f:
json_dat = json.load(f)
body = json_dat['wsContents']['body']
body_bin = body.encode('latin_1')
body_txt = zlib.decompress(body_bin).decode('utf8')
sql_file = sql_dir/f"{files_description[item]['normalizedName']}{files_description[item]['stemSuffix']}.sql"
with open(sql_file, 'w') as sql_f:
sql_f.write(body_txt)
creation_stamp = files_description[item]['created']/1000
os.utime(sql_file, (creation_stamp,creation_stamp))
print('Done!')
As mentioned at Is there any option in snowflake to save or load worksheets? (and in Snowflake's own documentation), in the Classic UI, the worksheets are saved at the user stage under #~/worksheet_data/.
You can download it with a get command like:
get #~/worksheet_data/<name> file:///<your local location>; (though you might need quoting if running from Windows).
The problem is that I do not know how to access it programmatically. The downloaded files look like JSON but it is not valid JSON. The main key is "wsContents" and contains most of the worksheet information. Its value includes two subkeys, "encoding" and "body".
The "encoding" key denotes that gzip is being used. The "body" key seems to be the actual worksheet data which looks a lot like a straight binary representation of the compressed text data. As such, any JSON reader will choke on it.
If it is anything like that, I do not currently know how to access it programmatically using Python.
I do see that a JSON like format exists, BSON, that is bundled into PyMongo. Trying to use this on these files fails. I even tried bson.is_valid and it returns False so I am assuming that it means that these files in Snowflake are not actually BSON.
Edited to add: Again, BSON is a dead end.
Examining the "body" value as just binary data, the first two bytes of sample files do seem to correspond to default zlib compression (0x789c). However, attempting to run straight zlib.decompress on the slice created from that first byte to the last corresponding to the first & last characters of the "body" value results in the error:
Error - 3 while decompressing data: invalid code lengths set
This makes me think that the bytes there, as is, are at least partly garbage and still need some processing before they can be decompressed.
One clue that I failed to mention earlier is that the metadata file (called "metadata" and which serves as an inventory of the remaining files at the #~/worksheet_data/ location) declares that the files use the punycode encoding. However, I have not known how to use that information. The data in these files doesn't particularly look like what I feel punycode should look like nor does it particularly make sense to me that you would use punycode on binary data that is not meant to ever be used to directly generate text such as zlib compressed data.
I need to read a text file with readLines() and I've already found this question, but the code in the answers always uses some variation of javaClass; it seems to work only inside a class, while I'm using just a simple Kotlin file with no declared classes. Writing it like this is correct syntax-wise but it looks really ugly and it always returns null, so it must be wrong:
val lines = object {}.javaClass.getResource("file.txt")?.toURI()?.toPath()?.readLines()
Of course I could just specify the raw path like this, but I wonder if there's a better way:
val lines = File("src/main/resources/file.txt").readLines()
Thanks to this answer for providing the correct way to read the file. Currently, reading files from resources without using javaClass or similar constructs doesn't seem to be possible.
// use this if you're inside a class
val lines = this::class.java.getResourceAsStream("file.txt")?.bufferedReader()?.readLines()
// use this otherwise
val lines = object {}.javaClass.getResourceAsStream("file.txt")?.bufferedReader()?.readLines()
According to other similar questions I've found, the second way might also work within a lambda but I haven't tested it. Notice the need for the ?. operator and the lines?.let {} syntax needed from this point onward, because getResourceAsStream() returns null if no resource is found with the given name.
Kotlin doesn't have its own means of getting a resource, so you have to use Java's method Class.getResource. You should not assume that the resource is a file (i.e. don't use toPath) as it could well be an entry in a jar, and not a file on the file system. To read a resource, it is easier to get the resource as an InputStream and then read lines from it:
val lines = this::class.java.getResourceAsStream("file.txt").bufferedReader().readLines()
I'm not sure if my response attempts to answer your exact question, but perhaps you could do something like this:
I'm guessing in the final use case, the file names would be dynamic - Not statically declared. In which case, if you have access to or know the path to the folder, you could do something like this:
// Create an extension function on the String class to retrieve a list of
// files available within a folder. Though I have not added a check here
// to validate this, a condition can be added to assert if the extension
// called is executed on a folder or not
fun String.getFilesInFolder(): Array<out File>? = with(File(this)) { return listFiles() }
// Call the extension function on the String folder path wherever required
fun retrieveFiles(): Array<out File>? = [PATH TO FOLDER].getFilesInFolder()
Once you have a reference to the List<out File> object, you could do something like this:
// Create an extension function to read
fun File.retrieveContent() = readLines()
// You can can further expand this use case to conditionally return
// readLines() or entire file data using a buffered reader or convert file
// content to a Data class through GSON/whatever.
// You can use Generic Constraints
// Refer this article for possibilities
// https://kotlinlang.org/docs/generics.html#generic-constraints
// Then simply call this extension function after retrieving files in the folder.
listOfFiles?.forEach { singleFile -> println(singleFile.retrieveContent()) }
In order to have the same url that work for both Jar or in local, the url (or path) needs to be a relative path from the repository root.
..meaning, the location of your file or folder from your src folder.
could be "/main/resources/your-folder/" or "/client/notes/somefile.md"
The url must be a relative path from the repository root.
it must be "src/main/resources/your-folder/" or "src/client/notes/somefile.md"
Now you get the drill, and luckily for Intellij Idea users, you can get the correct path with a right-click on the folder or file -> copy Path/Reference.. -> Path From Repository Root (this is it)
Last, paste it and do your thing.
Trying to put together a file diff route... could someone help? here is what I have ->
CsvDataFormat csv = new CsvDataFormat();
csv.setDelimiter(",");
from("file:inputdir?delete=true&sortBy=ignoreCase:file:name")
.unmarshal(csv)
.pollEnrich("file:backup?fileName=test.csv&sendEmptyMessageWhenIdle=true")
.unmarshal(csv)
// Need to aggregate here!!!!
.log("test");
A csv file gets dropped in the /input directory and then a backup file is consumed from the /backup directory. I would like to compare these two files and output the difference.
This is not a specific Camel problem. In order to solve this problem you may implement a diff functionality on your own, or you may use an existing library such as java-diff-utils.
Pseudocode:
// read file 1 into a list "list1"
// read file 2 into a list "list2"
// use java-diff-utils to calculate the difference
Patch patch = DiffUtils.diff(list1, list2);
I'm currently trying to attach image files to a model directly from a zip file (i.e. without first saving them on a disk). It seems like there should be a clearer way of converting a ZipEntry to a Tempfile or File that can be stored in memory to be passed to another method or object that knows what to do with it.
Here's my code:
def extract (file = nil)
Zip::ZipFile.open(file) { |zip_file|
zip_file.each { |image|
photo = self.photos.build
# photo.image = image # this doesn't work
# photo.image = File.open image # also doesn't work
# photo.image = File.new image.filename
photo.save
}
}
end
But the problem is that photo.image is an attachment (via paperclip) to the model, and assigning something as an attachment requires that something to be a File object. However, I cannot for the life of me figure out how to convert a ZipEntry to a File. The only way I've seen of opening or creating a File is to use a string to its path - meaning I have to extract the file to a location. Really, that just seems silly. Why can't I just extract the ZipEntry file to the output stream and convert it to a File there?
So the ultimate question: Can I extract a ZipEntry from a Zip file and turn it directly into a File object (or attach it directly as a Paperclip object)? Or am I stuck actually storing it on the hard drive before I can attach it, even though that version will be deleted in the end?
UPDATE
Thanks to blueberry fields, I think I'm a little closer to my solution. Here's the line of code that I added, and it gives me the Tempfile/File that I need:
photo.image = zip_file.get_output_stream image
However, my Photo object won't accept the file that's getting passed, since it's not an image/jpeg. In fact, checking the content_type of the file shows application/x-empty. I think this may be because getting the output stream seems to append a timestamp to the end of the file, so that it ends up looking like imagename.jpg20110203-20203-hukq0n. Edit: Also, the tempfile that it creates doesn't contain any data and is of size 0. So it's looking like this might not be the answer.
So, next question: does anyone know how to get this to give me an image/jpeg file?
UPDATE:
I've been playing around with this some more. It seems output stream is not the way to go, but rather an input stream (which is which has always kind of confused me). Using get_input_stream on the ZipEntry, I get the binary data in the file. I think now I just need to figure out how to get this into a Paperclip attachment (as a File object). I've tried pushing the ZipInputStream directly to the attachment, but of course, that doesn't work. I really find it hard to believe that no one has tried to cast an extracted ZipEntry as a File. Is there some reason that this would be considered bad programming practice? It seems to me like skipping the disk write for a temp file would be perfectly acceptable and supported in something like Zip archive management.
Anyway, the question still stands:
Is there a way of converting an Input Stream to a File object (or Tempfile)? Preferably without having to write to a disk.
Try this
Zip::ZipFile.open(params[:avatar].path) do |zipfile|
zipfile.each do |entry|
filename = entry.name
basename = File.basename(filename)
tempfile = Tempfile.new(basename)
tempfile.binmode
tempfile.write entry.get_input_stream.read
user = User.new
user.avatar = {
:tempfile => tempfile,
:filename => filename
}
user.save
end
end
Check out the get_input_stream and get_output_stream messages on ZipFile.
I'm researching hours and hours, but I could not find any clear, efficient way to make it :/
I have a codeigniter base website in English and I have to add a Polish language now. What is the best way to make my site in 2 language depending visitor selection?
is there any way to create array files for each language and call them in view files depends on Session from lang selection? I don't wanna use database.
Appreciate helps! I'm running out of deadline :/ thanks!!
Have you seen CodeIgniter's Language library?
The Language Class provides functions
to retrieve language files and lines
of text for purposes of internationalization.
In your CodeIgniter system folder you'll
find one called language containing sets
of language files. You can create your
own language files as needed in order
to display error and other messages in
other languages.
Language files are typically stored in
your system/language directory. Alternately
you can create a folder called language
inside your application folder and store
them there. CodeIgniter will look first
in your application/language directory.
If the directory does not exist or the
specified language is not located there
CI will instead look in your global
system/language folder.
In your case...
you need to create a polish_lang.php and english_lang.php inside application/language/polish
then create your keys inside that file (e.g. $lang['hello'] = "Witaj";
then load it in your controller like $this->lang->load('polish_lang', 'polish');
then fetch the line like $this->lang->line('hello'); Just store the return value of this function in a variable so you can use it in your view.
Repeat the steps for the english language and all other languages you need.
Also to add the language to the session, I would define some constants for each language, then make sure you have the session library autoloaded in config/autoload.php, or you load it whenever you need it. Add the users desired language to the session:
$this->session->set_userdata('language', ENGLISH);
Then you can grab it anytime like this:
$language = $this->session->userdata('language');
In the controller add following lines when you make the cunstructor
i.e, after
parent::Controller();
add below lines
$this->load->helper('lang_translate');
$this->lang->load('nl_site', 'nl'); // ('filename', 'directory')
create helper file lang_translate_helper.php with following function and put it in directory system\application\helpers
function label($label, $obj)
{
$return = $obj->lang->line($label);
if($return)
echo $return;
else
echo $label;
}
for each of the language, create a directory with language abbrevation like en, nl, fr, etc., under
system\application\languages
create language file in above (respective) directory which will contain $lang array holding pairs label=>language_value as given below
nl_site_lang.php
$lang['welcome'] = 'Welkom';
$lang['hello word'] = 'worde Witaj';
en_site_lang.php
$lang['welcome'] = 'Welcome';
$lang['hello word'] = 'Hello Word';
you can store multiple files for same language with differently as per the requirement
e.g, if you want separate language file for managing backend (administrator section) you can use it in controller as $this->lang->load('nl_admin', 'nl');
nl_admin_lang.php
$lang['welcome'] = 'Welkom';
$lang['hello word'] = 'worde Witaj';
and finally
to print the label in desired language, access labels as below in view
label('welcome', $this);
OR
label('hello word', $this);
note the space in hello & word you can use it like this way as well :)
whene there is no lable defined in the language file, it will simply print it what you passed to the function label.
I second Randell's answer.
However, one could always integrate a GeoIP such as http://www.maxmind.com/app/php
or http://www.ipinfodb.com/. Then you can save the results with the codeigniter session class.
If you want to use the ipinfodb.com api You can add the ip2locationlite.class.php file to your codeigniter application library folder and then create a model function to do whatever geoip logic you need for your application, such as:
function geolocate()
{
$ipinfodb = new ipinfodb;
$ipinfodb->setKey('API KEY');
//Get errors and locations
$locations = $ipinfodb->getGeoLocation($this->input->ip_address());
$errors = $ipinfodb->getError();
//Set geolocation cookie
if(empty($errors))
{
foreach ($locations as $field => $val):
if($field === 'CountryCode')
{
$place = $val;
}
endforeach;
}
return $place;
}
For easier use CI have updated this so you can just use
$this->load->helper('language');
and to translate text
lang('language line');
and if you want to warp it inside label then use optional parameter
lang('language line', 'element id');
This will output
// becomes <label for="form_item_id">language_key</label>
For good reading
http://ellislab.com/codeigniter/user-guide/helpers/language_helper.html
I've used Wiredesignz's MY_Language class with great success.
I've just published it on github, as I can't seem to find a trace of it anywhere.
https://github.com/meigwilym/CI_Language
My only changes are to rename the class to CI_Lang, in accordance with the new v2 changes.
When managing the actual files, things can get out of sync pretty easily unless you're really vigilant. So we've launched a (beta) free service called String which allows you to keep track of your language files easily, and collaborate with translators.
You can either import existing language files (in PHP array, PHP Define, ini, po or .strings formats) or create your own sections from scratch and add content directly through the system.
String is totally free so please check it out and tell us what you think.
It's actually built on Codeigniter too! Check out the beta at http://mygengo.com/string
Follow this https://github.com/EllisLab/CodeIgniter/wiki/CodeIgniter-2.1-internationalization-i18n
its simple and clear, also check out the document # http://ellislab.com/codeigniter/user-guide/libraries/language.html
its way simpler than
I am using such code in config.php:
$lang = 'ru'; // this language will be used if there is no any lang information from useragent (for example, from command line, wget, etc...
if (!empty($_SERVER['HTTP_ACCEPT_LANGUAGE'])) $lang = substr($_SERVER['HTTP_ACCEPT_LANGUAGE'],0,2);
$tmp_value = $_COOKIE['language'];
if (!empty($tmp_value)) $lang = $tmp_value;
switch ($lang)
{
case 'ru':
$config['language'] = 'russian';
setlocale(LC_ALL,'ru_RU.UTF-8');
break;
case 'uk':
$config['language'] = 'ukrainian';
setlocale(LC_ALL,'uk_UA.UTF-8');
break;
case 'foo':
$config['language'] = 'foo';
setlocale(LC_ALL,'foo_FOO.UTF-8');
break;
default:
$config['language'] = 'english';
setlocale(LC_ALL,'en_US.UTF-8');
break;
}
.... and then i'm using usualy internal mechanizm of CI
o, almost forget! in views i using buttons, which seting cookie 'language' with language, prefered by user.
So, first this code try to detect "preffered language" setted in user`s useragent (browser). Then code try to read cookie 'language'. And finaly - switch sets language for CI-application
you can make a function like this
function translateTo($language, $word) {
define('defaultLang','english');
if (isset($lang[$language][$word]) == FALSE)
return $lang[$language][$word];
else
return $lang[defaultLang][$word];
}
Friend, don't worry, if you have any application installed built in codeigniter and you wanna add some language pack just follow these steps:
1. Add language files in folder application/language/arabic (i add arabic lang in sma2 built in ci)
2. Go to the file named setting.php in application/modules/settings/views/setting.php. Here you find the array
<?php /*
$lang = array (
'english' => 'English',
'arabic' => 'Arabic', // i add this here
'spanish' => 'Español'
Now save and run the application. It's worked fine.