Solved, now working example!
I have a set of time series that is populated in a folder structure as follows:
TimeSeriesData\Config_000000\seed_001\Aggregate_Quantity.txt
…
TimeSeriesData\Config_000058\seed_010\Aggregate_Quantity.txt
And in each file there is a first line containing a character, followed by lines containing the data. E.g.
share
-21.75
-20.75
…
Now, I would like to import all those files (at best, without specifying ex post the number of configs and seeds, at worst with supplying it) into single time series of the like: “AggQuant_config_seed” where config relates to the config ID and seed to the seed id.
I tried the following (using the non-preferred way), but “parsing” the “path” does not work / I do not know how to do it.
string base = "TimeSeriesData/Config_"
string middle = "/seed_"
string endd = "/Aggregate_Quantity.txt"
string path = ""
loop for (i=0;i<=58;i+=1)
loop for (j=1;j<=10;j+=1)
path = ""
sprintf path "%s%06d%s%03d%s",base,i,middle,j,endd
append #path #will be named share
rename share Agg_Q_$i_$j #rename
endloop
endloop
To sum up the problem, the following does not work:
string path="somwhere/a_file.txt" #string holding path
#wrong: append $path #use append on string
append #path #works!
And if possible, a way to search recursively through a set of folders, using the folder information together with file-information for the variable name, would be nice. Is that possible within gretl?
I realize that in many cases I would like to refer to a specific “help” section like those for functions and commands, but for operators instead (like “$”, which is obviously wrong here).
Related
Classic Snowflake Web UI and the new Snowsight are great at importing sql from a file but neither allows you to export sql to a file. Is there a workaround?
You can use an IDE to connect to snowflake and write queries. Then the scripts can be downloaded using IDE features and can sync with git repo as well.
dbeaver is one such IDE which supports snowflake :
https://hevodata.com/learn/dbeaver-snowflake/
The query pane is interactive so the obvious workaround will be:
CTRL + A (select all)
CTRL + C (copy)
<open_favourite_text_editor>
CTRL + P (paste)
CTRL + S (save)
This tool can help you while the team develops a native feature to export worksheets:
"Snowflake Snowsight Extensions wrap Snowsight features that do not have API or SQL alternatives, such as manipulating Dashboards and Worksheets, and retrieving Query Profile and step timings."
https://github.com/Snowflake-Labs/sfsnowsightextensions
Further explained on this post:
https://medium.com/snowflake/importing-and-exporting-snowsight-dashboards-and-worksheets-3cd8e34d29c8
For example, to save to a file within PowerShell:
PS > $dashboards | foreach {$_.SaveToFolder(“path/to/folder”)}
PS > $dashboards[0].SaveToFile(“path/to/folder/mydashboard.json”)
ETA: I'm adding this edit to the front because this is what actually worked.
Again, BSON was a dead end & punycode is irrelevant. I don't know why punycode is referenced in the metadata file; but my best guess is that they might use punycode to encode the worksheet name itself (though I'm not sure why that would be needed since it shouldn't need to be part of a URL).
After doing terrible things and trying a number of complex ways of dealing with escape character hell, I found that the actual encoding is very simple. It just works as an 8 bit encoding with anything that might cause problems escaped away (null, control codes, double quotes, etc.). To load, treat the file as a text file using an 8-bit encoding; extract the data as a JSON field, then re-encode that extracted data as that same encoding. I just used latin_1 to read; but it may not even matter which encoding you use as long as you are consistent and use the same one to re-encode. The encoded field will then be valid zlib compressed data.
I decided that I wanted to start from scratch so I needed to back the worksheets first and I made a Python script based on my findings above. Be warned that this may return even worksheets that you previously closed for good. After running this and verifying that backups were created, I just ran rm #~/worksheet_data/;, closed the tab & reopened it.
Here's the code (fill in the appropriate base directory location):
import os
from collections import OrderedDict
import configparser
from sqlalchemy import create_engine, exc
from snowflake.sqlalchemy import URL
import pathlib
import json
import zlib
import string
def format_filename(s: str) -> str: # From https://gist.github.com/seanh/93666
"""Take a string and return a valid filename constructed from the string.
Uses a whitelist approach: any characters not present in valid_chars are
removed. Also spaces are replaced with underscores.
Note: this method may produce invalid filenames such as ``, `.` or `..`
When I use this method I prepend a date string like '2009_01_15_19_46_32_'
and append a file extension like '.txt', so I avoid the potential of using
an invalid filename.
"""
valid_chars = "-_.() %s%s" % (string.ascii_letters, string.digits)
filename = ''.join(c for c in s if c in valid_chars)
# filename = filename.replace(' ','_') # I don't like spaces in filenames.
return filename
def trlng_dash(s: str) -> str:
"""Removes trailing character if present."""
return s[:-1] if s[-1] == '-' else s
sso_authenticate = True
# Assumes CLI config file exists.
config = configparser.ConfigParser()
home = pathlib.Path.home()
config_loc = home/'.snowsql/config' # Assumes it's set up from Snowflake CLI.
base_dir = home/r'{your Desired base directory goes here.}'
json_dir = base_dir/'json' # Location for your worksheet stage JSON files.
sql_dir = base_dir/'sql' # Location for your worksheets.
# Assumes CLI config file exists.
config.read(config_loc)
# Add connection parameters here (assumes CLI config exists).
# Using sso so only 2 are needed.
# If there's no config file, etc. enter by hand here (or however you want to do it).
connection_params = {
'account': config['connections']['accountname'],
'user': config['connections']['username'],
}
if sso_authenticate:
connection_params['authenticator'] = 'externalbrowser'
if config['connections'].get('password', None) is not None:
connection_params['password'] = config['connections']['password']
if config['connections'].get('rolename', None) is not None:
connection_params['role'] = config['connections']['rolename']
if locals().get('database', None) is not None:
connection_params['database'] = database
if locals().get('schema', None) is not None:
connection_params['schema'] = schema
sf_engine = create_engine(URL(**connection_params))
if not base_dir.exists():
base_dir.mkdir()
if not json_dir.exists():
json_dir.mkdir()
if not (sql_dir).exists():
sql_dir.mkdir()
with sf_engine.connect() as connection:
connection.execute(f'get #~/worksheet_data/ \'file://{str(json_dir.as_posix())}\';')
for file in [path for path in json_dir.glob('*') if path.is_file()]:
if file.suffix != '.json':
file.replace(file.with_suffix(file.suffix + '.json'))
with open(json_dir/'metadata.json', 'r') as metadata_file:
files_meta = json.load(metadata_file)
# List of files from metadata file will contain some empty worksheets.
files_description_orig = OrderedDict((file_key_value['name'], file_key_value) for file_key_value in sorted(files_meta['activeWorksheets'] + list(files_meta['inactiveWorksheets'].values()), key=lambda x: x['name']) if file_key_value['name'])
# files_description will only track non empty worksheets
files_description = files_description_orig.copy()
# Create updated files description filtering out empty worksheets.
for item in files_description_orig:
json_file = json_dir/f"{files_description_orig[item]['name']}.json"
# If a file didn't make it or was deleted by hand, we should
# remove from the filtered description & continue to the next item.
if not (json_file.exists() and json_file.is_file()):
del files_description[item]
continue
with open(json_file, 'r', encoding='latin_1') as f:
json_dat = json.load(f)
# If the file represents a worksheet with a body field, we want it.
if not json_dat['wsContents'].get('body'):
del files_description[item]
## Delete JSON files corresponsing to empty worksheets.
# f.close()
# try:
# (json_dir/f"{files_description_orig[item]['name']}.json").unlink()
# except:
# pass
# Produce a list of normalized filenames (no illegal or awkward characters).
file_names = set(
format_filename(trlng_dash(files_description[item]['encodedDetails']['scriptName']).strip())
for item in files_description)
# Add useful information to our files_description OrderedDict
for file_name in file_names:
repeats_cnt = 0
file_name_repeats = (
item
for item
in files_description
if file_name == format_filename(trlng_dash(files_description[item]['encodedDetails']['scriptName']).strip())
)
for file_uuid in file_name_repeats:
files_description[file_uuid]['normalizedName'] = file_name
files_description[file_uuid]['stemSuffix'] = '' if repeats_cnt == 0 else f'({repeats_cnt:0>2})'
repeats_cnt += 1
# Now we iterate on non-empty worksheets only.
for item in files_description:
json_file = json_dir/f"{files_description[item]['name']}.json"
with open(json_file, 'r', encoding='latin_1') as f:
json_dat = json.load(f)
body = json_dat['wsContents']['body']
body_bin = body.encode('latin_1')
body_txt = zlib.decompress(body_bin).decode('utf8')
sql_file = sql_dir/f"{files_description[item]['normalizedName']}{files_description[item]['stemSuffix']}.sql"
with open(sql_file, 'w') as sql_f:
sql_f.write(body_txt)
creation_stamp = files_description[item]['created']/1000
os.utime(sql_file, (creation_stamp,creation_stamp))
print('Done!')
As mentioned at Is there any option in snowflake to save or load worksheets? (and in Snowflake's own documentation), in the Classic UI, the worksheets are saved at the user stage under #~/worksheet_data/.
You can download it with a get command like:
get #~/worksheet_data/<name> file:///<your local location>; (though you might need quoting if running from Windows).
The problem is that I do not know how to access it programmatically. The downloaded files look like JSON but it is not valid JSON. The main key is "wsContents" and contains most of the worksheet information. Its value includes two subkeys, "encoding" and "body".
The "encoding" key denotes that gzip is being used. The "body" key seems to be the actual worksheet data which looks a lot like a straight binary representation of the compressed text data. As such, any JSON reader will choke on it.
If it is anything like that, I do not currently know how to access it programmatically using Python.
I do see that a JSON like format exists, BSON, that is bundled into PyMongo. Trying to use this on these files fails. I even tried bson.is_valid and it returns False so I am assuming that it means that these files in Snowflake are not actually BSON.
Edited to add: Again, BSON is a dead end.
Examining the "body" value as just binary data, the first two bytes of sample files do seem to correspond to default zlib compression (0x789c). However, attempting to run straight zlib.decompress on the slice created from that first byte to the last corresponding to the first & last characters of the "body" value results in the error:
Error - 3 while decompressing data: invalid code lengths set
This makes me think that the bytes there, as is, are at least partly garbage and still need some processing before they can be decompressed.
One clue that I failed to mention earlier is that the metadata file (called "metadata" and which serves as an inventory of the remaining files at the #~/worksheet_data/ location) declares that the files use the punycode encoding. However, I have not known how to use that information. The data in these files doesn't particularly look like what I feel punycode should look like nor does it particularly make sense to me that you would use punycode on binary data that is not meant to ever be used to directly generate text such as zlib compressed data.
I wonder how to convert a dm3 file into .jpg/jpeg images? there is test annotation and scale bar on the image. I setup a script but it always show that "the format cannot contain the data to be saved". This can be done via file/batch convert function. So how to realize the same function in script？ Thanks
image test:=IntegerImage("test",2,1,100,100)
test.ShowImage()
image frontimage:=GetFrontImage()
string filename=getname(frontimage)
imagedisplay disp = frontImage.ImageGetImageDisplay(0)
disp.applydatabar()
ImageDocument frontDoc = GetFrontImageDocument()
string directoryname, pathname
number length
if(!SaveAsDialog("","Do Not Change Me",directoryname)) exit(0)
length=len(directoryname)-16
directoryname=mid(directoryname,0,length)
pathname=directoryname+filename
frontDoc.ImageDocumentSaveToFile( "JPG Format", pathname )
To convert to jpg you have to use "JPEG/JFIF Format" as the handler (=format).
It has to be exactly this string in the ImageDocument.ImageDocumentSaveToFile() function. Other formats are mentioned in the help (F1 > Scripting > Objects > Document Object Model > ImageDocument Object > ImageDocumentSaveToFile() function). Those are (for example):
'Gatan Format'
'Gatan 3 Format'
'GIF Format'
'BMP Format'
'JPEG/JFIF Format'
'Enhanced Metafile Format'
In your code you are using the SaveAsDialog() to get a directory. This is not necessary. You can use GetDirectoryDialog() to get a directory. This saves you the name operation for the directoryname and avoids problems when users do change your filename.
Also for concatinating paths I prefer using PathConcatenate(). On the first hand this makes your code a lot more readable since its name tells what you are doing. On the other hand this also takes care of the directory ending with \ or not and other path related things.
The following code is what I think you need:
Image test := IntegerImage("test", 2, 1, 100, 100);
test.ShowImage();
Image frontimage := GetFrontImage();
ImageDisplay disp = frontImage.ImageGetImageDisplay(0);
disp.applydatabar();
ImageDocument frontDoc = GetFrontImageDocument();
string directoryname;
if(!GetDirectoryDialog("Select directory", "C:\\\\", directoryname)){
// ↑
// You can of course use something else as the start point for selection here
exit(0);
}
string filename = GetName(frontimage);
string pathname = directoryname.PathConcatenate(filename);
frontDoc.ImageDocumentSaveToFile("JPEG/JFIF Format", pathname);
This answer is correct and should be accepted. Your problem is the wrong file-type string. You want to use "JPEG/JFIF Format"
A bit more general information on image file saving in DigitalMicrograph.
One doesn't save images but always imageDocuments that can contain one, more, or even zero image objects in them. Script-commands that save an image like SaveAsGatan() really just call things like: ImageGetOrCreateImageDocument().ImageDocumentSaveToFile()
The difference doesn't really matter for simple one-image-in-document type images, but it can make a difference when there are multiple images in a document, or when a single image is displayed multiple times simultaneously (which can be done.) So it is always good to know what "really" goes on.
ImageDocuments contain some properties relating to saving:
A save format (“Gatan Format”, “TIFF Format”, …)
Default value: What it was opened with, or last used save-format in case of creation
Script commands: ImageDocumentGetCurrentFileSaveFormat() ImageDocumentSetCurrentFileSaveFormat()
A current file path:
Default value: What it was opened from, or empty
Script commands: ImageDocumentGetCurrentFile() ImageDocumentSetCurrentFile()
A dirty-state:
Default value: clean when opened, dirty when created
Script commands: ImageDocumentIsDirty() ImageDocumentClean()
A linked-to-file state:
Default value: true when opened, false when created
Script commands: ImageDocumentIsLinkedToFile()
There are two ways of saving an imageDocument:
Saving the current document itself to disc:
void ImageDocumentSave( ImageDocument imgDoc, Number save_style ) This utilizes the current properties of the imageDocument to save it to current path in current format, marking it clean in the process. The save_style parameter determines how the program deals with missing info:
0 = never ask for path
1 = ask if not linked (or empty path)
2 = always ask
Saving a copy of the current document to disc:
void ImageDocumentSaveToFile( ImageDocument imgDoc, String handler, String fileName ) This makes a copy and save the file under provided path in the provided format. The imageDocument in memory does not change its properties. Most noticeable: It does not become clean, and it is not linked to the provided file on disc. The filename parameter specifies the saving location including the filename. If a file extension is provided, it has to match the file-format, but it can be left out. The handler parameter specified the file-format and can be anything GMS currently supports, such as:
Gatan Format
Gatan 3 Format
GIF Format
BMP Format
JPEG/JFIF Format
Enhanced Metafile Format
In short:
To save the currently opened imageDocument with a different format, you would want to do:
imageDocument doc = GetFrontImageDocument()
doc.ImageDocumentSetCurrentFileSaveFormat("TIFF Format")
doc.ImageDocumentSave(0)
While to just save a copy of the current state you would use:
imageDocument doc = GetFrontImageDocument()
string path = doc.ImageDocumentGetCurrentFile() // full path including extension!
path = PathExtractDirectory(path,0) + PathExtractBaseName(path,0) // path without file extension
doc.ImageDocumentSaveToFile("TIFF Format", path )
I am trying to write a GUI that will display the name of the sketch it was generated from using a simple text() command. However, I am running into trouble getting any of the general JS solutions to work for me. Many solutions I have found use the filename reserved word but that does not seem to be reserved in Processing 3.5.4. I have also tried parsing the strings using a similar method to what can be found here. I am very new to processing and this is only my 2nd attempt at using Processing.
Any advice would be greatly appreciated
You can get the path (as a string) to the sketch with sketchPath().
From there you could either parse the string (pull off everything after the last slash) to get the sketch name, or you can use sketchFile() to get a reference to the file itself and get the name from there:
String path = sketchPath();
File file = sketchFile(path);
String sketchName = file.getName();
println(sketchName);
You could combine this all into one line like so:
String sketchName = sketchFile(sketchPath()).getName();
Problem:
Hello, I have been struggling recently in my programming endeavours. I have managed to receive the output below from Google Speech to Text, but I cannot figure out how draw data from this block.
Excerpt 1:
[VoiceMain]: Successfully initialized
{"result":[]}
{"result":[{"alternative":[{"transcript":"hello","confidence":0.46152416},{"transcript":"how low"},{"transcript":"how lo"},{"transcript":"how long"},{"transcript":"Polo"}],"final":true}],"result_index":0}
[VoiceMain]: Successfully initialized
{"result":[]}
{"result":[{"alternative":[{"transcript":"hello"},{"transcript":"how long"},{"transcript":"how low"},{"transcript":"howlong"}],"final":true}],"result_index":0}
Objective:
My goal is to extract the string "hello" (without the quotation marks) from the first transcript of each block and set it equal to a variable. The problem arises when I do not know what the phrase will be. Instead of "hello", the phrase may be a string of any length. Even if it is a different string, I would still like to set it to the same variable to which the phrase "hello" would have been set to.
Furthermore, I would like to extract the number after the word "confidence". In this case, it is 0.46152416. Data type does not matter for the confidence variable. The confidence variable appears to be more difficult to extract from the blocks because it may or may not be present. If it is not present, it must be ignored. If it is present however, it must be detected and stored as a variable.
Also please note that this text block is stored within a file named "CurlOutput.txt".
All help or advice related to solving this problem is greatly appreciated.
You could do this with regex, but then I am assuming you will want to use this as a dict later in your code. So here is a python approach to building this result as a dictionary.
import json
with open('CurlOutput.txt') as f:
lines = f.read().splitlines()
flag = '{"result":[]} '
for line in lines: # Loop through each lin in file
if flag in line: # check if this is a line with data on it
results = json.loads(line.replace(flag, ''))['result'] # Load data as a dict
# If you just want to change first index of alternative
# results[0]['alternative'][0]['transcript'] = 'myNewString'
# If you want to check all alternative for confidence and transcript
for result in results[0]['alternative']: # Loop over each alternative
transcript = result['transcript']
confidence = None
if 'confidence' in result:
confidence = result['confidence']
# now do whatever you want with confidence and transcript.
I would like to get all the files that a sub-folder holds in a string array.
So, I have tried something like the following:
var IOstore = IsolatedStorageFile.GetUserStoreForApplication();
string searchpath = System.IO.Path.Combine("product", ProductName);
string filesInSubDirs[] = IOstore.GetFileNames(searchpath);
But I got all the files in the "product" folder. I have also tried with "productname" only as the parameter.
Thanks for your help.
The search pattern for a sub-folder needs to include "*.*" at the end to pattern match any file, which would make your code something like the following:
var IOstore = IsolatedStorageFile.GetUserStoreForApplication();
string searchpath = System.IO.Path.Combine("product", ProductName);
searchpath = string.Format("{0}\\*.*", searchpath);
string filesInSubDirs[] = IOstore.GetFileNames(searchpath);
Something you might want to try. (this is sort of a left field answer, sorry). In my dropbox client http://sharpdropbox.codeplex.com/) I have a set of facades for System.IO.File, System.IO.FileInfo, System.IO.Directory, and System.IO.DirectoryInfo. They work pretty good and I have tested them.
Basically, you add a Using or Import for System.IO.IsolatedStorage and then PSFile, PSDirectory, PSFileInfo, or PSDirectoryInfo. It's saved me from having to remember all the nuances... for instance if you are querying a directory, it knows to add a slash, etc. BTW, the "PS" prefix stands for "Persisted Storage" which is what IsolatedStorage is sometimes called (starting them with an "I" implies they are interfaces.. and having no prefix makes things even more confusing).
Anyway, you can grab the code from source or I believe the last release had the DLLs for them (it's called something like "IsolatedStorageFacade-WP7")