I'm trying to use GML produced by Mapserver to create popup with openlayers, if datasource comes from shp file, everything is working fine, however, postgis datasource brings exception, like "This may be due to a corruption of the heap, which indicates a bug in HTTPFormServer.exe or any of the DLLs it has loaded." Another, if I just read layer data from postgis to render label instead of as WFS, it's working fine as well
who can help tell what happend?
environment: mapserver5.6.1, postgis8.4, openlayers2.10
for shp data source, works
LAYER
NAME poi_point
METADATA
"wfs_title" "poi_point" ##REQUIRED
"wfs_typename" "poi_point" ## REQUIRED
"gml_include_items" "all" ## Optional (serves all attributes for layer)
"gml_featureid" "ID" ## REQUIRED
"gml_geometries" "geometry"
"gml_geometry_type" "point"
END
PROJECTION
"proj=latlong"
"ellps=GRS80"
"datum=NAD27"
END
DATA poi_point
STATUS ON
TYPE POINT
DUMP TRUE
MAXSCALEDENOM 2400
CLASS
STYLE
COLOR 0 0 0
OUTLINECOLOR 255 255 255
END
END
END
for postgis datasource, exception happend
LAYER
NAME poi_point
DATA "the_geom from poi_point"
METADATA
"wfs_title" "poi_point" ##REQUIRED
"wfs_typename" "poi_point" ## REQUIRED
"gml_include_items" "all" ## Optional (serves all attributes for layer)
"gml_featureid" "ID" ## REQUIRED
"gml_geometries" "geometry"
"gml_geometry_type" "point"
END
PROJECTION
"proj=latlong"
"ellps=GRS80"
"datum=NAD27"
END
CONNECTION "user=postgres password=springtime dbname=postgis host=localhost port=5432"
CONNECTIONTYPE postgis
STATUS ON
TYPE POINT
DUMP TRUE
MAXSCALEDENOM 2400
CLASS
STYLE
COLOR 0 0 0
OUTLINECOLOR 255 255 255
END
END
END
I find it's a bug in mapserver-5.6.1, that is, allocated memory is not enough in msPostGISBuildSQLBox, enlarge is ok.
Related
Classic Snowflake Web UI and the new Snowsight are great at importing sql from a file but neither allows you to export sql to a file. Is there a workaround?
You can use an IDE to connect to snowflake and write queries. Then the scripts can be downloaded using IDE features and can sync with git repo as well.
dbeaver is one such IDE which supports snowflake :
https://hevodata.com/learn/dbeaver-snowflake/
The query pane is interactive so the obvious workaround will be:
CTRL + A (select all)
CTRL + C (copy)
<open_favourite_text_editor>
CTRL + P (paste)
CTRL + S (save)
This tool can help you while the team develops a native feature to export worksheets:
"Snowflake Snowsight Extensions wrap Snowsight features that do not have API or SQL alternatives, such as manipulating Dashboards and Worksheets, and retrieving Query Profile and step timings."
https://github.com/Snowflake-Labs/sfsnowsightextensions
Further explained on this post:
https://medium.com/snowflake/importing-and-exporting-snowsight-dashboards-and-worksheets-3cd8e34d29c8
For example, to save to a file within PowerShell:
PS > $dashboards | foreach {$_.SaveToFolder(“path/to/folder”)}
PS > $dashboards[0].SaveToFile(“path/to/folder/mydashboard.json”)
ETA: I'm adding this edit to the front because this is what actually worked.
Again, BSON was a dead end & punycode is irrelevant. I don't know why punycode is referenced in the metadata file; but my best guess is that they might use punycode to encode the worksheet name itself (though I'm not sure why that would be needed since it shouldn't need to be part of a URL).
After doing terrible things and trying a number of complex ways of dealing with escape character hell, I found that the actual encoding is very simple. It just works as an 8 bit encoding with anything that might cause problems escaped away (null, control codes, double quotes, etc.). To load, treat the file as a text file using an 8-bit encoding; extract the data as a JSON field, then re-encode that extracted data as that same encoding. I just used latin_1 to read; but it may not even matter which encoding you use as long as you are consistent and use the same one to re-encode. The encoded field will then be valid zlib compressed data.
I decided that I wanted to start from scratch so I needed to back the worksheets first and I made a Python script based on my findings above. Be warned that this may return even worksheets that you previously closed for good. After running this and verifying that backups were created, I just ran rm #~/worksheet_data/;, closed the tab & reopened it.
Here's the code (fill in the appropriate base directory location):
import os
from collections import OrderedDict
import configparser
from sqlalchemy import create_engine, exc
from snowflake.sqlalchemy import URL
import pathlib
import json
import zlib
import string
def format_filename(s: str) -> str: # From https://gist.github.com/seanh/93666
"""Take a string and return a valid filename constructed from the string.
Uses a whitelist approach: any characters not present in valid_chars are
removed. Also spaces are replaced with underscores.
Note: this method may produce invalid filenames such as ``, `.` or `..`
When I use this method I prepend a date string like '2009_01_15_19_46_32_'
and append a file extension like '.txt', so I avoid the potential of using
an invalid filename.
"""
valid_chars = "-_.() %s%s" % (string.ascii_letters, string.digits)
filename = ''.join(c for c in s if c in valid_chars)
# filename = filename.replace(' ','_') # I don't like spaces in filenames.
return filename
def trlng_dash(s: str) -> str:
"""Removes trailing character if present."""
return s[:-1] if s[-1] == '-' else s
sso_authenticate = True
# Assumes CLI config file exists.
config = configparser.ConfigParser()
home = pathlib.Path.home()
config_loc = home/'.snowsql/config' # Assumes it's set up from Snowflake CLI.
base_dir = home/r'{your Desired base directory goes here.}'
json_dir = base_dir/'json' # Location for your worksheet stage JSON files.
sql_dir = base_dir/'sql' # Location for your worksheets.
# Assumes CLI config file exists.
config.read(config_loc)
# Add connection parameters here (assumes CLI config exists).
# Using sso so only 2 are needed.
# If there's no config file, etc. enter by hand here (or however you want to do it).
connection_params = {
'account': config['connections']['accountname'],
'user': config['connections']['username'],
}
if sso_authenticate:
connection_params['authenticator'] = 'externalbrowser'
if config['connections'].get('password', None) is not None:
connection_params['password'] = config['connections']['password']
if config['connections'].get('rolename', None) is not None:
connection_params['role'] = config['connections']['rolename']
if locals().get('database', None) is not None:
connection_params['database'] = database
if locals().get('schema', None) is not None:
connection_params['schema'] = schema
sf_engine = create_engine(URL(**connection_params))
if not base_dir.exists():
base_dir.mkdir()
if not json_dir.exists():
json_dir.mkdir()
if not (sql_dir).exists():
sql_dir.mkdir()
with sf_engine.connect() as connection:
connection.execute(f'get #~/worksheet_data/ \'file://{str(json_dir.as_posix())}\';')
for file in [path for path in json_dir.glob('*') if path.is_file()]:
if file.suffix != '.json':
file.replace(file.with_suffix(file.suffix + '.json'))
with open(json_dir/'metadata.json', 'r') as metadata_file:
files_meta = json.load(metadata_file)
# List of files from metadata file will contain some empty worksheets.
files_description_orig = OrderedDict((file_key_value['name'], file_key_value) for file_key_value in sorted(files_meta['activeWorksheets'] + list(files_meta['inactiveWorksheets'].values()), key=lambda x: x['name']) if file_key_value['name'])
# files_description will only track non empty worksheets
files_description = files_description_orig.copy()
# Create updated files description filtering out empty worksheets.
for item in files_description_orig:
json_file = json_dir/f"{files_description_orig[item]['name']}.json"
# If a file didn't make it or was deleted by hand, we should
# remove from the filtered description & continue to the next item.
if not (json_file.exists() and json_file.is_file()):
del files_description[item]
continue
with open(json_file, 'r', encoding='latin_1') as f:
json_dat = json.load(f)
# If the file represents a worksheet with a body field, we want it.
if not json_dat['wsContents'].get('body'):
del files_description[item]
## Delete JSON files corresponsing to empty worksheets.
# f.close()
# try:
# (json_dir/f"{files_description_orig[item]['name']}.json").unlink()
# except:
# pass
# Produce a list of normalized filenames (no illegal or awkward characters).
file_names = set(
format_filename(trlng_dash(files_description[item]['encodedDetails']['scriptName']).strip())
for item in files_description)
# Add useful information to our files_description OrderedDict
for file_name in file_names:
repeats_cnt = 0
file_name_repeats = (
item
for item
in files_description
if file_name == format_filename(trlng_dash(files_description[item]['encodedDetails']['scriptName']).strip())
)
for file_uuid in file_name_repeats:
files_description[file_uuid]['normalizedName'] = file_name
files_description[file_uuid]['stemSuffix'] = '' if repeats_cnt == 0 else f'({repeats_cnt:0>2})'
repeats_cnt += 1
# Now we iterate on non-empty worksheets only.
for item in files_description:
json_file = json_dir/f"{files_description[item]['name']}.json"
with open(json_file, 'r', encoding='latin_1') as f:
json_dat = json.load(f)
body = json_dat['wsContents']['body']
body_bin = body.encode('latin_1')
body_txt = zlib.decompress(body_bin).decode('utf8')
sql_file = sql_dir/f"{files_description[item]['normalizedName']}{files_description[item]['stemSuffix']}.sql"
with open(sql_file, 'w') as sql_f:
sql_f.write(body_txt)
creation_stamp = files_description[item]['created']/1000
os.utime(sql_file, (creation_stamp,creation_stamp))
print('Done!')
As mentioned at Is there any option in snowflake to save or load worksheets? (and in Snowflake's own documentation), in the Classic UI, the worksheets are saved at the user stage under #~/worksheet_data/.
You can download it with a get command like:
get #~/worksheet_data/<name> file:///<your local location>; (though you might need quoting if running from Windows).
The problem is that I do not know how to access it programmatically. The downloaded files look like JSON but it is not valid JSON. The main key is "wsContents" and contains most of the worksheet information. Its value includes two subkeys, "encoding" and "body".
The "encoding" key denotes that gzip is being used. The "body" key seems to be the actual worksheet data which looks a lot like a straight binary representation of the compressed text data. As such, any JSON reader will choke on it.
If it is anything like that, I do not currently know how to access it programmatically using Python.
I do see that a JSON like format exists, BSON, that is bundled into PyMongo. Trying to use this on these files fails. I even tried bson.is_valid and it returns False so I am assuming that it means that these files in Snowflake are not actually BSON.
Edited to add: Again, BSON is a dead end.
Examining the "body" value as just binary data, the first two bytes of sample files do seem to correspond to default zlib compression (0x789c). However, attempting to run straight zlib.decompress on the slice created from that first byte to the last corresponding to the first & last characters of the "body" value results in the error:
Error - 3 while decompressing data: invalid code lengths set
This makes me think that the bytes there, as is, are at least partly garbage and still need some processing before they can be decompressed.
One clue that I failed to mention earlier is that the metadata file (called "metadata" and which serves as an inventory of the remaining files at the #~/worksheet_data/ location) declares that the files use the punycode encoding. However, I have not known how to use that information. The data in these files doesn't particularly look like what I feel punycode should look like nor does it particularly make sense to me that you would use punycode on binary data that is not meant to ever be used to directly generate text such as zlib compressed data.
few years ago I wrote a WPF (Visual Basic 2012 based) app which included "printing" some report using standard WPF printdialogs/paginators/features....
everyting worked fine since then (2011/2015), and I did not change a line of that working code...
Now I had to change something (not printing related) to that project and of course I checked the printing features as well...
now the "printing" code fails when using it... actually is not the "paginator" code, but accessing the PrintTicket/Que objects fails reporting
"PrintTicket provider failed to convert PrintTicket to DEVMODE. Win32 error: The parameter is incorrect. "
but it's weird as it does not always fail, and it fails differently with different printers...
using for instance "PDF Creator" virtual printer, it fails writing/settings some properties of my Que object and not others, while using a different printer fails on others...
for instance, PDF Creator virtual printer:
1) myQue.DefaultPrintTicket = myTicket = OK
2) myQue.CurrentJobSettings.Description = "some title" = OK with some printer, FAILS with a different printer
but 2) only fails the very first time it is set, and I can "resolve" "looping" a few times with a minor time sleep.... again, the second or third time it succeded...
so I wrote an ugly code like
Dim dlg As New PrintDialog
.....
myTicket = dlg.PrintTicket
myQue = dlg.PrintQueue
'-----
' to check in a loop if the interesting
' properties has been set
Dim ticketIsSet(1) As Boolean
For iLoop As Integer = 1 To 5
'it can fail with DEVMODE = NULL
Try
If Not ticketIsSet(0) Then
myQue.DefaultPrintTicket = myTicket
ticketIsSet(0) = True
End If
If Not ticketIsSet(1) Then
myQue.CurrentJobSettings.Description = Me.Title
ticketIsSet(1) = True
End If
Catch ex As Exception
Errors.Add(ex.Message)
System.Threading.Thread.CurrentThread.Sleep(1000)
End Try
If ticketIsSet(0) AndAlso ticketIsSet(1) Then Exit For
Next
If Errors.Count Then
' it's still ok if title can't be set... :(
If ticketIsSet(0) Then Errors.Clear()
End If
so far, "so good" as I get the job done... but only partially... as my Paginator is used to (succesfully) populate a System.Windows.Controls.DocumentViewer... this alternate documentviewer window can later actually print if requested...
and again, with different printers it succed or fail with "PrintTicket provider failed to convert PrintTicket to DEVMODE. Win32 error: The parameter is incorrect. " exception...
with PDF Creator virtual printer it succed, with a different physical printer it fails...
I tried the very same looping trick as above, like
Dim Errors As New Specialized.StringCollection
Dim bDone As Boolean = False
For iLoop As Integer = 1 To 5
Try
Dim writer As System.Windows.Xps.XpsDocumentWriter = System.Printing.PrintQueue.CreateXpsDocumentWriter(m_printQueue)
writer.Write(Me.docViewer.Document.DocumentPaginator, Me.m_printTicket)
writer = Nothing
bDone = True
Catch ex As Exception
Errors.Add(ex.Message)
System.Threading.Thread.CurrentThread.Sleep(1000)
End Try
If bDone Then Exit For
Next
If bDone Then Errors.Clear()
If Errors.Count <> 0 Then BasShowErrors(Errors, Me, False)
and here it fails (again, on some printers) on
writer.Write(Me.docViewer.Document.DocumentPaginator, Me.m_printTicket)
with "PrintTicket provider failed to convert PrintTicket to DEVMODE. Win32 error: The parameter is incorrect. " exception...
obviously that printer is ok as regards other programs like World, Excel and the like, and it's ok as well with all other (NOT WPF based) projects... and again this only occur "NOW", as it worked like a charme at the time of the initial development...
my current settings are:
Win7 Ultimate sp 1
SQL 2014 Dev Edition (not relevant)
Visual Studio 2012 Ultimate Version 11.0.61219.00 Update 5
Microsoft .NET Framework version 4.6.01055
I even tried "repairing" the NET Framework with NetFramwork Repair Tool (https://www.microsoft.com/en-us/download/details.aspx?id=30135) with no success...
any hint is very appreciated :)
TIA
asql
I'm brand new to MS-Access and had a few guideline-questions,
My organization uses MS-Access to track a large electronic-part inventory. These parts have a hyperlink field that links to the product webpage. Here's an example:
Part Number Part Type Value Description Component_Height Voltage Tolerance Schematic Part Layout PCB Footprint Manufacturer Part Number Manufacturer Distributor Part Number Distributor Price Availability Link
UMK105CG100DV-F Ceramic 10pF CAP CER 10PF 50V NP0 0402 0.35 MM 50V ±0.5pF xxxxx\C_NP,xxxxx\C_NP_Small c_0402 UMK105CG100DV-F Taiyo Yuden 587-1947-2-ND Digi-Key 0.00378 In Stock http://www.digikey.com/product-detail/en/UMK105CG100DV-F/587-1947-2-ND/1473246
Links Here:
http://www.digikey.com/product-detail/en/UMK105CG100DV-F/587-1947-2-ND/1473246
Nearly the entire majority of our hyperlinks point to the supplier DigiKey.
Right now the verification flow goes like this:
Every month or so a large group of us sits down and one by one copies the hyperlink into google.
We then open the corresponding webpage and verify component availability etc.
We have nearly 1000 components and this process takes hours. All I'm looking for is advice on how to improve our workflow. I was hoping there was say a way to write a "open hyperlink with default browser and search string" macro or scripting interface. The pseudo-script would then check that the string "Quantity Available" was greater than 1, and if it wasn't (the part was out of stock) mark the part as obsolete.
Any advice would be greatly appreciated, I'm really aiming to optimize our workflow.
You can traverse the DOM of the web page. A quick look at the web page and you can see a table with a name of product-details.
So the following VBA code would load the sample web page, and pull out the values.
Option Compare Database
Option Explicit
Enum READYSTATE
READYSTATE_UNINITIALIZED = 0
READYSTATE_LOADING = 1
READYSTATE_LOADED = 2
READYSTATE_INTERACTIVE = 3
READYSTATE_COMPLETE = 4
End Enum
Sub GetWebX()
Dim ie As New InternetExplorer
Dim HTML As New HTMLDocument
Dim strURL As String
Dim Htable As New HTMLDocument
Dim i As Integer
strURL = "http://www.digikey.com/product-detail/en/UMK105CG100DV-F/587-1947-2-ND/1473246"
ie.Navigate strURL
Do While ie.READYSTATE < READYSTATE_COMPLETE
DoEvents
Loop
Set HTML = ie.Document
Set Htable = HTML.getElementById("product-details")
For i = 0 To Htable.Rows.Length - 1
With Htable.Rows(i)
Debug.Print Trim(.Cells(0).innerText), Trim(.Cells(1).innerText)
End With
Next I
ie.Quit
Set ie = Nothing
End Sub
output of above:
Digi-Key Part Number 587-1947-2-ND
Quantity Available 230,000
Can ship immediately
Manufacturer Taiyo Yuden
Manufacturer Part Number UMK105CG100DV-F
Description CAP CER 10PF 50V NP0 0402
Expanded Description 10pF ±0.5pF 50V Ceramic Capacitor C0G, NP0 0402(1005 Metric)
Lead Free Status / RoHS Status Lead free / RoHS Compliant
Moisture Sensitivity Level (MSL) 1 (Unlimited)
Manufacturer Standard Lead Time 11 Weeks
Since the above is a array, then you could place a button right on the form, and have a few extra lines of VBA to write the values into the form. So a user would just have to go to the given record/form in Access - press a button and the above values would be copied right into the form.
the above VBA code requires a reference to:
Microsoft Internet Controls
Microsoft HTML Object Library
I would suggest that after testing you use late binding for the above two libraries.
I have downloaded a .raw datafile that I want to view in Paraview.
The dataset I'm using is taken from:
http://volvis.org/ -> download dataset -> CT Scan of bonsaitree.
Whenever I import the dataset I do as explained here in this link:
http://www.paraview.org/Wiki/ParaView/Data_formats#Raw_files
Everything goes fine until I hit apply. I then get these two error messages, and the view is empty:
ERROR: In C:\DBD\pvs-x64\paraview\src\paraview\ParaViewCore\VTKExtensions\Rendering\vtkTexturePainter.cxx, line 295
vtkTexturePainter (00000000136105F0): Incorrect dimensionality.
Warning: In C:\DBD\pvs-x64\paraview\src\paraview\VTK\Rendering\Core\vtkRenderer.cxx, line 1030
vtkOpenGLRenderer (0000000009CF08C0): Resetting view-up since view plane normal is parallel
How do I get rid of these error messages?
A raw file does not have a header with information about the data inside (orientation, dimension...). You have to provide that information, and it has to be compatible with the content of the image. I tried a couple of time with the data you provided and succeeded with the following parameters:
when prompted reader: raw binary file (I just checked the file content with notepad)
data extent: 0 255 0 255 0 255 (the website says that the image is 256x256x256)
data scalar type: char (I confess that I just guessed it at the first time, a more correct way would be to multiply the data type by the number of voxels and compare it with the file dimension).
Having learned loads from answers on this site (thanks!), it's finally time to ask my own question.
I'm using R (tm and lsa packages) to create, clean and simplify, and then run LSA (latent semantic analysis) on, a corpus of about 15,000 text documents. I'm doing this in R 3.0.0 under Mac OS X 10.6.
For efficiency (and to cope with having too little RAM), I've been trying to use either the 'PCorpus' (backend database support supported by the 'filehash' package) option in tm, or the newer 'tm.plugin.dc' option for so-called 'distributed' corpus processing). But I don't really understand how either one works under the bonnet.
An apparent bug using DCorpus with tm_map (not relevant right now) led me to do some of the preprocessing work with the PCorpus option instead. And it takes hours. So I use R CMD BATCH to run a script doing things like:
> # load corpus from predefined directory path,
> # and create backend database to support processing:
> bigCcorp = PCorpus(bigCdir, readerControl = list(load=FALSE), dbControl = list(useDb = TRUE, dbName = "bigCdb", dbType = "DB1"))
> # converting to lower case:
> bigCcorp = tm_map(bigCcorp, tolower)
> # removing stopwords:
> stoppedCcorp = tm_map(bigCcorp, removeWords, stoplist)
Now, supposing my script crashes soon after this point, or I just forget to export the corpus in some other form, and then I restart R. The database is still there on my hard drive, full of nicely tidied-up data. Surely I can reload it back into the new R session, to carry on with the corpus processing, instead of starting all over again?
It feels like a noodle question... but no amount of dbInit() or dbLoad() or variations on the 'PCorpus()' function seem to work. Does anyone know the correct incantation?
I've scoured all the related documentation, and every paper and web forum I can find, but total blank - nobody seems to have done it. Or have I missed it?
The original question was from 2013. Meanwhile, in Feb 2015, a duplicate, or similar question, has been answered:
How to reconnect to the PCorpus in the R tm package?. That answer in that post is essential, although pretty minimalist, so I'll try to augment it here.
These are some comments I've just discovered while working on a similar problem:
Note that the dbInit() function is not part of the tm package.
First you need to install the filehash package, which the tm-Documentation only "suggests" to install. This means it is not a hard dependency of tm.
Supposedly, you can also use the filehashSQLite package with library("filehashSQLite") instead of library("filehash"), and both of these packages have the same interface and work seamlesslessly together, due to object-oriented design. So also install "filehashSQLite" (edit 2016: some functions such as tn::content_transformer() are not implemented for filehashSQLite).
then this works:
library(filehashSQLite)
# this string becomes filename, must not contain dots.
# Example: "mydata.sqlite" is not permitted.
s <- "sqldb_pcorpus_mydata" #replace mydat with something more descriptive
suppressMessages(library(filehashSQLite))
if(! file.exists(s)){
# csv is a data frame of 900 documents, 18 cols/features
pc = PCorpus(DataframeSource(csv), readerControl = list(language = "en"), dbControl = list(dbName = s, dbType = "SQLite"))
dbCreate(s, "SQLite")
db <- dbInit(s, "SQLite")
set.seed(234)
# add another record, just to show we can.
# key="test", value = "Hi there"
dbInsert(db, "test", "hi there")
} else {
db <- dbInit(s, "SQLite")
pc <- dbLoad(db)
}
show(pc)
# <<PCorpus>>
# Metadata: corpus specific: 0, document level (indexed): 0
#Content: documents: 900
dbFetch(db, "test")
# remove it
rm(db)
rm(pc)
#reload it
db <- dbInit(s, "SQLite")
pc <- dbLoad(db)
# the corpus entries are now accessible, but not loaded into memory.
# now 900 documents are bound via "Active Bindings", created by makeActiveBinding() from the base package
show(pc)
# [1] "1" "2" "3" "4" "5" "6" "7" "8" "9"
# ...
# [900]
#[883] "883" "884" "885" "886" "887" "888" "889" "890" "891" "892"
#"893" "894" "895" "896" "897" "898" "899" "900"
#[901] "test"
dbFetch(db, "900")
# <<PlainTextDocument>>
# Metadata: 7
# Content: chars: 33
dbFetch(db, "test")
#[1] "hi there"
This is what the database backend looks like. You can see that the documents from the data frame have been encoded somehow, inside the sqlite table.
This is what my RStudio IDE shows me: