PostGIS not recognizing geometry type after writing to database with geopandas - postgis

I have a PostGIS database set up that is spatially enabled.
engine = create_engine(f'postgresql://user:password#localhost:5432/{dbname}')
I have a valid geopandas dataframe (i.e., if I export as .shp or .gpkg I can load into a GIS app with no errors) with a geometry column named 'geometry'. I write these data to the database with:
gdf.to_postgis('table_name', con=engine, if_exists='append', index_label='id')
If I read all records from this table:
gpd.read_postgis('table_name', con=engine, geom_col='geometry')
I get the following warning:
SAWarning: Did not recognize type 'geometry' of column 'geometry' pandas_sql.meta.reflect(only=[sql])
If I try to execute an postgresql statement such as:
gpd.read_postgis('SELECT DISTINCT(st_cnty_fips) FROM table_name', con=self.engine, geom_col='geometry')
I get an error:
line 65, in _df_to_geodf
raise ValueError("Query missing geometry column '{}'".format(geom_col))
ValueError: Query missing geometry column 'geometry'
Using PGAdmin4 I can see that the column exists. What am I doing wrong?

Geopandas requires a geometry columna after filtering out geometry additionally or adding it should work.
The parameter geom_col='geometry' allows you to specify the name of this column that geopandas needs.
Need to add geometries.
For example:
SELECT your_query_with_out_geom, geometry FROM table_name;
In your case, could:
gpd.read_postgis('SELECT DISTINCT(st_cnty_fips), geometry FROM table_name', con=self.engine, geom_col='geometry')

Related

Relation IDs mismatch - Mapping OWL to Oracle DB with Ontop

As a Part of my little App I try to map Data between my Ontology and an Oracle DB with ontop. But my first mapping is not accepted by the reasoner and it's not clear why.
As my first attempt I use the following target:
:KIS/P_PVPAT_PATIENT/{PPVPAT_PATNR} a :Patient .
and the following source:
select * from P_PVPAT_PATIENT
Here KIS is the schema, p_pvpat_patient the table and ppvpat_patnr the key.
Caused by: it.unibz.inf.ontop.exception.InvalidMappingSourceQueriesException:
Error: Relation IDs mismatch: P_PVPAT_PATIENT v "KIS"."P_PVPAT_PATIENT" P_PVPAT_PATIENT
Problem location: source query of triplesMap
[id: MAP_PATIENT
target atoms: triple(s,p,o) with
s/RDF(http://www.semanticweb.org/grossmann/ontologies/kis-ontology#KIS/P_PVPAT_PATIENT/{}(TmpToVARCHAR2(PPVPAT_PATNR)),IRI), p/<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>, o/<http://www.semanticweb.org/grossmann/ontologies/kis-ontology#Patient>
source query: select * from P_PVPAT_PATIENT]
As the error said my source query was wrong because I forgot to use the schema in my sql.
the correct sql is
select * from kis.P_PVPAT_PATIENT

Exporting Kusto table to SQL Server GENERATED ALWAYS column

I'm setting up a data pipeline to export from a Kusto table to a SQL Server table. The only problem is the target table has two GENERATED ALWAYS columns. Looking for some help implementing the solution using Kusto.
This is the export statement:
.export async to sql ['CompletionImport']
h#"connection_string_here"
with (createifnotexists="true", primarykey="CompletionSearchId")
<|set notruncation;
apiV2CompletionSearchFinal
| where hash(SourceRecordId, 1) == 0
Which gives the error:
Cannot insert an explicit value into a GENERATED ALWAYS column in table 'server.dbo.CompletionImport'.
Use INSERT with a column list to exclude the GENERATED ALWAYS column, or insert a DEFAULT into GENERATED ALWAYS column.
So I'm a little unsure how to implement this solution in Kusto. Would I just add a project pipe excluding the GENERATED ALWAYS columns? Maybe ideally how could I insert a DEFAULT value into GENERATED ALWAYS SQL Server columns using a Kusto query?
Edit: Trying to use materialize to create a temporary table in the cache and export this cached table. However I can't find any documentation on this and the operation is failing:
let dboV2CompletionSearch = apiV2CompletionSearchFinal
| project every, variable, besides, generated, always, ones;
let cachedCS = materialize(dboV2CompletionSearch);
.export async to sql ['CompletionImport']
h#"connect_string"
with (createifnotexists="true", primarykey="CompletionSearchId")
<|set notruncation;
cachedCS
| where hash(SourceRecordId, 1) == 0
with the following error message:
Semantic error: 'set notruncation;
cachedCS
| where hash(SourceRecordId, 1) == 0'
has the following semantic error: SEM0100:
'where' operator: Failed to resolve table or column expression named 'cachedCS'.

Import JSON into ClickHouse

I create table with this statement:
CREATE TABLE event(
date Date,
src UInt8,
channel UInt8,
deviceTypeId UInt8,
projectId UInt64,
shows UInt32,
clicks UInt32,
spent Float64
) ENGINE = MergeTree(date, (date, src, channel, projectId), 8192);
Raw data looks like:
{ "date":"2016-03-07T10:00:00+0300","src":2,"channel":18,"deviceTypeId ":101, "projectId":2363610,"shows":1232,"clicks":7,"spent":34.72,"location":"Unknown", ...}
...
Files with data loaded with the following command:
cat *.data|sed 's/T[0-9][0-9]:[0-9][0-9]:[0-9][0-9]+0300//'| clickhouse-client --query="INSERT INTO event FORMAT JSONEachRow"
clickhouse-client throw exception:
Code: 117. DB::Exception: Unknown field found while parsing JSONEachRow format: location: (at row 1)
Is it possible to skip fields from JSON object that not presented in table description?
The latest ClickHouse release (v1.1.54023) supports input_format_skip_unknown_fields user option which eneables skipping of unknown fields for JSONEachRow and TSKV formats.
Try
clickhouse-client -n --query="SET input_format_skip_unknown_fields=1; INSERT INTO event FORMAT JSONEachRow;"
See more details in documentation.
Currently, it is not possible to skip unknown fields.
You may create temporary table with additional field, INSERT data into it, and then do INSERT SELECT into final table. Temporary table may have Log engine and INSERT into that "staging" table will work faster than into final MergeTree table.
It is relatively easy to add possibility to skip unknown fields into code (something like setting 'format_skip_unknown_fields').

Get geometry from a varchar(max) in an update query

I've got a table full of WKT (well known text) in a varchar(max) column. This is formatted exactly as geometry. So, for example, one of the values is:
POLYGON ((174.893529167059 -37.0260462162965,174.89351593407 -37.0260221329151,174.893508034056 -37.0260077002766,174.893444415879 -37.0258916500588,174.893416916056 -37.0258414997842,174.893481733297 -37.0258186834198,174.893492016755 -37.0258150663863,174.89349653254 -37.025823316032,174.893512415978 -37.0258522827285,174.893556883897 -37.0259333832477,174.893591032956 -37.0259956661343,174.893604265986 -37.0260197504078,174.893575149738 -37.0260300006258,174.893529167059 -37.0260462162965))
However, I need to convert this varchar(max) field into a geometry field. Unfortunately, SQL Server needs to have some other information in order to do this, so I can't just change the type of the field from varchar(max) to geometry.
I've created a blank geometry column in the same table (MyGeometry) but I'm failing in trying to convert it. Here is my code (where 2193 is the CRS I'm dealing with). WKT is my varchar(max) field and MyGeometry is my new geometry field.
UPDATE MY_TABLE
SET MyGeometry = geometry::STPolyFromText('' + WKT + '', 2193)
EDIT - currently coming back saying the WKT is not valid (System.FormatException 24111 - the input isn't valid).
Which is strange as it matches up perfectly with some other inputs I have in how it's styled.
This will do :
UPDATE MY_TABLE
SET MyGeometry = geometry::STGeomFromText('POLYGON ((174.893529167059 -37.0260462162965,174.89351593407 -37.0260221329151,174.893508034056 -37.0260077002766,174.893444415879 -37.0258916500588,174.893416916056 -37.0258414997842,174.893481733297 -37.0258186834198,174.893492016755 -37.0258150663863,174.89349653254 -37.025823316032,174.893512415978 -37.0258522827285,174.893556883897 -37.0259333832477,174.893591032956 -37.0259956661343,174.893604265986 -37.0260197504078,174.893575149738 -37.0260300006258,174.893529167059 -37.0260462162965))', 0)
You can change that 'POLYGON....' to your VARCHAR(MAX) field.
0 at the end, which is SRID only relevant if you're using Geography.
You can also try :-
UPDATE MY_TABLE SET MyGeometry = GEOMETRY::STGeomFromText('POLYGON ((174.893529167059 -37.0260462162965,174.89351593407 -37.0260221329151,174.893508034056 -37.0260077002766,174.893444415879 -37.0258916500588,174.893416916056 -37.0258414997842,174.893481733297 -37.0258186834198,174.893492016755 -37.0258150663863,174.89349653254 -37.025823316032,174.893512415978 -37.0258522827285,174.893556883897 -37.0259333832477,174.893591032956 -37.0259956661343,174.893604265986 -37.0260197504078,174.893575149738 -37.0260300006258,174.893529167059 -37.0260462162965))',4326);
The value of SRID passed should be 4326 and not 0
You can check the URL below. I found this information in another similar question (older post) so pasting it without any changes
I found solution, SQL Server Spatial Tools
http://sqlspatialtools.codeplex.com/
Followings are the methods solved my problem.
IsValidGeographyFromText(string inputWKT, int srid)
Check if an input WKT can represent a valid geography. This function requires that the WTK coordinate values are longitude/latitude values, in that order and that a valid geography SRID value is supplied. This function will not throw an exception even in edge conditions (i.e. longitude/latitude coordinates are reversed to latitude/longitude).
SqlGeography MakeValidGeographyFromText(string inputWKT, int srid)
Convert an input WKT to a valid geography instance. This function requires that the WKT coordinate values are longitude/latitude values, in that order and that a valid geography SRID value is supplied.

Backup table to xml or csv file, content is long text

I have a table that contains content (like blog posts, so fairly long text) that I want to export to a xml file.
So I want it like:
<table>
<column1>1231</column1>
<column2>January 1, 2001</column2>
<column3>some very long text will all types of characters in it</column3>
</table>
Is there a built in way to do this?
Basically each column will have its own element.
The content should ideally be CDATA since the content can contain any type of character potentially.
I have sql server 2008 express.
From SQL Server 2005, the FOR XML clause provides a way to convert the results of an SQL query to XML.
E.g.
Consider a table building with Blgd, Suit, SQFT, PDate columns.
SELECT * FROM building FOR XML AUTO
will convert the contents of table to the following XML:
<building Bldg="1" SUit="1" SQFT="1000" PDate="2012-09-24T00:00:00" />
<building Bldg="1" SUit="1" SQFT="1500" PDate="2011-12-31T00:00:00" />
If you want the columns to be elements, then
SELECT * FROM building FOR XML AUTO, ELEMENTS
would convert the contents to following XML:
<building>
<Bldg>1</Bldg>
<SUit>1</SUit>
<SQFT>1000</SQFT>
<PDate>2012-09-24T00:00:00</PDate>
</building>
<building>
<Bldg>1</Bldg>
<SUit>1</SUit>
<SQFT>1500</SQFT>
<PDate>2011-12-31T00:00:00</PDate>
</building>
If you want to model your text fields as CDATA sections, then you should use the FOR XML EXPLICIT clause and define your XML schema as per the guidelines here.
If the above Building table has a text_col column of type TEXT that should be modeled as CDATA section in the generated XML, then the SELECT query would be as follows:
SELECT
1 as Tag,
NULL as Parent,
Bldg AS [Building!1!Bldg!ELEMENT],
text_col AS [Building!1!!CDATA]
FROM Building
WHERE text_col IS NOT NULL
FOR XML EXPLICIT
The results would be as follows:
<Building><Bldg>1</Bldg><![CDATA[From SQL Server 2005, the FOR XML clause provides a way to convert the results of an SQL query to XML.
E.g. Consider a table building with Blgd, Suit, SQFT, PDate columns.
SELECT * FROM building FOR XML AUTO
will convert the contents of table to the following XML:
<building Bldg="1" SUit="1" SQFT="1000" PDate="2012-09-24T00:00:00" />
<building Bldg="1" SUit="1" SQFT="1500" PDate="2011-12-31T00:00:00" />
If you want the columns to be elements, then
SELECT * FROM building FOR XML AUTO, ELEMENTS
would convert the contents to following XML:
<building>
<Bldg>1</Bldg>
<SUit>1</SUit>
<SQFT>1000</SQFT>
<PDate>2012-09-24T00:00:00</PDate>
</building>
<building>
<Bldg>1</Bldg>
<SUit>1</SUit>
<SQFT>1500</SQFT>
<PDate>2011-12-31T00:00:00</PDate>
</building>]]></Building>
You can use the FOR XML SQL construct to do this. Please read here

Resources