We need to run a spatial query and we are also using Breeze, and I understand that Breeze currently does not support spatial queries. So I was wondering if it would be possible to pass a regular query to the controller using the regular Odata format; run the query and then UNION the results with a spatial query run directly against Entity Framework?
In this way, we can have the client prepare a demographic query and do a level one "filter" of the persons we are interested in, and then apply the spatial query to do a level two "filter".
Is this possible? What would the syntax look like?
Thanks!!!!!
Bob
Breeze cannot construct an OData filter on a spatial type but any Breeze WebApi/WebApi2 server can "return" spatial types. So if your query result returns an entity or a projection containing a spatial type, it will be available on the client. ( The client side format is dependent on the JSON.NET formatter).
If this is acceptable, then you can use Breeze's NamedQuery facility to pass extra parameters to your server side query and use these to construct your own spatial filtering.
Hope this makes sense.
Related
I'm dealing with a three piece datapath: client application, host integration server, db server. Client application (MS Mashup Engine) is generating queries that pass through MS SQL Server to a legacy IBM iSeries DB backend.
I'm running into issues where the client is generating queries like
select * from x where numericValue = 1.46510+003
I'm checking the execution plan for these on the SQL Server and they result in a full data load with the comparison occurring on SQL Server (which is acting as the Host Integration Server).
By comparison, a human generated query
select * from x where numericValue = 1465.1
results in no scan and performance two orders of magnitude faster.
I have tried playing with the client application to force it to generate something like the human generated query, but I've had no luck.
I'm not sure if I can massage the way the query plan is generated in SQL server by playing with column data types. I.e. exposing a view over the backend DB with explicitly defined data types. Or otherwise forcing query plan generation?
Any thoughts?
I'm not sure if I can massage the way the query plan is generated in SQL server by playing with column data types. I.e. exposing a view over the backend DB with explicitly defined data types. Or otherwise forcing query plan generation?
No, the answer is no using all of the following:
casting types prior to delivering to power query/power bi changes nothing
adapter properties set for DB2OLEDB or IBMDASQL ( f.ex. Decimal As Numeric=True; Derive Parameters=True; etc...) have no effect
creating stored procedures over the data requires parametrization in power query which is fine, but does not integrate with the ui. I.e. - using a parameter field instead of the column filters is not clean.
TBH, this has become less and less of a problem as backend performance improves and frontend caching is available.
Hope it helps,
-Alex T.
I'm a SSIS Developer. I do lots of SQL stored procedure lookup concepts in SSIS. But when coming to Azure Data Factory I haven't any idea how to perform a lookup using a SQL stored procedure.
Could anyone please guide me on this?
Thanks in advance !
Jay
Azure Data Factory (ADF) is more of an ELT tool rather than ETL, therefore direct lookups are not supported. Instead, this type of operation, along with other transforms is pushed down into the compute you are actually using. For example, if you are moving data to SQL Server, Azure SQL Database or Azure SQL Data Warehouse, you would ensure all data is on the same server and use a Stored Procedure task to execute the lookups using T-SQL and joins. If you are using Azure Data Lake Analytics (ADLA) you would use the U-SQL Activity to run U-SQL or execute ADLA stored procedures, again doing lookups via joins or custom U-SQL code such as Combiner, Applier, Reducer. In fact you can use any of the ADF compute options like SQL, HDInsight (including Hive, Pig, Map Reduce, Streaming and Spark script), Machiine Learning or custom .net activities.
So you need to think about things differently with ADF. Have a look through this article to gain greater understanding of transforming data in ADF:
Transform data in Azure Data Factory
https://learn.microsoft.com/en-us/azure/data-factory/data-factory-data-transformation-activities
As an aside, I would rarely use Lookups in SSIS as performance in early versions used to be poor. Although this has been improved in later versions, generally if you can do it in SQL you probably should. This pattern harnesses the power of SQL Server, rather than dragging data up into the SSIS pipeline, eg for the purposes of lookups (which are essentially joins) and pushing the data back out again. I reserve Data Flow transformations mainly when non-relational data is involved, eg xml or joining your email server with relational data. This is my personal view anyway : )
I'm new to elastic search and I have a basic question.
I want to load data from database and search them by using elastic search in MVC.NET project, but cause of data I have in my database's table I cant't convert all of them to the json and search in thme by using elastic search. How should I fill data of the elastic search from the database in an mvc.net project. I don't want the whole solution because it is impossible just a general and brief explanation. thank you very much.
First of all you should be able to model your data from SQL to ElasticSearch.
As ElasticSearch is a NoSQL and document oriented database/search engine.
You need an indexer to index SQL data to ElasticSearch.
Get all the columns associated with one record that you want to search in ElasticSearch from your SQL database (use joins if data is in multiple tables).
Use a dedicated Stored Procedure to get only needed data and construct a document class, serialize to JSON and index in your ElasticSearch cluster.
Use ElasticSearch.net client as they very neatly expose bulk index APIs.
Hope this will get you started. Have fun
I was trying to understand the basic advantage of using XML DataType in SQL Server 2005. I underwent the article here, saying that in case you want to delete multiple records. Serialize the XMl, send it in Database and using below query you can delete it..
I was curious to look into any other advantage of using this DataType...
EDIT
Reasons for Storing XML Data in SQL Server 2005
Here are some reasons for using native XML features in SQL Server 2005 as opposed to managing your XML data in the file system:
You want to use administrative functionality of the database server for managing your XML data (for example, backup, recovery and replication).
My Understanding - Can you share some knowledge over it to make it clear?
You want to share, query, and modify your XML data in an efficient and transacted way. Fine-grained data access is important to your application. For example, you may want to insert a new section without replacing your whole document.
My Understanding - XML is in specific column row, In order to add new section in this row's cell, Update is required, so whole document will be updated. Right?
You want the server to guarantee well-formed data, and optionally validate your data according to XML schemas.
My Understanding - Can you share some knowledge over it to make it clear?
You want indexing of XML data for efficient query processing and good scalability, and the use a first-rate query optimizer.
My Understanding - Same can be done by adding individual columns. Then why XML column?
Pros:
Allows storage of xml data that can be automatically controlled by an xml schema - thereby guaranteeing a certain level of data quality
Many web/desktop apps store data in xml form, these can then be easily stored and queried in the database - so it is a great place to store xml data that an app may need to use (e.g. for configuration settings)
Cons:
Be careful about using xml fields, they may start off as innocent storage but can become a performance nightmare if you want to search, analyse and report on many records.
Also, if xml fields will be added to, changed or deleted this can be slow and leads to complex t-sql.
In replication, the whole xml gets updated even if only one node changes - therefore you could have many more conflicts that cannot easily be resolved.
I would say of the 4 advantages you've listed, these two are critical:
You want to share, query, and modify your XML data in an efficient and transacted way
SQL Server stores the XML in an optimised way that it wouldn't for plain strings, and lets you query the XML in an efficient way, rather than requiring you to bring the entire XML document back to the client. Think how inefficient it is if you want to query 10,000 XML columns, each containing 1k of data. For a small XPath query you would need to return 10k of data across the wire, for each client, each time.
You want indexing of XML data for efficient query processing and good scalability, and the use a first-rate query optimizer
This ties into what I said above, it's far more efficiently stored than a plain text column which would also run into page fragmentation issues.
Is it possible to use Sql Server XML columns as a substitute for a real Document DB (such as Couch or Mongo) ?
If I were to create a table with a guid PK Id and an XML column for the document.
What would be the main problems compared to using a document DB?
Sql Server supports indexing over XML columns so querying should not be completely horrible?
You've got several questions in here:
Is it possible to use Sql Server XML columns as a substitute for a real Document DB (such as Couch or Mongo) ? Yes, you can use it as a substitute, but no, you probably wouldn't be satisfied with performance if you're exclusively storing XML and not leveraging any of SQL Server's relational tools.
If I were to create a table with a guid PK Id and an XML column for the document. What would be the main problems compared to using a document DB? In a nutshell, scaling out. SQL Server doesn't scale this kind of thing out well. You can do it with replication, but it's painful to manage relative to a "real" Document DB.
Sql Server supports indexing over XML columns so querying should not be completely horrible? The problem is that SQL Server's XML indexes can take several times the storage space of the original data. These indexes can't be maintained online (as in defrags), so you end up with locking issues during maintenance windows.
I'm doing some experimenting with this on:
http://rogeralsing.com/2011/03/02/linq-to-sqlxml-projections/
Query speed is 'decent' , it's nothing I'd use for scaling.
But the joy of schema free storage running on standard infrastructure is quite nice.
Yes, you can. Storing a document inside a SqlServer XML column will work and if you use standard XML serialization that will leave you with a decent ACID complant key/value store. Also, it will allow you to do queries on it with relative ease and you can join the results to data that you store in a more relational way. We do so, it works. If you store content in XML fields, storage demands are a lot lower than using NTEXT and querying it will be more flexible and faster.
What SqlServer will not get you (comparing to mongo) is the seamless failover of replica-sets an the autosharding of mongo. Also, atomic operations like incrementing a specific property deep inside a document is hard (though not impossible with the XQuery update function). Updates tend to be faster on most NoSql databases, because they are more relaxed on the "data is only safe on disk" principle.
Yes, it is possible. As to whether it's a good idea, this is just my 2 cents...
Before the XML datatype came along I worked on a system storing XML in an NTEXT column - that wasn't pleasant, and to get any real use out of the data meant shredding some of that data out into relational form.
OK, the XML datatype now makes it easier to query an XML blob and to extract certain values/index them. But personally, in general, I wouldn't. I'm not saying never use XML as there are scenarios for that - rather if that's all your planning on doing then I'd be thinking "is this the right tool for the job". Using a RDBMS as a document database makes me feel a bit uneasy. Whereas something like MongoDB has been built from the ground up as a document database.
In all honesty, I haven't done any performance testing on storing data as XML so I can't give you an indication of what performance would be like. Would be interested to know how this performs at scale.