I have a list of n records and for each record id, I need to fetch a list of integers (partids) from the database.
Currently from the app, I have to loop n times passing the main id to fetch each list from db.
Obviously this is affecting the performance of the app (.NET Winforms app using vb.net).
Is there any way that I can avoid multiple calls?
Like can I send a list of ids n get my desired result as output?
Example
input : record1 having id 100
output: id 100, partid 1, 2, 3
Same for n rows
I don't know how.. any help please
If you share your query your using we'll be able to give you feedback. Assuming however you have a query like
SELECT PartId FROM tblParts WHERE CategoryId = 100
You can use
SELECT CategoryId, PartId FROM tblParts WHERE CategoryId in (100,101,102 etc)
to query the database and return the relevant records all at once. (or even omit the WHERE clause to return everything in the table).
Then in the local datatable that you get back you can use the method .Select("CategoryId=101") to execute a local query against your data and then process your data as normal.
Related
I'm currently using a database where I'll be sending 10 unique Id's at a time and I'm expecting a list of objects in return.
Is there any way where I can use a native query to get this.
select * from tbl_bank_customer_accounts where uid in ('05b6864b-ad10-4efe-af8d-b556e02dfc75','19226ae6-72f3-40dc-a690-079c979a3044','1a196c99-007f-4edd-a8a1-556f070cc852' ,'1e88f4fb-101c-4f6c-acd0-100bd1d02237' ,'20511b9b-8e7b-44fb-ba79-5c2103679b39' ,'3564b980-59e3-411f-8cc2-1bffa3cf9883' ,'4491b692-b6a4-4581-b150-411ab03151a4' ,'6364b0eb-015c-422f-a5e2-ae00828c70d6' ,'6ed7a6a2-265d-4d59-8764-d7742296aa17' ,'93c99403-d099-401f-a544-44984dcfc925');
the above one is my query can it be implemented using stored procedure
I have a table in Snowflake where one of the fields, called 'value' is sometimes plain text sometimes JSON, this field is stored as string in Snowflake
I created this view to get only the rows where there is a Json format
CREATE OR REPLACE VIEW tmp_events AS
SELECT PARSE_JSON(value) as json_data,
id
FROM SessionEvent
WHERE session_event_type_id=7;
Then I flatten the rows to create a new field
CREATE OR REPLACE VIEW tmp_events_step2 AS
SELECT id,
json_data:meta:selected::string AS choice
from tmp_events ,
LATERAL FLATTEN(input => tmp_events.json_data)
WHERE choice IS NOT NULL
Everything runs fine until now, I can preview data from these two views, no error and I get the results I was expecting.
The error comes when I try to get distinct values from choice
SELECT DISTINCT choice from tmp_events_step2;
Error parsing JSON: unknown keyword "brain", pos 6
This name Brain seems to come from my initial table without the WHERE statement.
If I run the query without DISTINCT there is no error.
Weird thing I noticed while trying to debug: when I put a limit in tmp_events_step2, the code works fine again, even though I put a limit that's bigger than the number of rows in the table
CREATE OR REPLACE VIEW tmp_events_step2 AS
SELECT id,
json_data:meta:selected::string AS choice
from tmp_events ,
LATERAL FLATTEN(input => tmp_events.json_data)
WHERE choice IS NOT NULL
LIMIT 10000000;
SELECT DISTINCT choice from tmp_events_step2;
What's the catch? Why does it work only with the limit?
The very simple answer to this is the built-in function TRY_PARSE_JSON()
Er, not. You seem to have problems with the Query optimizer that may do incorrect predicate pushdowns. One way to prevent the optimizer from doing this is to use the secure view option:
CREATE SECURE VIEW tmp_events_step2 ...
and file a support ticket...
We reported this error two years ago and they said they where not going to fix, because by hoisting the JSON access prior to running the filters in the WHERE clause that makes the cast valid/safe, impacted performance.
create table variant_cast_bug(num number, var variant);
insert into variant_cast_bug
select column1 as num, parse_json(column2) as var
from values (1, '{"id": 1}'),
(1, '{"id": 2}'),
(2, '{"id": "text"}')
v;
select * from variant_cast_bug;
select var:id from variant_cast_bug;
select var:id from variant_cast_bug where num = 1;
select var:id::number from variant_cast_bug where num = 1; -- <- broken
select TRY_TO_NUMBER(var:id) from variant_cast_bug where num = 1; -- <- works
Sometimes you can nest the select and it will work, and then you can add another SELECT layer around it, and do some aggregation and the cost explodes again.
The only two safe solutions are SERCURE VIEW as Hans mentions, but that is a performance nightmare.
Or to understand this problem and use TRY_TO_NUMBER or it's friends.
At the time this was made bad worse because JSON boolean values where not valid values to pass to TRY_TO_BOOLEAN..
One of the times we got burnt by this was after a snowflake release when code that had been running for a year, started getting this error, because it was complex enough the hoisting did not impact, and then after release it did. This is where Snowflake are rather responsive, and then rolled the release back, and we put TRY_TO on a chunk of already working SQL just to play it safe.
Please submit a support case for this issue.
I'm using Google-Appengine-NDB. And I'm tried to get distinct values from database but it's not working.
Now my code is:
query_set = cls.query().order(cls.ls) # Getting ordered queries.
set_of_field = set([data.field for data in query_set]) # And using this loop for differ.`
But the loop is taking too long time (over 12 sec).
Please help me, how can I speed up, or how to get distinct values from ndb?
Try the distinct query,
if your field is indexed you can use this:
https://developers.google.com/appengine/docs/python/ndb/queries#projection
query_set = cls.query(projection=["field"], distinct=True)
set_of_field = [data.field for data in query_set]
But if you have a huge list you can either do this in a taskqueue and store the results somewhere or just keep a distinct data in another model.
I'm using the lsqlite3 lua wrapper and I'm making queries into a database. My DB has ~5million rows and the code I'm using to retrieve rows is akin to:
db = lsqlite3.open('mydb')
local temp = {}
local sql = "SELECT A,B FROM tab where FOO=BAR ORDER BY A DESC LIMIT N"
for row in db:nrows(sql) do temp[row['key']] = row['col1'] end
As you can see I'm trying to get the top N rows sorted in descending order by FOO (I want to get the top rows and then apply the LIMIT not the other way around). I indexed the column A but it doesn't seem to make much of a difference. How can I make this faster?
You need to index the column on which you filter (i.e. with the WHERE clause). THe reason is that ORDER BY comes into play after filtering, not the other way around.
So you probably should create an index on FOO.
Can you post your table schema?
UPDATE
Also you can increase the sqlite cache, e.g.:
PRAGMA cache_size=100000
You can adjust this depending on the memory available and the size of your database.
UPDATE 2
I you want to have a better understanding of how your query is handled by sqlite, you can ask it to provide you with the query plan:
http://www.sqlite.org/eqp.html
UPDATE 3
I did not understand your context properly with my initial answer. If you are to ORDER BY on some large data set, you probably want to use that index, not the previous one, so you can tell sqlite to not use the index on FOO this way:
SELECT a, b FROM foo WHERE +a > 30 ORDER BY b
Im using WCF data services to get my data from a Silverlight application, and I have a query of which I dont know how to write.
I have 2 tables, Resources and UnavailableResources they join on Resource.id and UnavailableResources.resource_id
the UnavailableResources table holds a record of a day that a resource is not available, so it is assumed that if there is no record in the table for a given date, the resource is available.
I would like to do a query for a date range (i.e. a given week) where for each day, I get all resources (if they are available or not), and, if they are unavailable, then get the status code (inside the UnavailableResources table which joins to a status table)
How can I do this?
From what I understood from your question:
var svcContext = new ServiceContext(svcURI);
var LeftJoin = from res in svcContext.Resources
join un_res in svcContext.UnavaialableResources
on res.id equals un_res.resource_id into joinedResources
from res in joinedResources.DefaultIfEmpty()
select new
{
// Properties you need
Status = GetStatus(isAvailable,res.id)
};
You can do more joins on the LeftJoin query to get what you want.
Let me know if I got your question right.