I'm having a table with about 30 rows and about 10 columns. The rows are a subrange of a much bigger set (which I do manually in order to avoid huge DOM). The columns are stored in a list like [{name: "firstname", width: 200}, {name: "married", type: "bool"}], which allows some flexibility (like showing the property "married" as a checkbox).
So there are just about 300 fields, yet the digest cycle takes about one second (on my i5-2400 CPU # 3.10GHz).
I'm having troubles interpreting the Batarang performance page. It says
p.name | translate 16.0% 139.6ms
e[c.name] 15.8% 138.4ms
c.name | translate 11.1% 96.3ms
The meaning of the (sparsely named) variables is clear to me:
e stands for entity, i.e., table row.
p stands for property and only occurs outside of the table.
c stands for column.
e[c.name] stands for the field content (from entity e the property named by c).
But the performance figures make little sense:
p.name gets only used maybe 10 times, how can it take that long?
c.name | translate occurs only 10 times, too (in the header row), how can it take that long?
I'm aware of {::a_once_only_bound_expression}, and I tried it, but without much success. What I'd actually need is the following:
When c changes, re-create the whole table (this happens only exceptionally, so I don't care about speed).
When e changes, re-create its whole row (when there's a change, then only in a single row).
Any way to achieve this?
A solution idea
I guess, what I need could be achieved using a directive stripping off all angular stuff from the row after rendering:
drop all child scopes
with all their watches
but keep all the HTML and listeners
I could add a single watch per row responsible for the repaint if needed.
Does it make sense?
Update
I've been rather busy working on the application - improving other things than performance. I was very lucky and got some performance as a bonus. Then I simplified the pages a little bit and the speed is acceptable now. At least for now.
Still:
I don't trust the above Batarang performance values.
I'm still curious how to implement the above solution idea and if it makes sense.
You may wanna look NgTable which places inputs from json data as rows and columns maybe solves your performance issue either, I recommend checkout for that
Related
So, I have this query:
results, cursor, more = MyModel.query(
ancestor=mykey,
).order(-MyModel.time).fetch_page(20)
So far so good, data returned is fine etc. Now, let's fetch some more, shall we? Seems logical to do just this:
results, cursor, more = MyModel.query() \
.order(-MyModel.time) \
.fetch_page(20, start_cursor=Cursor(urlsafe=request.cursor))
And... weird things happen. Definetely too many results, unordered results... What's going on?
So I change it to:
results, cursor, more = MyModel.query(ancestor=mykey) \
.order(-MyModel.time) \
.fetch_page(20, start_cursor=Cursor(urlsafe=request.cursor))
Suddenly, wat less results... let's add
.order(-MyModel.time)
And I get what I expected.
Now... Am I missing something here? Shouldn't passing cursor already take care of ordering and ancestor? There is ordering example for fetching the initial page in the documentation - https://cloud.google.com/appengine/docs/python/ndb/queries#cursors - but nowhere it is said, that subsequent pages also require ordering to be set. I would just like to know, if that is really working as intended, or it's a bug?
If it's really working as intended, is there anywhere I can read about what information exactly is stored in cursor? Would be really helpful to avoid bugs like this in future.
From Query Cursors (highlight from me):
A query cursor is a small opaque data structure representing a
resumption point in a query. This is useful for showing a user a
page of results at a time; it's also useful for handling long jobs
that might need to stop and resume. A typical way to use them is with
a query's fetch_page() method. It works somewhat like fetch(), but it
returns a triple (results, cursor, more). The returned more flag
indicates that there are probably more results; a UI can use this, for
example, to suppress a "Next Page" button or link. To request
subsequent pages, pass the cursor returned by one fetch_page() call
into the next.
A cursor exists (and makes sense) only in the context of the original query from which it was produced, you can't use the cursor produced in the context of one query (the ancestor query in your case) to navigate results from another query (your non-ancestor query). I mean it might not barf (as your experiment proves) but the results are likely not what you expect :)
Fundamentally the cursor simply represents the current position (index if you want) in the list of the query's result. Using that index in some other list might not crash, but won't make a lot of sense either (unless specifically designed to).
Probably a good habit to use a variable to store the query for re-use instead of re-building it every time, to avoid such accidental mistakes. As illustrated in the snippets.py example on that doc:
# Set up.
q = Bar.query()
q_forward = q.order(Bar.key)
q_reverse = q.order(-Bar.key)
# Fetch a page going forward.
bars, cursor, more = q_forward.fetch_page(10)
# Fetch the same page going backward.
r_bars, r_cursor, r_more = q_reverse.fetch_page(10, start_cursor=cursor)
Side note: this example actually uses the cursor from one query to navigate results in another query, but the 2 queries are designed to be "compatible".
I have two view objects in Oracle ADF.
LineVO represents order lines -- with one line per product.
Products are differentiated by several attributes... say "model" and "color". So, VO #1 contains a row for each model/color combination.
ModelVO represents a model-level summary of the lines.
Both VOs have a "quantity" field (an Integer).
There is a ViewLink between them and each has a row accessor to the other.
I want to achieve two-way coordination between these two view objects, such that:
When a user queries data, ModelVO.Quantity equals the sum of LineVO.Quantity, for the associated rows
When a user updates any LineVO.Quantity, the ModelVO.Quantity is immediately updated to reflect the new total
When a user updates a ModelVO.Quantity, the quantity is spread among the associated LineVO rows (according to complex business logic which I hope is not relevant here).
I have tried many different ways to do this and cannot get it perfect.
Right now, I am working on variations where ModelVO.Quantity is set to a Groovy expression "LineVO.sum('Quantity')". Unfortunately, everything I try either has the summing from LineVO->ModelVO working or the spreading from ModelVO->LineVO working, but never both at the same time.
Can someone suggest a way to do this? I want to do it in the model layer (either a EO or VO or combination).
Nevermind.. it turns out to be simple:
ModelVO.Quantity must be set to a Groovy "LineVO.sum('Quantity')" and it must have a recalcExpression set to a method where I can control things so it only recalculates when I am changing a LineVO.Quantity value.
The reason my approach didn't work initially was because, when the user updated a LineVO.Quantity value and I wanted to recalculate, I was getting the ModelVO row by lineVORow.getModelVO()... i.e., via a view accessor.
Apparently, that returns some sort of internal copy of the row and not the actual row.
When I got the parent row via the applicationModule.getModelVO().getCurrentRow(), the whole thing works perfectly.
I've posted another question about why accessing the row via the view accessor did not work. That part is still a mystery to me.
I just inherited some cakePHP code and I am not very familiar with it (or any other php/serverside language). I need to set the id of the item I am adding to the database to be the value of the last item plus one, originally I did a call like this:
$id = $this->Project->find('count') + 1;
but this seems to add about 8 seconds to my page loading (which seems weird because the database only has about 400 items) but that is another problem. For now I need a faster way to find the id of the last item in the database, is there a way using find to quickly retrieve the last item in a given table?
That's a very bad approach on setting the id.
You do know that, for example, MySQL supports auto-increment for INT-fields and therefore will set the id automatically for you?
The suggested functions getLastInsertId and getInsertId will only work after an insert and not always.
I also can't understand that your call adds 8 seconds to your siteload. If I do such a call on my table (which also has around 400 records) the call itself only needs a few milliseconds. There is no delay the user would notice.
I think there might be a problem with your database-setup as this seems very unlikely.
Also please have a look if your database supports auto-increment (I can't imagine that's not possible) as this would be the easiest way of adding your wanted functionality.
I would try
$id = $this->Project->getLastInsertID();
$id++;
The method can be found in cake/libs/model/model.php in line 2768
As well as on this SO page
Cheers!
If you are looking for the cakePHP3 solution to this you simply use last().
ie:
use Cake\ORM\TableRegistry;
....
$myrecordstable=Tableregistry::get('Myrecords');
$myrecords=$myrecordstable->find()->last();
$lastId = $myrecords->id;
....
I've got some SQL which performs complex logic on combinations of GL account numbers and cost centers like this:
WHEN (#IntGLAcct In (
882001, 882025, 83000154, 83000155, 83000120, 83000130,
83000140, 83000157, 83000010, 83000159, 83000160, 83000161,
83000162, 83000011, 83000166, 83000168, 83000169, 82504000,
82504003, 82504005, 82504008, 82504029, 82530003, 82530004,
83000000, 83000100, 83000101, 83000102, 83000103, 83000104,
83000105, 83000106, 83000107, 83000108, 83000109, 83000110,
83000111, 83000112, 83000113, 83100005, 83100010, 83100015,
82518001, 82552004, 884424, 82550072, 82552000, 82552001,
82552002, 82552003, 82552005, 82552012, 82552015, 884433,
884450, 884501, 82504025, 82508010, 82508011, 82508012,
83016003, 82552014, 81000021, 80002222, 82506001, 82506005,
82532001, 82550000, 82500009, 82532000))
Overall, the whole thing is poorly performing in a UDF, especially when it's all nested and the order of the steps is important etc. I can't make it table-driven just yet, because the business logic is so terribly convoluted.
So I'm doing a little exploratory work in moving it into SSIS to see about doing it in a little bit of a different way. Inside my script task, however, I've got to use VB.NET, so I'm looking for an alternative to this:
Select Case IntGLAcct = 882001 OR IntGLAcct = 882025 OR ...
Which is obviously a lot more verbose, and would make it terribly hard to port the process.
Even something like ({90605, 90607, 90610} AS List(Of Integer)).Contains(IntGLAcct) would be easier to port, but I can't get the initializer to give me an anonymous array like that. And there are so many of these little collections, I'm not sure I can create them all in advance.
It really all NEEDS to be in one place. The business changes this logic regularly. My strategy was to use the udf to mirror their old "include" file, but performance has been poor. Now each of the functions takes just 2 or three parameters. It turns out that in a dark corner of the existing system they actually build a multi-million row table of all these results - even though the pre-calced table is not used much.
So my new experiment is to (since I'm still building the massive cross join table to reconcile that part of the process) go ahead and use the table instead of the code, but go ahead and populate this table during an SSIS phase instead of calling the udf 12 million times - because my udf version just basically stopped working within a reasonable time frame and the DBAs are not of much help right now. Yet, I know that SSIS can process these rows pretty efficiently - because each month I bring in the known good results dozens of multi-million row tables from the legacy system in minutes AND run queries to reconcile that there are no differences with the new versions.
The SSIS code would theoretically become the keeper of the business logic, and the efficient table would be built from that (based on all known parameter combinations). Of course, if I can simplify the logic down to a real logic table, that would be the ultimate design - but that's not really foreseeable at this point.
Try this:
Array.IndexOf(New Integer() {90605, 90607, 90610}, IntGLAcct) >-1
What if you used a conditional split transform on your incoming data set and then used expressions or something similar (I'm not sure if your GL Accounts are fixed or if you're going to dynamically pass them in) to apply to the results? You can then take the resulting data from that and process as necessary.
So, I have an autocomplete dropdown with a list of townships. Initially I just had the 20 or so that we had in the database... but recently, we have noticed that some of our data lies in other counties... even other states. So, the answer to that was buy one of those databases with all towns in the US (yes, I know, geocoding is the answer but due to time constraints we are doing this until we have time for that feature).
So, when we had 20-25 towns the autocomplete worked stellarly... now that there are 80,000 it's not as easy.
As I type I am thinking that the best way to do this is default to this state, then there will be much less. I will add a state selector to the page that defaults to NJ then you can pick another state if need be, this will narrow down the list to < 1000. Though, I may have the same issue? Does anyone know of a work around for an autocomplete with a lot of data?
should I post teh codez of my webservice?
Are you trying to autocomplete after only 1 character is typed? Maybe wait until 2 or more...?
Also, can you just return the top 10 rows, or something?
Sounds like your application is suffocating on the amount of data being returned, and then attempted to be rendered by the browser.
I assume that your database has the proper indexes, and you don't have a performance problem there.
I would limit the results of your service to no more than say 100 results. Users will not look at any more than that any how.
I would also only being retrieving the data from the service once 2 or 3 characters are entered which will further reduce the scope of the query.
Good Luck!
Stupid question maybe, but... have you checked to make sure you have an index on the town name column? I wouldn't think 80K names should be stressing your database...
I think you're on the right track. Use a series of cascading inputs, State -> County -> Township where each succeeding one grabs the potential population based on the value of the preceding one. Each input would validate against its potential population to avoid spurious inputs. I would suggest caching the intermediate results and querying against them for the autocomplete instead of going all the way back to the database each time.
If you have control of the underlying SQL, you may want to try several "UNION" queries instead of one query with several "OR like" lines in its where clause.
Check out this article on optimizing SQL.
I'd just limit the SQL query with a TOP clause. I also like using a "less than" instead of a like:
select top 10 name from cities where #partialname < name order by name;
that "Ce" will give you "Cedar Grove" and "Cedar Knolls" but also "Chatham" & "Cherry Hill" so you always get ten.
In LINQ:
var q = (from c in db.Cities
where partialname < c.Name
orderby c.Name
select c.Name).Take(10);