Updates to RRD fail after new datasource added - database

A little background
I have a single RRD that exists to hold aggregated values of 1500+ individual RRDs (there are 1500+ devices i am monitoring). I do this so that I do not hit 1500+ RRDs when I am looking to get values from every monitored device that holds the data I am looking for. I am constantly growing this group of monitored devices so I do some xml editing (much like the contrib perl script that adds new datasources to an already existing RRD) to account for my new devices. the update to the RRD happens once an hour.
the RRD was created with this
--step 3600
--start now
DS: [$cabinet-totalw] :GAUGE:7200:U:U"
RRA:AVERAGE:0.5:1:4392
RRA:AVERAGE:0.5:24:366
RRA:AVERAGE:0.5:744:36
RRA:MIN:0.5:24:732
RRA:MAX:0.5:24:732
FYI - $cabinet-totalw is in fact a variable in a for loop. The initial build looped through something like 1300 cabinets. I didn't want to list everything here.
The issue
As a new device is added to the monitored group, the datasource is added to the aggregation RRD file. However, when the update fires, it doesn't actually update the RRD for some unknown reason. when i do this manually updatev exists with a zero. if i look at xport output, i have NAN for any new datasource data. however, all existing datasources seem to update without issue.
At the moment I'm lost as to why this is happening. Things seem to be in order yet the update to the new RRD datasources does not take. even more interesting is that i've added datasources to this file in the past and have had those update without issue. it just seems to be recent updates do not take.
I should also add that lastupdate does in fact show the ... well last update correctly. so i assume its a lack of RRD knowledge on my part?
ADDITION
I wrote a script that grabs the index of the DS i am interested in. I then parse through the output of a rrdtool fetch to find that requested value based on the index per time interval. I found that the values are in fact being updated and stored in the RRD. Interestingly enough, the timestamp is showing 7 mins after an allotted time slot (step is 3600 so data should be stored on the hour). I now believe this to be an interpolation issue?

I found my issue. When i am updating the rrd data in the xml file (after it has been dumped) i was mistakenly adding wrong default values to the ds value and the min/max values. needed to change node values from NaN to 0.0000000000e+00 and min/max values from 0.0000000000e+00 to NaN.
thanks if anyone was trying to help.

Related

Data Studio Linking API and Calculated Fields

I am working with the Linking API and am finding that if I include ANY of the "ds." parameters in the URL, that the resulting data source loses all calculated fields from the original data source. Based on the documentation, it seems like we should be able specify some of the "ds." parameters, but as long as we don't specify ds.connector, it should still Update (which would keep the calculated fields?). But I'm finding that the only way to get the resulting data source to include calculated fields is to specify NONE of the ds. parameters. Any assistance would be so appreciated!
This is a bug in Data Studio, thanks for flagging it!
We aren't copying calculated fields or parameters from the original datasource unless ds.<alias>.refreshFields is false, and the default is true for Sheets and BigQuery datasources. I suspect that you're only seeing this when you add a "ds" parameter because when you don't have any "ds" parameters and your original data source is reusable, that original data source gets added to the new report instead of creating a new one.
We have a fix that should be going out in the release tomorrow (July 26, ~2pm PDT) but you can work around it in the meantime by adding either
ds.<alias>.refreshFields=false (to fix one datasource) or ds.*.refreshFields=false (to fix all datasources, if you have more than one).

Values from Analytics different in a table than a time series

I'm having an issue where the figures pulled from Analytics are different in a table format than the time series.
As you can see in the image below the 'Nutzer' (user) value in the table for Sep 2019 is 6692 but on the time series is 7789. This then affects the calculated values for 'Umsatz pro Nutzer'.
Does anyone know why this happens and how to stop it?
I think the most likely answer is that there is a filter active on one or both of the objects that is altering the displayed data.
To test this out, create a copy of the sheet and delete the chart. Then make a copy of your table and change it to a time series using the chart options. I just did this process using a connection to the GA property for our mobile app and I see identical data for users for both the table and the time series.

Is incremental adds are possible in neo4j?

I have a quick question. I have database of one million nodes and 4 million relationships. This all data in neo4j i have created with import csv command. Now after testing the graph database and analyzing the queries according to my need. Now i want to make a php program where the data will be automatically loaded and i will get the results in the end (according to my query). Now here is the question, as my data will update after 15 min. Is neo4j has a ability of incremental adds. Like to show which new relationships or nodes added in this specific time.i was thinking to use the time command to see which data was created in that time. Correct me if i am wrong. i only want to see the new addition. because i dont want neo4j to waste time on the calculation of already existing nodes/relationships.is there any other way to do that.
thanks in advance.
You could add a property to store a string of the date/time that the nodes are added. Then you could query for everything since the last date/time. I'm not 100% sure on the index performance of that, though.
However, if all you care about is showing the the most recently imported, you could have a boolean value with an index:
CREATE INDEX ON :Label(recently_added)
Then when you import your data you can unset all of the current ones and set the new ones like this:
MATCH (n:Label {recently_added: true})
REMOVE n.recently_added
Second query:
MATCH (n:Label {id: {id_list}})
SET n.recently_added = true
That is assuming that you have some sort of unique identifier on the nodes which you can use to set the ones which you just added.

Can't change data type on MS Access 2007

I have a huge database (800MB) which consists of a field called 'Date Last Modified' at the moment this field is entered as a text data type but need to change it to a Date/Time field to carry out some queries.
I have another exact same database but with only 35MB of data inside it and when I change the data type it works fine, but when I try to change data type on big database it gives me an error:
Micorosoft Office Access can't change the data type.
There isn't enough disk space or memory
After doing some research some sites mentioned of changing the registry file (MaxLocksPerFile) tried that as well, but no luck :-(
Can anyone help please?
As John W. Vinson says here, the problem you're running into is that Access wants to hold a copy of the table while it makes the changes, and that causes it to exceed the maximum allowable size of an Access file. Compacting and repairing might help get the file under the size limit, but it didn't work for me.
If, like me, you have a lot of complex relationships and reports on the old table that you don't want to have to redo, try this variation on #user292452's solution instead:
Copy the table (i.e. 'YourTable') then paste Structure Only back
into your database with a different name (i.e. 'YourTable_new').
Copy YourTable again, and paste-append the data to YourTable_new.
(To paste-append, first paste, and select Append Data to Existing
Table.)
You may want to make a copy of your Access database at this point,
just in case something goes wrong with the next part.
Delete all data in YourTable using a delete query---select all
fields, using the asterisk, and then run with default settings.
Now you can change the fields in YourTable as needed and save
again.
Paste-append the data from YourTable_new to YourTable, and check
that there were no errors from type conversion, length, etc.
Delete YourTable_new.
One relatively tedious (but straightforward) solution would be to break the big database up into smaller databases, do the conversion on the smaller databases, and then recombine them.
This has an added benefit that if, by some chance, the text is an invalid date in one chunk, it will be easier to find (because of the smaller chunk sizes).
Assuming you have some kind of integer key on the table that ranges from 1 to (say) 10000000, you can just do queries like
SELECT *
INTO newTable1
FROM yourtable
WHERE yourkey >= 0 AND yourkey < 1000000
SELECT *
INTO newTable2
FROM yourtable
WHERE yourkey >= 1000000 AND yourkey < 2000000
etc.
Make sure to enter and run these queries seperately, since it seems that Access will give you a syntax error if you try to run more than one at a time.
If your keys are something else, you can do the same kind of thing, but you'll have to be a bit more tricky about your WHERE clauses.
Of course, a final thing to consider, if you can swing it, is to migrate to a different database that has a little more power. I'm guessing you have reasons that this isn't easy, but with the amount of data you're talking about, you'll probably be running into other problems as well as you continue to use Access.
EDIT
Since you are still having some troubles, here is some more detail in the hopes that you'll see something that I didn't describe well enough before:
Here, you can see that I've created a table "OutputIDrive" similar to what you're describing. I have an ID tag, though I only have three entries.
Here, I've created a query, gone into SQL mode, and entered the appropriate SQL statement. In my case, because my query only grabs value >= 0 and < 2, we'll just get one row...the one with ID = 1.
When I click the run button, I get a popup that tells/warns me what's going to happen...it's going to put a row into a new table. That's good...that's what we're looking for. I click "OK".
Now our new table has been created, and when I click on it, we can see that our one line of data with ID = 1 has been copied over to this new table.
Now you should be able to just modify the table name and the number values in your SQL query, and run it again.
Hopefully this will help you with whatever tripped you up.
EDIT 2:
Aha! This is the trick. You have to enter and run the SQL statements one at a time in Access. If you try to put multiple statements in and run them, you'll get that error. So run the first one, then erase it and run the second one, etc. and you should be fine. I think that will do it! I've edited the above to make it clearer.
Adapted from Karl Donaubauer's answer on an MSDN post:
Switch to immediate window (Ctl + G)
Execute the following statement:
DBEngine.SetOption dbMaxLocksPerFile, 200000
Microsoft has a KnowledgeBase article that addresses this problem directly and describes the cause:
The page locks required for the transaction exceed the MaxLocksPerFile value, which defaults to 9500 locks. The MaxLocksPerFile setting is stored in the Windows registry.
The KnowledgeBase article says it applies to Access 2002 and 2003, but it worked for me when changing a field in an .mdb from Access 2013.
It's entirely possible that in a database of that size, you've got text data that won't convert to a valid Date/Time.
I would suggest (and you may hate me for this) that you export all those prospective date values from "Big" and go through them (perhaps in Excel) to see which ones are not formatted the way you'd expect.
Assuming that the error message is accurate, you're running up against a disk or memory limitation. Assuming that you have more than a couple of gigabytes free on your disk drive, my best guess is that rebuilding the table would put the database (including work space) over the 2 gigabyte per file limit in Access.
If that's the case you'll need to:
Unload the data into some convenient format and load it back in to an empty database with an already existing table definition.
Move a subset of the data into a smaller table, change the data type in the smaller table, compact and repair the database, and repeat until all the data is converted.
If the error message is NOT correct (which is possible), the most likely cause is a bad or out-of-range date in your text-date column.
Copy the table (i.e. 'YourTable') then paste just its structure back into your database with a different name (i.e. 'YourTable_new').
Change the fields in the new table to what you want and save it.
Create an append query and copy all the data from your old table into the new one.
Hopefully Access will automatically convert the old text field directly to the correct value for the new Date/Time field. If not, you might have to clear out the old table and re-append all the data and use a string to date function to convert that one field when you do the append.
Also, if there is an autonumber field in the old table this might not work because there is no way to ensure that the old autonumber values will line up with the new autonumber values that get assigned.
You've been offered a bunch of different ways to get around the disk space error message.
Have you tried adding a new field to your existing table using Date data type and then updating the field with the value the existing string date field? If that works, you can then delete the old field and rename the new one to the old name. That would probably take up less temp space than doing a direct conversion from string to date on a single field.
If it still doesn't work, you may be able to do it with a sceond table with two columns, the first long integer (make it the primary key), the second, date. Then append the PK and string date field to this empty table. Then add a new date field to the existing table, and using a join, update the new field with the values from the two-column table.
This may run into the same problem. It depends on number of things internal to the Jet/ACE database engine over which we have no real control.

How to fix possible db corruption?

I'm at a client doing some quick fixes to their access application. It was a while I had a go with access, but I'm recovering quickly. However, I've discovered an interesting problem:
For some reports, I get a "Record is deleted" error. I've checked the reports, and it seems like there's a problem with one table. When opening that table, I find a record where all columns are marked "#deleted". So obviously, this row seems to be the culprit. However, when I try to delete that row, nothing really happens. If I re-open the table, the row still exists.
Is there a corruption in the db? How can I remove this record for good?
Edit: It's a MS2000-version
Solution: Simply compress/repair did not work. I converted the database to the 2003 file format instead, which did the trick. I've marked the first answer suggesting compress/repair, since it pointed me in the right direction. Thanks!
Have you tried the built in Access compact/repair tool? This should flush deleted records from the database.
The exact location varies according to the version of Access you're running, but on Access 2003 it's under Tools > Database Utilities > Compact and repair database. Some earlier versions of Access had two separate tools - one for compact, one for repair - but they were accessed from a similar location. If they are separate on the version the client has, you need to run both.
This should be a non-destructive operation, but it would be best to test this on a copy of the MDB file (apologies for stating the obvious).
Tony Toews, Access MVP, has a comprehensive guide to corruption:
Corrupt Microsoft Access MDBs FAQ
Some corruption symptoms
Determining the workstation which caused the corruption
Corruption causes
To retrieve your data
As an aside, decompile is very useful for sorting out odd happenings when coding and for improving start-up times.
you can also try this Command line utility
//andy
Compacting and importing won't fix the problem for the error reported, which is clearly a corrupted pointer for a memo field. The only thing you can do is delete and recreate the record that is causing the problem. And you need to find ways to edit memo data (or eliminate memo fields -- do you really need more than 255 characters or not?) that does not expose you to corruption risk. That means avoiding bound controls on forms for memo fields.
Instead, use an unbound textbox, and in the form's OnCurrent event, assign the current data from the form's underlying recordsource:
Me!txtMyMemo = Me!MyMemo
To save edits to the unbound control, use the control's AfterUpdate event:
Me!MyMemo = Me!txtMyMemo
Me.Dirty = False ' save the whole record
Why are memo fields subject to corruption? Because they aren't stored in the same data page as the non-memo fields, but instead, all that is in the record's main data page is a pointer to some other data page (or set of data pages if it's a large chunk of data) where the actual memo data is stored. If it weren't done this way, a record with a memo in it would very quickly exceed the maximum record length.
The pointer is relatively easily corrupted, most often by a fatal problem during editing in a bound control. Editing with an unbound control does not eliminate the problem entirely, but means that the time in which you're exposed to danger is very, very short (i.e., the time it takes for those two lines of code to execute in the AfterUpdate event).
Aside from the options already posted above, I've used another simple method aswell: Simply create a new MDB file and import all objects from the corrupted one. Don't forget to get system and/or hidden objects when you go this way.

Resources