I have a very large array of structs (more than 100k structs) that have to be saved to a file. Later, these have to be loaded and processed one at a time. The current approach is to save it using just save. This takes ~8s to save and ~100s to load.
I've tried a couple of ways to speed this up:
Using the -v6 flags with save. This sped things up, but not significantly.
Serializing and deserializing using getByteStreamFromArray() and getArrayFromByteStream() respectively. This had no effect. Specifically, serializing and deserializing took just as long as simply saving and loading.
(still working on this) Serializing the array, saving it, loading it, then only deserializing each structure as it is processed (rather than the whole array)
Does anyone have any recommendations to improve performance in this situation? It seems like it would be a common problem.
I believe that getByteStreamFromArray() and getArrayFromByteStream() are used by save() and load() under the hood, so your results are not very surprising to me. You might get better performance using hand-crafted serialization functions that crawl down your structs and only save what's really needed. Additional saving can possibly be achieved by compressing the saved data. You can read some implementation ideas here: http://undocumentedmatlab.com/blog/improving-save-performance
Note #1 - YMMV based on Matlab release, data, and platform
Note #2 - for readers who are not aware of this, getByteStreamFromArray() and getArrayFromByteStream() are both undocumented Matlab functions. The only [unofficial] explanation of their behavior, AFAIK, is provided here: http://undocumentedmatlab.com/blog/serializing-deserializing-matlab-data
Related
Here is the Story of current implementation:
I have over 50 Ad Campaigns. To track the user-behavior, I have implemented Floodlight Count tag for all of those. However, it is eating up lot of container size. Therefore, I am looking for a solution with which, I can dynamically fire Floodlights or without implementing Floodlights, I can get the similar result.
I already have implemented this solution. However, it increases the Loadtime of the webpage as it contains RegEx table(and my RegEx table has over 50 entries).
I am looking for a solution which involves minimal use of Custom variables by mostly using what is available by-default in GTM.
What's available by default in GTM (not counting custom variables) wouldn't be sufficient for covering 50 ad campaigns in an effective manner. You could always create 50 triggers and 50 tags, hardcoding your ids and maintain them separately in different tags, which is quite far from being the best practice. It actually may be the worst practice, especially given how the container size is hard-limited for GTM.
Even a large rLUT should be very insignificant given that the variable against which it's executed is reasonably small and the url is most definitely a tiny string. Have you actually measured the page load speed with and without the rLUT? Have you debugged the load speed issues and narrowed them down to the rLUT? It's actually quite an achievement to make GTM significantly affect the page load speed, given how it's all async and non-blocking.
You could reimplement the rLUT in a CSJ variable, which may be more preferable if you're good at JS, but it won't improve performance. Just like an rLUT won't hinder performance.
I'm trying to find a way to serialize execution/stack state, in such a way that the state can be archived and restored at a later time where execution can be made to pick up where it left off. Sort of like continuations, but with the feature that the stack state should be able to be serialized to disk and rehydrated on subsequent runs. I'm working in C (and/or Objective-C, if that helps.)
Protothreads looked somewhat close to what I'm looking for, but uses the GCC labels-as-values extension to resume from stored state. It seems to me that this is not likely to be robust for serialization/deserialization of the state across different compilations (and probably not even across runs of the same binary in the presence of ASLR.) In the abstract, there are going to be versioning challenges to be sure, but it doesn't seem like protothreads even gets that far.
I guess the canonical approach here would be to convert the code to a state machine, and then serialize/deserialize the state of the SM, but I find it a lot easier to think in terms of code, and struggle to envision how one would go about translating large swaths of non-trivial code into a state machine (even assuming that said code already has no global state or non-serializable heap structures, etc.)
It seems likely that any viable approach is going to require you to make allowances for it in your code. I imagine you'd have to store all your stack state in a "managed" stack structure that mirrored the stack, etc. I suspect it won't come "free", but I guess I'm hoping someone has already invented this wheel.
Anyone know of tools/libraries that address this problem, or alternately some method for converting arbitrary (but conforming) code into a serializable state machine? I found ragel, which goes the other way (SM specification -> code), but nothing generalize going this way.
I have subsequently discovered that C#'s compiler converts yield-based IEnumerator implementations into state machines in private/anonymous inner classes, so I guess it is possible to automate this for arbitrary code. Anyone know of a generalized method of doing this? (i.e. assuming you could structure the code like a yielding iterator.)
The bulk of the examples I can find for libxml2 are all about loading/parsing XML files. But I'm only interested in writing them; the code will never have to parse any files. There is an example using different writers, where it shows how to use the file, memory, DOM and tree models.
Looking through the code, I don't see any significant differences between them when it comes to writing. How does one decide which is better to use? (In other words, in what cases is one better than the others?)
The differences between the 4 functions you specify are minimal, it's all about where the contents go. As Alex mentioned, if memory is a concern, using xmlNewTextWriterFilename has the advantage of not needing to hold the result in memory.
The xmlWriter API, to which all the methods you mentioned belong, is one of the APIs offered. The other of note is the tree API. xmlWriter is more like calling write() to print to a file, and the tree is more like building nested structs in memory.
The tree-based versions can be good if your data is constructed in a non-linear fasion, going back and adding/changing things based on later information, etc. This would require some workarounds/caching with the streaming xmlWriter interface, as you can't change things once they've been output. The in-memory tree, however, can be fully tweaked until the instant it's serialized.
The tree API has the downside of the fact it has to keep the entire thing im memory; the rule of thumb is the memory requirements for a parsed tree is rougly 4x the size of serialized xml file.
My decision is usually dependent on whether I expect to create large documents. If not, I use the if the tree api, as the flexibility will be there if I want it. If I know efficiency will be a concern or I'll be working with large stuff, the streaming xmlWriter is the way to go.
tree API examples can be found here: http://xmlsoft.org/examples/index.html#Tree
If you're on a device with limited memory, you probably don't want to use DOM or memory-based approaches. In that case, you probably want to write out the file as you iterate through the data structure you want to write to XML.
In many embedded applications there is a tradeoff between making the code very efficient or isolating the code from the specific system configuration to be immune to changing requirements.
What kinds of C constructs do you usually employ to achieve the best of both worlds (flexibility and reconfigurabilty without losing efficiency)?
If you have the time, please read on to see exactly what I am talking about.
When I was developing embedded SW for airbag controllers, we had the problem that we had to change some parts of the code every time the customer changed their mind regarding the specific requirements. For example, the combination of conditions and events that would trigger the airbag deployment changed every couple weeks during development. We hated to change that piece of code so often.
At that time, I attended the Embedded Systems Conference and heard a brilliant presentation by Stephen Mellor called "Coping with changing requirements". You can read the paper here (they make you sign-up but it's free).
The main idea of this was to implement the core behavior in your code but configure the specific details in the form of data. The data is something you can change easily and it can even be programmable in EEPROM or a different section of flash.
This idea sounded great to solve our problem. I shared this with my colleague and we immediately started reworking some of the SW modules.
When trying to use this idea in our coding, we encountered some difficulty in the actual implementation. Our code constructs got terribly heavy and complex for a constrained embedded system.
To illustrate this I will elaborate on the example I mentioned above. Instead of having a a bunch of if-statements to decide if the combination of inputs was in a state that required an airbag deployment, we changed to a big table of tables. Some of the conditions were not trivial, so we used a lot of function pointers to be able to call lots of little helper functions which somehow resolved some of the conditions. We had several levels of indirection and everything became hard to understand. To make a long story short, we ended up using a lot of memory, runtime and code complexity. Debugging the thing was not straightforward either. The boss made us change some things back because the modules were getting too heavy (and he was maybe right!).
PS: There is a similar question in SO but it looks like the focus is different. Adapting to meet changing business requirements?
As another point of view on changing requirements ... requirements go into building the code. So why not take a meta-approach to this:
Separate out parts of the program that are likely to change
Create a script that will glue parts of source together
This way you are maintaining compatible logic-building blocks in C ... and then sticking those compatible parts together at the end:
/* {conditions_for_airbag_placeholder} */
if( require_deployment)
trigger_gas_release()
Then maintain independent conditions:
/* VAG Condition */
if( poll_vag_collision_event() )
require_deployment=1
and another
/* Ford Conditions */
if( ford_interrupt( FRONT_NEARSIDE_COLLISION ))
require_deploymen=1
Your build script could look like:
BUILD airbag_deployment_logic.c WITH vag_events
TEST airbag_deployment_blob WITH vag_event_emitter
Thinking outloud really. This way you get a tight binary blob without reading in config.
This is sort of like using overlays http://en.wikipedia.org/wiki/Overlay_(programming) but doing it at compile-time.
Our system is subdivided into many components, with exposed configuration and test points. There is a configuration file that is read at start-up that actually helps us instantiate components, attach them to each other, and configure their behavior.
It's very OO-like, in C, with the occasional hack to implement something like inheritance.
In the defense/avionics world software upgrades are very strictly controlled, and you can't just upgrade SW to fix issues... however, for some bizarre reason you can update a configuration file without a major fight. So it's been darn useful for us to be able to specify a lot of our implementation in those configuration files.
There is no magic, just good separation of concerns when designing the system and a bit of foresight on the part of the developers.
What are you trying to save exactly? Effort of code re-work? The red tape of a software version release?
It's possible that changing the code is reasonably straight-forward, and quite possibly easier than changing data in tables. Moving your often-changing logic from code to data is only helpful if, for some reason, it's less effort to modify data rather than code. That might be true if the changes are better expressed in a data form (e.g. numeric parameters stored in EEPROM). Or it might be true if the customer's requests make it necessary to release a new version of software, and a new software version is a costly procedure to build (lots of paperwork, or perhaps OTP chips burned by the chip maker).
Modularity is very good principle for these sort of things. Sounds as though you're already doing it to some degree. It's good to aim to isolate the often-changing code to as small an area as possible, and try to keep the rest of the code ("helper" functions) separate (modular) and as stable as possible.
I don't make the code immune to requirements changes per se, but I always tag a section of code that implements a requirement by putting a unique string in a comment. With the requirements tags in place, I can easily search for that code when the requirement needs a change. This practice also satisfies a CMMI process.
For example, in the requirements document:
The following is a list of
requirements related to the RST:
[RST001] Juliet SHALL start the RST with 5 minute delay when the ignition
is turned OFF.
And in the code:
/* Delay for RST when ignition is turned off [RST001] */
#define IGN_OFF_RST_DELAY 5
...snip...
/* Start RST with designated delay [RST001] */
if (IS_ROMEO_ON())
{
rst_set_timer(IGN_OFF_RST_DELAY);
}
I suppose what you could do is to specify several valid behaviors based on a byte or word of data that you could fetch from EEPROM or an I/O port if necessary and then create generic code to handle all possible events described by those bytes.
For instance, if you had a byte that specified the requirements for releasing the airbag it could be something like:
Bit 0: Rear collision
Bit 1: Speed above 55mph (bonus points for generalizing the speed value!)
Bit 2: passenger in car
...
Etc
Then you pull in another byte that says what events happened and compare the two. If they're the same, execute your command, if not, don't.
For adapting to changing requirements I would concentrate on making the code modular and easy to change, e.g. by using macros or inline functions for parameters which are likely to change.
W.r.t. a configuration which can be changed independently from the code, I would hope that the parameters which are reconfigurable are specified in the requirements, too. Especially for safety-critical stuff like airbag controllers.
Hooking in a dynamic language can be a lifesaver, if you've got the memory and processor power for it.
Have the C talk to the hardware, and then pass up a known set of events to a language like Lua. Have the Lua script parse the event and callback to the appropriate C function(s).
Once you've got your C code running well, you won't have to touch it again unless the hardware changes. All of the business logic becomes part of the script, which in my opinion is a lot easier to create, modify and maintain.
I'm working on a basic editor application. It uses an array of varying size that I want to store to disk. This will eventually be in an AIR application, but for now it's just an AS3 project in Flex.
I want to store the array in a file. The application edits the data, so it doesn't need to be human readable. I want it to be in whatever format will be quickest to store and load back into the array when I need that data again.
Any recommendations?
Edit: It strikes me that importing/exporting in such a way that it can be immediately cast as an Array() would probably be the cheapest thing rather than some sort of iterating - if that's possible. Another obvious option is getting the data as a simple comma delineated string and using the String.split() function to get an array. Though again, the question is what would be cheapest - and I'm not quite convinced that's it.
I'll also add that it needs to be in some sort of permanent file, so a shared object - while possibly the fastest, isn't really a long term solution.
I think the fastest and easiest way is to use a shared object. It stores native objects, so there is no serialization / deserialization steps involved. Just assign the value and read it back.
Performance wise, probably the fastest route as well. If you are looking for a large dataset and are sure it's an AIR app, you can use AIR's db, but that will definitely take much more work.
First, take a look at this answer.
As for saving the contents of an Array, consider JSON using the export tools provided by Adobe.