I'm building an application that uses an embedded H2 database. I used the tutorial to test it out and everything seemed to work fine:
import java.sql.*;
public class Test {
public static void main(String[] a)
throws Exception {
Class.forName("org.h2.Driver");
Connection conn = DriverManager.
getConnection("jdbc:h2:~/test", "sa", "");
// add application code here
conn.close();
}
}
I am curious however, in my home directory I now have a "test.h2" file along with a "test.lock" file. Why does an empty database end up being 2 MB? It seems kind of large, I would expect something in KB at most, given that all that is in it would be some default empty memory and some instructions for storing data in it. Is 2 MB the default memory allocated?
The database file size of an empty (or almost empty) database is only 2 MB while the database is open. If it is closed, the file shrinks.
On some file systems, resizing files is relatively slow. Because of that, H2 allocates more space than it needs: to reduce the number of resize operations.
The exact algorithm to expand the file size may change in the future. Currently, the minimum file size is about 2 MB. The file grows at 35%, but at most 256 MB at a time.
When you close the database, the file shrinks to the real size needed.
Related
I need to serve some content from an Action in the form of a file: basically, I am creating CSV content on the fly and sending it to the client.
I cannot do it using sendFile, since the file does not really exist; I tried using the chunked transfer, but I get a really slow response (in localhost I got the file at about 100KB/s, which I think is really strange).
Is there a way for me to set the content type and write the response "line by line", without having to specify the content length "a priori"?
Here's one way using a simple predefined Enumerator that will produce the response from bytes written to an OutputStream:
def csv = Action {
val enumerator = Enumerator.outputStream { out =>
out.write(...)
// Keep writing to the Enumerator
out.close()
}
Ok.chunked(enumerator.andThen(Enumerator.eof)).withHeaders(
"Content-Type" -> "text/csv",
"Content-Disposition" -> s"attachment; filename=test.csv"
)
}
This is simple enough for relatively small files (or if the process of generating the file is slow by nature), however note that from the documentation this has no back-pressure, reading a large file into the OutputStream can quickly fill up memory if the client can't download it quickly enough.
Update:
After testing this some more it seems like the size of the Byte arrays you write to the OutputStream make a huge difference in throughput.
Using this sample stream:
val s = Stream.continually(0.toByte)
Writing in chunks of 1KB to the OutputStream like this resulted in 6MB/s of throughput:
(0 until 1024*1024).foreach{i =>
out.write(s.take(1024).toArray)
}
However if I only write 10 bytes at a time, the throughput slows to less than 100KB/s. So my suggestion for using this method to write CSVs in a chunked form would be to write multiple rows at a time to the OutputStream rather than one row at a time.
In our iPad app we need to load fbx files(of size 1MB - 100MB) as asset bundles. I started testing with a fbx file of size 26MB and it creates an asset bundle of size 17MB. When I run the test scene to load this asset bundle in the iPad it load after around 1 minute. And then I get a memory warning. I monitored the memory using Xcode and it seems that memory usage increase from around 1MB to 65Mb and then just before the model show up memory increase to around 175MB at once. Below is my code.
I have seen similar issues posted by other users. But I didn't see a proper solution to it in any of these threads. As I read in these threads, I think the memory increases when uncompressing the asset bundle .But I don't understand why it goes unto around 170MB.
What can we do to reduce the memory usage?
Thanks
public class CachingLoad : MonoBehaviour {
public string BundleURL;
public string AssetName;
public int version;
void Start() {
StartCoroutine (DownloadAndCache());
}
IEnumerator DownloadAndCache (){
// Wait for the Caching system to be ready
while (!Caching.ready)
yield return null;
BundleURL = "http://10.30.3.228:8080/TestDownload/assetBundle1.unity3d";
// Load the AssetBundle file from Cache if it exists with the same version or download and store it in the cache
using(WWW www = WWW.LoadFromCacheOrDownload (BundleURL, version)){
yield return www;
if (www.error != null)
throw new Exception("WWW download had an error:" + www.error);
AssetBundle bundle = www.assetBundle;
if (AssetName == "")
Instantiate(bundle.mainAsset);
else
Instantiate(bundle.Load(AssetName));
// Unload the AssetBundles compressed contents to conserve memory
bundle.Unload(false);
} // memory is freed from the web stream (www.Dispose() gets called implicitly)
}
}
I have developed a ETL which is consuming flat files. The size of flat files varies from 250 MB - 300 MB.
It is working absoultely fine when file present in the folder. But it fails when the file is in generation mode.
Ex: This ETL package runs from 8 AM to 10 AM to check whether the file is present in the folder or not. Now, at any instance(let say 9 AM) if the file is starting generated and till now it is 10 MB. ETL start processing the file and just hang and fail after 4-5 min ( hang at script task which is reading that the file is present in the folder or not).
What is the best way to trigger SSIS package only when the file generation is completely done?
Note: I have no control over the file generation.
Add a For Loop Container with a Boolean variable bFileAccessible:
The Init expression is #bFileAccessible=False
The Eval expression is #bFileAccessible==False
Inside the For Loop Container add a Script Task with a ReadWriteVariable User::bFileAccessible and the following C# script (showing only the Main() method):
public void Main()
{
try
{
using (Stream stream = new FileStream("Path\to\your\file", FileMode.Open))
{
Dts.Variables["bFileAccessible"].Value = true;
}
}
catch
{
Dts.Variables["bFileAccessible"].Value = false;
}
Dts.TaskResult = (int)ScriptResults.Success;
}
You should also use a variable for the filename and maybe a little wait interval. For more information about the script see here.
Check the FIle modified time everytime and comapre the same with previous one....
it's not good logic but a good idea if no perfect alternative
I have a Firbird 1.0 data file weighting aprox 25 GB that I am working with it. It has a table which has stored documents and doc's pics as blob. So, I am asking is it possible to open such big data file using fib datasets, i firstly tried to open dataset in runtime = no success as grid was empty so another try was to set it active in design mode which it was also unable to open as it's active property is set to true but no fetched data in grid!
Have you any idea to make it work ? Do I have to set any blob cashe options?
or it is not possible at all?
Now I am developing using my laptop computer (Win 7 x64 4GB Ram ), and later it'll be deployed to my server machine!
I've fixed it!
So another my question is about loading blob data using stream to a TImage component
i am doing like this but it pops out an Access violation
here is my code which you may look at
DM->stImage->Active=true;
try {
TMemoryStream *ms=new TMemoryStream();
TStream *ps=DM->stImage->CreateBlobStream(DM->stImage->FieldByName("PHOTO") ,bmRead);
ms->Position=0;
ms->CopyFrom(ps,ps->Size);
ms->SaveToFile("c:\\1.jpg");
// imgPass->Picture->LoadFromStream(ms);
imgPass->Picture->Graphic->LoadFromStream(ps);
delete ms;
delete ps;
}
catch (Exception &e) {
ShowMessage(e.ToString());
}
it can save it but imgPass->Picture->Graphic->LoadFromStream(ps); does not work!
what could be a problem?
To avoid the AV you need to reset the stream position, that was moved forward during the call to "CopyFrom" function.
So, your code should look like (only the relevant lines):
ms->CopyFrom(ps,ps->Size);
ms->SaveToFile("c:\\1.jpg");
ps->Position = 0; //<<<<<<<<<< here we reset the stream position
imgPass->Picture->Graphic->LoadFromStream(ps);
//imgPass->Picture->Bitmap->LoadFromStream(ps); // <<< if a bitmap and not JPEG
Hope this helps you.
P.S.: this question should be tagged C++ (or C++Builder) because it is not only a database subject.
Here is some background about my app:
I am developing an Android app that will display a random quote or verse to the user. For this I am using an SQLite database. The size of the DB would be approximately 5K to 10K records, possibly increasing to upto 1M in later versions as new quotes and verses are added. Thus the user would need to update the DB as and when newer versions are of the app or DB are released.
After reading through some forums online, there seem to be two feasible ways I could provide the DB:
1. Bundle it along with the .APK file of the app, or
2. Upload it to my app's website from where users will have to download it
I want to know which method would be better (if there is yet another approach other than these, please do let me know).
After pondering this problem for some time, I have these thoughts regarding the above approaches:
Approach 1:
Users will obtain the DB along with the app, and won't have to download it separately. Installation would thereby be easier. But, users will have to reinstall the app every time there is a new version of the DB. Also, if the DB is large, it will make the installable too cumbersome.
Approach 2:
Users will have to download the full DB from the website (although I can provide a small, sample version of the DB via Approach 1). But, the installer will be simpler and smaller in size. Also, I would be able to provide future versions of the DB easily for those who might not want newer versions of the app.
Could you please tell me from a technical and an administrative standpoint which approach would be the better one and why?
If there is a third or fourth approach better than either of these, please let me know.
Thank you!
Andruid
I built a similar app for Android which gets periodic updates with data from a government agency. It's fairly easy to build an Android compatible db off the device using perl or similar and download it to the phone from a website; and this works rather well, plus the user gets current data whenever they download the app. It's also supposed to be possible to throw the data onto the sdcard if you want to avoid using primary data storage space, which is a bigger concern for my app which has a ~6Mb database.
In order to make Android happy with the DB, I believe you have to do the following (I build my DB using perl).
$st = $db->prepare( "CREATE TABLE \"android_metadata\" (\"locale\" TEXT DEFAULT 'en_US')");
$st->execute();
$st = $db->prepare( "INSERT INTO \"android_metadata\" VALUES ('en_US')");
$st->execute();
I have an update activity which checks weather updates are available and if so presents an "update now" screen. The download process looks like this and lives in a DatabaseHelperClass.
public void downloadUpdate(final Handler handler, final UpdateActivity updateActivity) {
URL url;
try {
close();
File f = new File(getDatabasePath());
if (f.exists()) {
f.delete();
}
getReadableDatabase();
close();
url = new URL("http://yourserver.com/" + currentDbVersion + ".sqlite");
URLConnection urlconn = url.openConnection();
final int contentLength = urlconn.getContentLength();
Log.i(TAG, String.format("Download size %d", contentLength));
handler.post(new Runnable() {
public void run() {
updateActivity.setProgressMax(contentLength);
}
});
InputStream is = urlconn.getInputStream();
// Open the empty db as the output stream
OutputStream os = new FileOutputStream(f);
// transfer bytes from the inputfile to the outputfile
byte[] buffer = new byte[1024 * 1000];
int written = 0;
int length = 0;
while (written < contentLength) {
length = is.read(buffer);
os.write(buffer, 0, length);
written += length;
final int currentprogress = written;
handler.post(new Runnable() {
public void run() {
Log.i(TAG, String.format("progress %d", currentprogress));
updateActivity.setCurrentProgress(currentprogress);
}
});
}
// Close the streams
os.flush();
os.close();
is.close();
Log.i(TAG, "Download complete");
openDatabase();
} catch (Exception e) {
Log.e(TAG, "bad things", e);
}
handler.post(new Runnable() {
public void run() {
updateActivity.refreshState(true);
}
});
}
Also note that I keep a version number in the filename of the db files, and a pointer to the current one in a text file on the server.
It sounds like your app and your db are tightly bound -- that is, the db is useless without the database and the database is useless without the app, so I'd say go ahead and put them both in the same .apk.
That being said, if you expect the db to change very slowly over time, but the app to change quicker, and you don't want your users to have to download the db with each new app revision, then you might want to unbundle them. To make this work, you can do one of two things:
Install them as separate applications, but make sure they share the same userID using the sharedUserId tag in the AndroidManifest.xml file.
Install them as separate applications, and create a ContentProvider for the database. This way other apps could make use of your database as well (if that is useful).
If you are going to store the db on your website then I would recommend that you just make rpc calls to your webserver and get data that way, so the device will never have to deal with a local database. Using a cache manager to avoid multiple lookups will help as well so pages will not have to lookup data each time a page reloads. Also if you need to update the data you do not have to send out a new app every time. Using HttpClient is pretty straight forward, if you need any examples please let me know