I have a Firbird 1.0 data file weighting aprox 25 GB that I am working with it. It has a table which has stored documents and doc's pics as blob. So, I am asking is it possible to open such big data file using fib datasets, i firstly tried to open dataset in runtime = no success as grid was empty so another try was to set it active in design mode which it was also unable to open as it's active property is set to true but no fetched data in grid!
Have you any idea to make it work ? Do I have to set any blob cashe options?
or it is not possible at all?
Now I am developing using my laptop computer (Win 7 x64 4GB Ram ), and later it'll be deployed to my server machine!
I've fixed it!
So another my question is about loading blob data using stream to a TImage component
i am doing like this but it pops out an Access violation
here is my code which you may look at
DM->stImage->Active=true;
try {
TMemoryStream *ms=new TMemoryStream();
TStream *ps=DM->stImage->CreateBlobStream(DM->stImage->FieldByName("PHOTO") ,bmRead);
ms->Position=0;
ms->CopyFrom(ps,ps->Size);
ms->SaveToFile("c:\\1.jpg");
// imgPass->Picture->LoadFromStream(ms);
imgPass->Picture->Graphic->LoadFromStream(ps);
delete ms;
delete ps;
}
catch (Exception &e) {
ShowMessage(e.ToString());
}
it can save it but imgPass->Picture->Graphic->LoadFromStream(ps); does not work!
what could be a problem?
To avoid the AV you need to reset the stream position, that was moved forward during the call to "CopyFrom" function.
So, your code should look like (only the relevant lines):
ms->CopyFrom(ps,ps->Size);
ms->SaveToFile("c:\\1.jpg");
ps->Position = 0; //<<<<<<<<<< here we reset the stream position
imgPass->Picture->Graphic->LoadFromStream(ps);
//imgPass->Picture->Bitmap->LoadFromStream(ps); // <<< if a bitmap and not JPEG
Hope this helps you.
P.S.: this question should be tagged C++ (or C++Builder) because it is not only a database subject.
Related
Windows 10 Pro
Latest Simulator
Java Swing Project
I would like to execute "Vector a1 = (Vector) Storage.getInstance().readObject(filePath);"
In a Java Swing Application running on Windows 10 platform, I tried import CodenameOne.jar in Swing package, however when executing above code, get null pointer exception in Storage.getInstance()
Is there a way to execute this in Swing?
Thoughts?
Best Regards.
Thanks, I did not init the Display, however "Display.init(Object m)" requires an Object Argument and the Init method is deprecated.
Can you please provide me the codenameone Display dependencies?
And perhaps a java Swing snippet of code to initialize Display in order to execute Storage.getInstance().readObject(filePath)
Thoughts?
Best Regards
Thanks, Passing init(working directory) solved the Exception thrown.
Here is the Code snippet used to allow me to execute:
Storage.getInstance().readObject(filePath).
String filePath = incSrv.Pwd();// gets working directory
try {
javax.swing.SwingUtilities.invokeLater(new Runnable() {
public void run() {
Display.init(filePath);
String fileName = "A1-MMA.properties";
Vector a1 = (Vector) Storage.getInstance().readObject(filePath);
}
});
} catch (Exception e) {
}
And it does appear to work,
However I am left with a blacked out form that appears modal.
How can I avoid this or dispose it?.
FYI: What I am creating here is a work around for serializing Vector in Codenameone. I Save Vector to file using "Storage.getInstance().writeObject(Path, Vector)"
I convert the file to bytes and write it to the Swing Server VIA socket.
Using Storage.getInstance().readObject(file) on the Swing Server I have deserialize the object into the Vector from my app.
This appears to work well and is more efficient than the current method I use to deliver complex Vectors from the app to the Swing Server.
Can you please let me know if you see a red flag with this workaround?
Like The ability to Storage.getInstance().readObject(file) on the Swing Server may go away?
This method will save a lot of time in movind Vector data to and from the App/Server.
Thoughts Best Regards
Storage.getInstance().readObject(file) // (A1ServiceSrv.java:571)
Caused this Exception:
java.lang.NullPointerException
at com.codename1.io.Storage.init(Storage.java:89)
at com.codename1.io.Storage.getInstance(Storage.java:112)
at Main.A1ServiceSrv.loadVectorFromFile(A1ServiceSrv.java:571)
Regards
12/11/2021:
Thanks Shai,
I am including in my classpath CodenameOne.jar with update date of 12/11/2021 after CN1 refresh.
Getting Same null pointer exception.
Passing in Path "C:\Src1\A1-Arms\A1-Server\A1-MMA.properties" (Absolute Path)
Also Tried "A1-MMA.properties", however I don't think Codenameone knows where my home path is since we are not initializing it as we did with
Display.init("Current Working Directory where files reside");
This is the Fresh Stack Trace w/o calling Display.init (12-20-2021)
java.lang.NullPointerException at
com.codename1.ui.Display.getResourceAsStream(Display.java:3086)
at com.codename1.io.Log.print(Log.java:327)
at com.codename1.io.Log.logThrowable(Log.java:299)
at com.codename1.io.Log.e(Log.java:285)
at com.codename1.io.Storage.readObject(Storage.java:271)
at Main.A1ServiceSrv.loadVectorFromFile(A1ServiceSrv.java:596)
vector = (Vector) Storage.getInstance().readObject(filePath); // (A1ServiceSrv.java:596)
This is hopefully fixed by this commit: https://github.com/codenameone/CodenameOne/commit/72bf283bdaaefe5207bb9fd6787578e3ef61522c if not let me know with a fresh stack
I have the two following methods and I am using them to store a special value locally and be able to access it on application restart:
(Store value locally:)
private void SaveSet(string key, string value)
{
ISharedPreferences prefs = PreferenceManager.GetDefaultSharedPreferences(this);
ISharedPreferencesEditor prefEditor = prefs.Edit();
prefEditor.PutString(key, value);
// editor.Commit(); // applies changes synchronously on older APIs
prefEditor.Apply(); // applies changes asynchronously on newer APIs
}
(Read it again:)
private string RetrieveSet(string key)
{
ISharedPreferences prefs = PreferenceManager.GetDefaultSharedPreferences(this);
return prefs.GetString(key, null);
}
This works perfectly. Now is it possible to access and edit this Shared Preferences externally? Unfortunately, I cannot find any file when searching in folder
Phone\Android\data\com.<company_name>.<application_name>\files
nor anywhere else. I want / try to edit this value from my computer, after connecting the phone to it. Is this possible?
Alternatively: Can anyone maybe show me how to create a new file in the given path above, write/read it programmatically and how it stays there, even if application is closed / started again? So I can then edit this file with my computer anyhow?
I tried it with the following code, but unfortunately it doesn't work / no file is created or at least i cannot see it in the given path above:
//"This code snippet is one example of writing an integer to a UTF-8 text file to the internal storage directory of an application:"
public void SaveValueIntoNewFile(int value)
{
var backingFile = Path.Combine(System.Environment.GetFolderPath(System.Environment.SpecialFolder.Personal), "newFile.txt");
using (var writer = System.IO.File.CreateText(backingFile))
{
writer.WriteLine(value.ToString());
}
}
Would be very happy about every answer, thanks in advance and best regards
What you're looking for is where Android stores the Shared Preference file for applications that make use of it's default PreferenceManager.
I'd refer to this SO post which answers your question pretty well
SharedPreferences are stored in an xml file in the app data folder,
i.e.
/data/data/YOUR_PACKAGE_NAME/shared_prefs/YOUR_PREFS_NAME.xml
or the default preferences at:
/data/data/YOUR_PACKAGE_NAME/shared_prefs/YOUR_PACKAGE_NAME_preferences.xml
SharedPreferences added during runtime are not stored in the Eclipse
project.
Note: Accessing /data/data/ requires superuser
privileges
A simple method is to use Android Device Monotor,you can open it by clicking Tools--> android-->Android Device Monotor...
For example:
The path in my device is as follows:
/data/data/YOUR_PACKAGE_NAME/shared_prefs/YOUR_PACKAGE_NAME_preferences.xml
And we notice three buttons in the upper right corner of the picture.
The first one is used toPull a file from the device,the second one is used to Push a file onto the device,and the last one is used to delete the preferences.xml file.
So we can pull the preferences.xml file from current device to our computer and edit it as we want, and then push the updated preferences.xml to the folder again.Then we will get the value of preferences.xml file .
I want to write entries to a log file stored in Azure file storage. I currently have this:
var log = "My log entry";
var client = _storageAccount.CreateCloudFileClient();
var share = client.GetShareReference(Config.LogShare);
share.CreateIfNotExists();
var root = share.GetRootDirectoryReference();
var logfile = root.GetFileReference("log.txt");
if (!logfile.Exists()) logfile.Create(0);
// What goes here to append to the file...?
I can see plenty of examples of how to do this with Blobs, or how to upload an entire file, but how do I just append to an existing file?
I have tried this:
var buffer = Encoding.GetEncoding("UTF-8").GetBytes(log.ToCharArray());
using (var fileStream = logfile.OpenWrite(0)) {
fileStream.Write(buffer, (int)logfile.Properties.Length, buffer.Length);
}
But then I get this error:
The remote server returned an error: (416) The range specified is invalid for the current size of the resource..
I managed to work this out myself. You just need to increase the size of the file by the number of new bytes you want to write to it, and then write the new data to that new empty space at the end of the file, like this:
var client = _storageAccount.CreateCloudFileClient();
var share = client.GetShareReference(Config.LogShare);
share.CreateIfNotExists();
var root = share.GetRootDirectoryReference();
var logfile = root.GetFileReference("log.txt");
if (!logfile.Exists()) logfile.Create(0);
var buffer = Encoding.UTF8.GetBytes($"{log}\r\n");
logfile.Resize(logfile.Properties.Length + buffer.Length);
using (var fileStream = logfile.OpenWrite(null)) {
fileStream.Seek(buffer.Length * -1, SeekOrigin.End);
fileStream.Write(buffer, 0, buffer.Length);
}
You can do this with blobs https://blogs.msdn.microsoft.com/windowsazurestorage/2015/04/13/introducing-azure-storage-append-blob/
Shame it doesn't work with files too
Azure file storage REST API doesn't support appending to an existing file. To achieve this, please mount the file share to your machine as a drive, and append to the file just like simple local files.
Actually, I don't think you really need appending functionality per your code above. You can specify the file size in CloudFile.OpenWrite() / CloudFile.Create(), or try CloudFile.UploadFromStream() instead of CloudFile.OpenWrite().
This error could also be due to multi-threaded access.
I bet if you tried to lock the file before you access it, you will not face this problem.
There are many ways to update the file.
Since you already managed to get the share, the root, the folder and the file.. Here is a portion of my code that worked for me.
if (!fileLock.IsWriteLockHeld) fileLock.EnterWriteLock();
try
{
using (var stream = new MemoryStream(content, false))
{
file.UploadFromStream(stream, null, options);
}
}
catch (Exception ex)
{
File.AppendAllText(FileName, ex.ToString());
}
finally
{
if (fileLock.IsWriteLockHeld)
fileLock.ExitWriteLock();
}
Where fileLock is declared as:
protected ReaderWriterLockSlim fileLock = new ReaderWriterLockSlim();
Having said that, I am not saying that this is the best way ever to do it.
The two things I would like you to keep in mind :
1-Lock the resource that is likely to be accessed by more than one thread (That is so common in AZURE)
2- Get familiar with asynchronous methods that Azure provides.. use them when they suit well.
Coming back to your original problem about appending to the existing file..
All the methods of the CloudFile will overwrite the existing file. Cloud Files are not for frequent writing, and they indeed impact performance if you keep writing on them frequently, add the lock impact on performance, they will be horrible.
Cloud files are meant to store big bulk of data once and for all, if you want to add another bulk you have the choice of creating another file.
Have all your data with the client till they reach some size and create an algorith to select the file name and upload them all at once.
I am trying to use Hadoop in java with multiple input files. At the moment I have two files, a big one to process and a smaller one that serves as a sort of index.
My problem is that I need to maintain the whole index file unsplitted while the big file is distributed to each mapper. Is there any way provided by the Hadoop API to make such thing?
In case if have not expressed myself correctly, here is a link to a picture that represents what I am trying to achieve: picture
Update:
Following the instructions provided by Santiago, I am now able to insert a file (or the URI, at least) from Amazon's S3 into the distributed cache like this:
job.addCacheFile(new Path("s3://myBucket/input/index.txt").toUri());
However, when the mapper tries to read it a 'file not found' exception occurs, which seems odd to me. I have checked the S3 location and everything seems to be fine. I have used other S3 locations to introduce the input and output file.
Error (note the single slash after the s3:)
FileNotFoundException: s3:/myBucket/input/index.txt (No such file or directory)
The following is the code I use to read the file from the distributed cache:
URI[] cacheFile = output.getCacheFiles();
BufferedReader br = new BufferedReader(new FileReader(cacheFile[0].toString()));
while ((line = br.readLine()) != null) {
//Do stuff
}
I am using Amazon's EMR, S3 and the version 2.4.0 of Hadoop.
As mentioned above, add your index file to the Distributed Cache and then access the same in your mapper. Behind the scenes. Hadoop framework will ensure that the index file will be sent to all the task trackers before any task is executed and will be available for your processing. In this case, data is transferred only once and will be available for all the tasks related your job.
However, instead of add the index file to the Distributed Cache in your mapper code, make your driver code to implement ToolRunner interface and override the run method. This provides the flexibility of passing the index file to Distributed Cache through the command prompt while submitting the job
If you are using ToolRunner, you can add files to the Distributed Cache directly from the command line when you run the job. No need to copy the file to HDFS first. Use the -files option to add files
hadoop jar yourjarname.jar YourDriverClassName -files cachefile1, cachefile2, cachefile3, ...
You can access the files in your Mapper or Reducer code as below:
File f1 = new File("cachefile1");
File f2 = new File("cachefile2");
File f3 = new File("cachefile3");
You could push the index file to the distributed cache, and it will be copied to the nodes before the mapper is executed.
See this SO thread.
Here's what helped me to solve the problem.
Since I am using Amazon's EMR with S3, I have needed to change the syntax a bit, as stated on the following site.
It was necessary to add the name the system was going to use to read the file from the cache, as follows:
job.addCacheFile(new URI("s3://myBucket/input/index.txt" + "#index.txt"));
This way, the program understands that the file introduced into the cache is named just index.txt. I also have needed to change the syntax to read the file from the cache. Instead of reading the entire path stored on the distributed cache, only the filename has to be used, as follows:
URI[] cacheFile = output.getCacheFiles();
BufferedReader br = new BufferedReader(new FileReader(#the filename#));
while ((line = br.readLine()) != null) {
//Do stuff
}
Here is some background about my app:
I am developing an Android app that will display a random quote or verse to the user. For this I am using an SQLite database. The size of the DB would be approximately 5K to 10K records, possibly increasing to upto 1M in later versions as new quotes and verses are added. Thus the user would need to update the DB as and when newer versions are of the app or DB are released.
After reading through some forums online, there seem to be two feasible ways I could provide the DB:
1. Bundle it along with the .APK file of the app, or
2. Upload it to my app's website from where users will have to download it
I want to know which method would be better (if there is yet another approach other than these, please do let me know).
After pondering this problem for some time, I have these thoughts regarding the above approaches:
Approach 1:
Users will obtain the DB along with the app, and won't have to download it separately. Installation would thereby be easier. But, users will have to reinstall the app every time there is a new version of the DB. Also, if the DB is large, it will make the installable too cumbersome.
Approach 2:
Users will have to download the full DB from the website (although I can provide a small, sample version of the DB via Approach 1). But, the installer will be simpler and smaller in size. Also, I would be able to provide future versions of the DB easily for those who might not want newer versions of the app.
Could you please tell me from a technical and an administrative standpoint which approach would be the better one and why?
If there is a third or fourth approach better than either of these, please let me know.
Thank you!
Andruid
I built a similar app for Android which gets periodic updates with data from a government agency. It's fairly easy to build an Android compatible db off the device using perl or similar and download it to the phone from a website; and this works rather well, plus the user gets current data whenever they download the app. It's also supposed to be possible to throw the data onto the sdcard if you want to avoid using primary data storage space, which is a bigger concern for my app which has a ~6Mb database.
In order to make Android happy with the DB, I believe you have to do the following (I build my DB using perl).
$st = $db->prepare( "CREATE TABLE \"android_metadata\" (\"locale\" TEXT DEFAULT 'en_US')");
$st->execute();
$st = $db->prepare( "INSERT INTO \"android_metadata\" VALUES ('en_US')");
$st->execute();
I have an update activity which checks weather updates are available and if so presents an "update now" screen. The download process looks like this and lives in a DatabaseHelperClass.
public void downloadUpdate(final Handler handler, final UpdateActivity updateActivity) {
URL url;
try {
close();
File f = new File(getDatabasePath());
if (f.exists()) {
f.delete();
}
getReadableDatabase();
close();
url = new URL("http://yourserver.com/" + currentDbVersion + ".sqlite");
URLConnection urlconn = url.openConnection();
final int contentLength = urlconn.getContentLength();
Log.i(TAG, String.format("Download size %d", contentLength));
handler.post(new Runnable() {
public void run() {
updateActivity.setProgressMax(contentLength);
}
});
InputStream is = urlconn.getInputStream();
// Open the empty db as the output stream
OutputStream os = new FileOutputStream(f);
// transfer bytes from the inputfile to the outputfile
byte[] buffer = new byte[1024 * 1000];
int written = 0;
int length = 0;
while (written < contentLength) {
length = is.read(buffer);
os.write(buffer, 0, length);
written += length;
final int currentprogress = written;
handler.post(new Runnable() {
public void run() {
Log.i(TAG, String.format("progress %d", currentprogress));
updateActivity.setCurrentProgress(currentprogress);
}
});
}
// Close the streams
os.flush();
os.close();
is.close();
Log.i(TAG, "Download complete");
openDatabase();
} catch (Exception e) {
Log.e(TAG, "bad things", e);
}
handler.post(new Runnable() {
public void run() {
updateActivity.refreshState(true);
}
});
}
Also note that I keep a version number in the filename of the db files, and a pointer to the current one in a text file on the server.
It sounds like your app and your db are tightly bound -- that is, the db is useless without the database and the database is useless without the app, so I'd say go ahead and put them both in the same .apk.
That being said, if you expect the db to change very slowly over time, but the app to change quicker, and you don't want your users to have to download the db with each new app revision, then you might want to unbundle them. To make this work, you can do one of two things:
Install them as separate applications, but make sure they share the same userID using the sharedUserId tag in the AndroidManifest.xml file.
Install them as separate applications, and create a ContentProvider for the database. This way other apps could make use of your database as well (if that is useful).
If you are going to store the db on your website then I would recommend that you just make rpc calls to your webserver and get data that way, so the device will never have to deal with a local database. Using a cache manager to avoid multiple lookups will help as well so pages will not have to lookup data each time a page reloads. Also if you need to update the data you do not have to send out a new app every time. Using HttpClient is pretty straight forward, if you need any examples please let me know