I've been using Google App Engine to develop a prototype inventory application. I've recently switched from using the old DB datastore library to using the new NDB library, and the features are cool, but in making a change to one of my entities, suddenly NDB is inserting a property called "metadata" in my entity. Here's the entity description (the change I made was to add the "inv_posid" and "inv_lastchange" properties, nothing major):
class Inventory(ndb.Model):
inv_product = ndb.KeyProperty(kind = Product)
inv_prdcr_name = ndb.StringProperty(default="")
inv_product_type = ndb.StringProperty(default="")
inv_product_name = ndb.StringProperty(default="")
inv_product_year = ndb.IntegerProperty(default=0)
inv_count = ndb.IntegerProperty(default=0)
inv_price = ndb.IntegerProperty(default=0)
inv_glass_price = ndb.IntegerProperty(default=0)
inv_bin = ndb.StringProperty(default="")
inv_posid = ndb.StringProperty(default="")
inv_lastchange = ndb.FloatProperty(default=0.0)
With adding the new properties, I intended to change my query to use the "inv_lastchange" as a filter, and since NDB never includes entities in results that don't have the appropriate property included, I wanted to run a quick sweep through my datastore, to add the properties to all the entities appropriately. So, here's what I did:
...
#ndb.tasklet
def fixInventory(invitem):
invitem.inv_posid = ""
invitem.inv_lastchange = 0.0
invkey = yield invitem.put_async()
inventory = Inventory.query()
output = inventory.map(fixInventory)
I thought it would be neat to play around with tasklets and see how the asynchronous calls worked. However, after doing that, when I went to look in the datastore viewer (on my local datastore), I saw this new "metadata" property, which I just assumed was something NDB needed to have, so I didn't think anything of it.
Until, the next time I tried to update one of my inventory items, I got this error:
File "/Programming/VirtualCellar/server/virtsom.py", line 2118, in get
inventory.put()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/model.py", line 3432, in _put
return self._put_async(**ctx_options).get_result()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/tasklets.py", line 326, in get_result
self.check_success()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/tasklets.py", line 369, in _help_tasklet_along
value = gen.throw(exc.__class__, exc, tb)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/context.py", line 810, in put
key = yield self._put_batcher.add(entity, options)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/tasklets.py", line 369, in _help_tasklet_along
value = gen.throw(exc.__class__, exc, tb)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/context.py", line 343, in _put_tasklet
keys = yield self._conn.async_put(options, datastore_entities)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/tasklets.py", line 455, in _on_rpc_completion
result = rpc.get_result()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
return self.__get_result_hook(self)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1882, in __put_hook
self.check_rpc_success(rpc)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1373, in check_rpc_success
raise _ToDatastoreError(err)
BadRequestError: cannot store entity with reserved property name '__metadata__'
So what's happening? Is there something that I'm forgetting to do here? It feels like the "metadata" property is supposed to be hidden or protected in some way, but it has been added just like a regular property, which is now preventing any other saves to be made to the entity. Anybody run across this before?
The __metadata__ property is added to entities when they are saved to the datastore and removed when they are read from the datastore. This is done in the _ToStorageEntity and _FromStorageEntity functions in google/appengine/datastore/datastore_stub_util.py.
Presumably internal failure has corrupted your entities somehow.
You might be able to recover by removing the __metadata__ property from your entity instances' _properties dict and saving.
For example:
for inv in Inventory.query():
del inv._properties['__metadata__']
inv.put()
(Maybe backup your local datastore file before trying this, as a precaution against accidents).
So, based on what snakecharmerb said above, I started looking at why my datastore viewer was also showing me the __metadata__ property, and it turned out that I had previously downloaded an older version of the google app engine SDK, and for some reason, my environment variables were still pointing to that old version, even though I had installed the latest version of the SDK a couple of times. I wiped all traces of the google app engine SDK off my machine, and re-installed the SDK from scratch, and voila! The __metadata__ properties went away, in both the datastore viewer, and my own code! My current hypothesis is that the older version of the google/appengine/datastore/datastore_stub_util.py file wasn't handling the __metadata__ property in the same way.
Thanks so much for the help!
Related
I want to know whether a particular database file exists in my codenameone app.
Here's the code I have:
if (CN.existsInFileSystem(Display.getInstance()
.getDatabasePath("my.db")))
overwriteDb = false;
It seems that the getDatabasePath method will create the database if it doesn't already exist, at least on the windows/eclipse simulator. (I haven't tried on any devices).
The javadocs say that getDatabasePath will return null when the database doesn't exist, however it returns the path of the newly created db file.
Is there a way to get the path without actually creating a zero byte file?
No it doesn't create the database, I just verified it in the code. The JavaDoc is incorrect and I fixed it for the next update. The code you have should work just fine.
This is the Android implementation of the method:
public String getDatabasePath(String databaseName) {
if (databaseName.startsWith("file://")) {
return databaseName;
}
File db = new File(getContext().getApplicationInfo().dataDir + "/databases/" + databaseName);
return db.getAbsolutePath();
}
You will notice null is never returned and nothing is created. The fixed docs look like this:
Returns the file path of the Database if support for database exists on the platform.
#param databaseName the name of the database with out / or path elements e.g. mydatabase.db
#return the file path of the database or null if database isn't supported
I have the two following methods and I am using them to store a special value locally and be able to access it on application restart:
(Store value locally:)
private void SaveSet(string key, string value)
{
ISharedPreferences prefs = PreferenceManager.GetDefaultSharedPreferences(this);
ISharedPreferencesEditor prefEditor = prefs.Edit();
prefEditor.PutString(key, value);
// editor.Commit(); // applies changes synchronously on older APIs
prefEditor.Apply(); // applies changes asynchronously on newer APIs
}
(Read it again:)
private string RetrieveSet(string key)
{
ISharedPreferences prefs = PreferenceManager.GetDefaultSharedPreferences(this);
return prefs.GetString(key, null);
}
This works perfectly. Now is it possible to access and edit this Shared Preferences externally? Unfortunately, I cannot find any file when searching in folder
Phone\Android\data\com.<company_name>.<application_name>\files
nor anywhere else. I want / try to edit this value from my computer, after connecting the phone to it. Is this possible?
Alternatively: Can anyone maybe show me how to create a new file in the given path above, write/read it programmatically and how it stays there, even if application is closed / started again? So I can then edit this file with my computer anyhow?
I tried it with the following code, but unfortunately it doesn't work / no file is created or at least i cannot see it in the given path above:
//"This code snippet is one example of writing an integer to a UTF-8 text file to the internal storage directory of an application:"
public void SaveValueIntoNewFile(int value)
{
var backingFile = Path.Combine(System.Environment.GetFolderPath(System.Environment.SpecialFolder.Personal), "newFile.txt");
using (var writer = System.IO.File.CreateText(backingFile))
{
writer.WriteLine(value.ToString());
}
}
Would be very happy about every answer, thanks in advance and best regards
What you're looking for is where Android stores the Shared Preference file for applications that make use of it's default PreferenceManager.
I'd refer to this SO post which answers your question pretty well
SharedPreferences are stored in an xml file in the app data folder,
i.e.
/data/data/YOUR_PACKAGE_NAME/shared_prefs/YOUR_PREFS_NAME.xml
or the default preferences at:
/data/data/YOUR_PACKAGE_NAME/shared_prefs/YOUR_PACKAGE_NAME_preferences.xml
SharedPreferences added during runtime are not stored in the Eclipse
project.
Note: Accessing /data/data/ requires superuser
privileges
A simple method is to use Android Device Monotor,you can open it by clicking Tools--> android-->Android Device Monotor...
For example:
The path in my device is as follows:
/data/data/YOUR_PACKAGE_NAME/shared_prefs/YOUR_PACKAGE_NAME_preferences.xml
And we notice three buttons in the upper right corner of the picture.
The first one is used toPull a file from the device,the second one is used to Push a file onto the device,and the last one is used to delete the preferences.xml file.
So we can pull the preferences.xml file from current device to our computer and edit it as we want, and then push the updated preferences.xml to the folder again.Then we will get the value of preferences.xml file .
I want to use external property file for database configuration in production environment. i have tried some of solution from blogs and stack overflow but its work only for development environment.
grailsVersion=3.3.2
First create properties file in src/main/resources (if resources dir does not exist, create it).
then remove configuration from application.yml (if won't then it will override).
in Application.groovy load the file with this :
def url = getClass().classLoader.getResource("myconfig.properties")
def confFile = new File(url.toURI())
Properties properties = new Properties()
confFile.withInputStream {
properties.load(it)
}
environment.propertySources.addFirst(new PropertiesPropertySource("local.config.location", properties))
I want to write entries to a log file stored in Azure file storage. I currently have this:
var log = "My log entry";
var client = _storageAccount.CreateCloudFileClient();
var share = client.GetShareReference(Config.LogShare);
share.CreateIfNotExists();
var root = share.GetRootDirectoryReference();
var logfile = root.GetFileReference("log.txt");
if (!logfile.Exists()) logfile.Create(0);
// What goes here to append to the file...?
I can see plenty of examples of how to do this with Blobs, or how to upload an entire file, but how do I just append to an existing file?
I have tried this:
var buffer = Encoding.GetEncoding("UTF-8").GetBytes(log.ToCharArray());
using (var fileStream = logfile.OpenWrite(0)) {
fileStream.Write(buffer, (int)logfile.Properties.Length, buffer.Length);
}
But then I get this error:
The remote server returned an error: (416) The range specified is invalid for the current size of the resource..
I managed to work this out myself. You just need to increase the size of the file by the number of new bytes you want to write to it, and then write the new data to that new empty space at the end of the file, like this:
var client = _storageAccount.CreateCloudFileClient();
var share = client.GetShareReference(Config.LogShare);
share.CreateIfNotExists();
var root = share.GetRootDirectoryReference();
var logfile = root.GetFileReference("log.txt");
if (!logfile.Exists()) logfile.Create(0);
var buffer = Encoding.UTF8.GetBytes($"{log}\r\n");
logfile.Resize(logfile.Properties.Length + buffer.Length);
using (var fileStream = logfile.OpenWrite(null)) {
fileStream.Seek(buffer.Length * -1, SeekOrigin.End);
fileStream.Write(buffer, 0, buffer.Length);
}
You can do this with blobs https://blogs.msdn.microsoft.com/windowsazurestorage/2015/04/13/introducing-azure-storage-append-blob/
Shame it doesn't work with files too
Azure file storage REST API doesn't support appending to an existing file. To achieve this, please mount the file share to your machine as a drive, and append to the file just like simple local files.
Actually, I don't think you really need appending functionality per your code above. You can specify the file size in CloudFile.OpenWrite() / CloudFile.Create(), or try CloudFile.UploadFromStream() instead of CloudFile.OpenWrite().
This error could also be due to multi-threaded access.
I bet if you tried to lock the file before you access it, you will not face this problem.
There are many ways to update the file.
Since you already managed to get the share, the root, the folder and the file.. Here is a portion of my code that worked for me.
if (!fileLock.IsWriteLockHeld) fileLock.EnterWriteLock();
try
{
using (var stream = new MemoryStream(content, false))
{
file.UploadFromStream(stream, null, options);
}
}
catch (Exception ex)
{
File.AppendAllText(FileName, ex.ToString());
}
finally
{
if (fileLock.IsWriteLockHeld)
fileLock.ExitWriteLock();
}
Where fileLock is declared as:
protected ReaderWriterLockSlim fileLock = new ReaderWriterLockSlim();
Having said that, I am not saying that this is the best way ever to do it.
The two things I would like you to keep in mind :
1-Lock the resource that is likely to be accessed by more than one thread (That is so common in AZURE)
2- Get familiar with asynchronous methods that Azure provides.. use them when they suit well.
Coming back to your original problem about appending to the existing file..
All the methods of the CloudFile will overwrite the existing file. Cloud Files are not for frequent writing, and they indeed impact performance if you keep writing on them frequently, add the lock impact on performance, they will be horrible.
Cloud files are meant to store big bulk of data once and for all, if you want to add another bulk you have the choice of creating another file.
Have all your data with the client till they reach some size and create an algorith to select the file name and upload them all at once.
I am trying to use Hadoop in java with multiple input files. At the moment I have two files, a big one to process and a smaller one that serves as a sort of index.
My problem is that I need to maintain the whole index file unsplitted while the big file is distributed to each mapper. Is there any way provided by the Hadoop API to make such thing?
In case if have not expressed myself correctly, here is a link to a picture that represents what I am trying to achieve: picture
Update:
Following the instructions provided by Santiago, I am now able to insert a file (or the URI, at least) from Amazon's S3 into the distributed cache like this:
job.addCacheFile(new Path("s3://myBucket/input/index.txt").toUri());
However, when the mapper tries to read it a 'file not found' exception occurs, which seems odd to me. I have checked the S3 location and everything seems to be fine. I have used other S3 locations to introduce the input and output file.
Error (note the single slash after the s3:)
FileNotFoundException: s3:/myBucket/input/index.txt (No such file or directory)
The following is the code I use to read the file from the distributed cache:
URI[] cacheFile = output.getCacheFiles();
BufferedReader br = new BufferedReader(new FileReader(cacheFile[0].toString()));
while ((line = br.readLine()) != null) {
//Do stuff
}
I am using Amazon's EMR, S3 and the version 2.4.0 of Hadoop.
As mentioned above, add your index file to the Distributed Cache and then access the same in your mapper. Behind the scenes. Hadoop framework will ensure that the index file will be sent to all the task trackers before any task is executed and will be available for your processing. In this case, data is transferred only once and will be available for all the tasks related your job.
However, instead of add the index file to the Distributed Cache in your mapper code, make your driver code to implement ToolRunner interface and override the run method. This provides the flexibility of passing the index file to Distributed Cache through the command prompt while submitting the job
If you are using ToolRunner, you can add files to the Distributed Cache directly from the command line when you run the job. No need to copy the file to HDFS first. Use the -files option to add files
hadoop jar yourjarname.jar YourDriverClassName -files cachefile1, cachefile2, cachefile3, ...
You can access the files in your Mapper or Reducer code as below:
File f1 = new File("cachefile1");
File f2 = new File("cachefile2");
File f3 = new File("cachefile3");
You could push the index file to the distributed cache, and it will be copied to the nodes before the mapper is executed.
See this SO thread.
Here's what helped me to solve the problem.
Since I am using Amazon's EMR with S3, I have needed to change the syntax a bit, as stated on the following site.
It was necessary to add the name the system was going to use to read the file from the cache, as follows:
job.addCacheFile(new URI("s3://myBucket/input/index.txt" + "#index.txt"));
This way, the program understands that the file introduced into the cache is named just index.txt. I also have needed to change the syntax to read the file from the cache. Instead of reading the entire path stored on the distributed cache, only the filename has to be used, as follows:
URI[] cacheFile = output.getCacheFiles();
BufferedReader br = new BufferedReader(new FileReader(#the filename#));
while ((line = br.readLine()) != null) {
//Do stuff
}