Trying to import a mediawiki.xml into Bitnami mediawiki instance hosted on Google Cloud.
Per Google instructions, moved the xml into the Google Cloud Storage bucket using the browser.
In that browser, I can see the file under what looks like a path - Buckets/2017-11-09-mediawiki_import/mediawiki.xml
Within the Google Cloud SSH client, I am in the Bitnami instance at the path:
me#bitnami-mediawiki-dm-8cff:/opt/bitnami/apps/mediawiki/htdocs/maintenance$ running:
php importDump.php < file name here...
I am not clear what the actual path is to the Google Buckets, or how to find my XML file through the SSH client.
Newb here - so it might be really simple, but I'm not familiar with these tools or command line, any help would be appreciated.
Accessing with the URL
If your file doesn't contain any confidential data you can make it public to everyone knowing the URL and download it.
From the Google Cloud Platform Console, select "Storage" in the menu, find your bucket and your file and then check the box of the column "Share publicly"
There is a URL for each file and folder and bucket hosted in the Google Cloud having the following format:
https://storage.googleapis.com/BUCKETNAME/FOLDER1/FOLDER2/.../FILENAME
Therefore if you have the rights to access the resource you can download the file with any browser or command line tool as curl or wget.
Using $ gsutil
Since your machine has been provided by Google Cloud Compute Engine, and Cloud SDK is already installed, you can use gsutil to access to the bucket.
To copy locally the XML file you stored you can run the following command:
$ gsutil cp gs://2017-11-09-mediawiki_import/mediawiki.xml .
If you get any error, you might need to run gcloud auth login to authenticate yourself and issue the previous command again.
Run the script
Following on of the two solutions, if you succeeded in downloading the file from the bucket, you can run the command:
php importDump.php < mediawiki.xml
Related
This question already has answers here:
How do I download the source code of a google app engine project?
(12 answers)
Closed 1 year ago.
What are the alternatives to download source code from app engine instead of using appcfg.py?
Yes, you are correct, downloading source code using appcfg.py is deprecated. The alternative way is by downloading in Cloud Storage bucket named staging.project-id.appspot.com but the name of the files are encrypted. For you to easy track the files use the following gcloud command:
To view all versions and services in App Engine:
gcloud app versions list
Select the version ID and service name from the output then execute it in the gcloud command below:
gcloud app versions describe VERSIONID -s SERVICENAME
From the output, look for:
deployment:
files:
filesample.txt:
sha1Sum: encryptedname
sourceUrl: https://storage.googleapis.com/staging.project-id.appspot.com/encryptedname
To download click the output of sourceUrl.
There's also a Feature Request for downloading source code for app engine, you can star the public issue tracker feature requests to ensure that you will receive the updates about it. But please note that there is no guarantee that this feature will be implemented.
I did compile a few options to fetch App Engine's code in this repo, but to sum it up:
If you use App Engine Standard:
Try to look in Stackdriver Debugger
The staging bucket contains the files used on deployment time, you can get them from there (although it's not a good idea if there are many files deployed).
In case of App Engine Flex:
SSH into any of the VMs and then download the files directly from there
Fetch the image from GCR that was created when deploying so you can download the files from there
I'm running a local instance of a PHP app-engine project, I have got some buckets setup in GCP specifically for the local dev version, however instead of the data that I write to the bucket appearing online, they are being saved locally into the dev_appserver Datastore. I can see the files in the local admin interface (localhost:8000) under Datastore.
This is an issue as the application I'm developing has a companion app which needs to also access the bucket files.
The
--support_datastore_emulator=[true|false]
flag is documented under
dev_appserver.py -h
But it doesn't seem to have any effect when using =false.
So my question is: How do I stop the dev_appserver from using the local Datastore and make it use the 'real' buckets on the web?
Try setting the --default_gcs_bucket_name flag documented here to establish the default GCS bucket to use:
dev_appserver.py app.yaml --default_gcs_bucket_name gs://BUCKET-NAME
Apologies for the seemingly obvious question, but I figure the answer might help others. I can't for the life of me find documentation on the filepath within the Google App Engine VM (Cloud Shell) where I can find the static files being served from. I need to pull the latest upstream changes from a private github repo.
Note that I navigated elsewhere in the VM and even restarting the session didn't put me in a default project root path within the VM as I expected it to.
There are several issues to address here:
The Cloud Shell is a virtual shell
Google Cloud Shell is an interactive shell environment for Google
Cloud Platform.
The environment where you're working is a container running in a VM in a Google-owned project inside GCP.
You can verify this by checking the metadata server (only available for GCP VMs):
curl -H 'Metadata-Flavor:Google' "http://metadata.google.internal/computeMetadata/v1/?recursive=true&alt=text"
In the metadata provided you'll see how this container is created and configured.
The Cloud Shell is tied to the user, so you'll always access the same environment if you access it with the same credentials, no matter the project. However, if you access with a different user, you'll get a different environment.
You can't access GAE standard instances
GAE is a fully managed environment, and you won't be able to access it. In this way, you won't be able to find the root of the running app engine project.
However, by the way GAE deploys your code, it uses a staging bucket to gather the code before compiling. You can find your staging bucket through the App Engine Admin API. This is usually staging.<PROJECT_ID>.appspot.com, although you can change this configuration. You can get your files from there.
You can access GAE flex apps
However, the deployment in flex gets your files, build a Docker container with them, and then deploys this container inside a VM.
As per the docs, you can connect directly to your container by running:
gcloud app instances ssh [INSTANCE-NAME] --service [SERVICE] --version [VERSION]
docker exec -it gaeapp /bin/bash
Regarding your issue
According what you say in the comments of the question, your issue could come from a myriad of places. From changing the shell you're connecting to, to resetting your shell environment (deleting all the files), to a thousand different possible problems.
The best way to think about it is regard the Cloud Shell as a temporal environment to run commands, but not as a virtual machine.
Knowing that, you could mount a persistent filesystem (GCS through GCSFuse, Cloud Filestore, ...) to persist your work, or simply use Git to have your work always synced on a repo.
GAE Flex has some nice CI integrations, so that's a plus for going the Git route.
I have uploaded several files into the same folder on Google Cloud Storage using the Google Cloud Console. I would now like to move several of the files to a newly created folder in Google Cloud Storage and I cannot see how to do that via the Google Cloud Console. I found instructions to move the files via command prompt instructions on gsutil. However, I am not comfortable with command line interfaces and have not been able to get gsutil to work on my machine.
Is there a way to move files in Google Cloud Storage from one folder to another via the Google Cloud Console?
Update: Google Cloud Shell provides a terminal within the Google Cloud Console site without having to manually create VMs; it comes with gsutil and Google Cloud SDK pre-installed and pre-authenticated.
Prior answer: If you're having issues installing gsutil on your computer, consider the following approach:
Spin up an f1-micro instance with the Google-provided Debian image which will have gsutil preinstalled.
Use the SSH button to connect to it using the browser interface (you can also use gcutil or gcloud commands, if you have those installed and available).
Run gcloud auth login --no-launch-browser within the instance. It will give you a URL to open with your browser. Once you open it, grant the OAuth permissions, and it will display a code. Paste that code back into the command-line window where you ran the command so that it gets the authentication token.
Run the gsutil mv command, as suggested by Travis Hobrla:
gsutil mv gs://bucket/source-object gs://bucket/dest-object
Once you're done with gsutil, delete the instance by clicking on the Delete button at the top of the VM instance detail page. Make sure that the box marked "Delete boot disk when instance is deleted" on the same VM instance page is checked, so that you don't leave an orphaned disk around, which you will be charged for.
You can also browse your persistent disks on the "Disks" tab right below the "VM instances" tab, and delete disks manually there, or make sure there aren't an orphaned disks in the future.
Given the current price of $0.013/hr for an f1-micro instance, this should cost you less than a penny to do this, as you'll only be charged while the instance exists.
There is not currently a way to do this via the Google Cloud Console.
Because folders in Google Cloud Storage are really just placeholder objects in a flat namespace, it's not possible to do an atomic move or rename of a folder, which is why this scenario is more complex than doing a folder move in a local filesystem (with a hierarchical namespace). That's why a more complex tool like gsutil is needed.
Google Cloud Storage now has the functionality to move files from one folder/bucket to another using Cloud Console. To do this, simply select the file(s), click on the 3 vertical dots to get the option of move. Select the target folder/bucket to move the file.
How can I customize the location of the datastore file while working with GAE/J.
The option --datastore_path doesn't seem to work with GAE/J.
And if it is possible, what option do I use in the maven-gae-plugin.
I assume you mean while running the dev app server locally. Try the --generated_dir option with <sdk>/bin/dev_appserver.sh:
--generated_dir=dir Set the directory where generated files are created.
The generated files include the local datastore file.