is there anyone knows the lifetime of files on the colab virtual machine?
for example, in a colab notebook, I save the data to a csv file as:
data.to_csv('data.csv')
then how long will the data.csv exist?
This is the scenario:
I want to maintain and update over 3000 small datasets everyday, but it seems that the interaction between colab and google drive by using pydrive is pretty slow(as I need to check every dataset everyday), so if the lifetime of files on the virtual machine is long enough, I can update the files on virtual machine everyday(which would be much faster) then synchronize them to google drive several days a time rather than everyday.
VMs are discarded after a period of inactivity, so your best bet is to save files to Drive that you'd like to keep as generated.
With pydrive, this is possible, but a bit cumbersome. An easier method is to use a FUSE interface to Drive so that you can automatically sync files as they are saved normally.
For an example, see:
https://colab.research.google.com/drive/1srw_HFWQ2SMgmWIawucXfusGzrj1_U0q
Related
I'm currently not having the option to check this question since I do not have 2 computers running Anylogic.
My question is if you share your anylogic .Alp file with somebody, is he able to open the model.
Or should you also need to share the things in your model file when you save it(3d map, databasemap and pictures)
Or do you even need more since when you save an anylogic file you also need to name the Java package.
I'm asking this because my computer is running on his end and I need to know what I've to put in the cloud when my computer will crash and will not be able to use it.
Im currently have put the map from the model map in the cloud and the database excel files. (But no file about the Java package cause I could not find it)
You only need the .alp file itself and all items that are in the model folder where the .alp sits. This can include:
images
database subfolder
lib subfolder (holding any required jar files)
logs subfolder
outputs subfolder
So if you simply zip the entire model folder and send it over, the other person unzips it AND IF he/she uses the same AnyLogic version, all will work.
If it is an older version of AnyLogic on the other end, it may not work (but also it may, it depends on the version difference).
Also, if you created the model with the Professional version and you used some elements that are not accessible in the free PLE version and if the other person only uses the PLE, it will also not work.
If you really just need the other person to run the model, consider compiling it or using the AnyLogic cloud.
My problem is the following:
I usually backup files (e.g. pictures) on external harddisc drives and the store them away in safe places. In the meantime also on NAS. But I don't want to have them connected and online all the time, for power and security reasons.
If I'm now looking for an old file (e.g. a special jpg from the holiday in April 2004) I would have to connect a few discs and search them for the needed file.
To overcome this problem I usually create a recursive dir-dump into a textfile for the whole disc after backup.
This way I can search the filename in the text-file.
But there still is a problem if I don't exactly know the file name that I am looking for. I know the Year and month and maybe the camera I was using then, but there must be hundreds of files in this month.
Therefore I would like to create a "dummy"-backup-filesystem with all the filesnames on the harddisc but without the actual data behind it. This way I could click through the folders and see the foldernames and filenames and easily find the respective file.
The question is: How do I create such a filesystem copy with the complete folderstructures but only the filenames and not the data?
I'm working on Linux, Opensuse, but I guess this is not a linux specific question.
In the meantime I found the solution I was looking for:
Virtual Volume View:
http://vvvapp.sourceforge.net/
Works with Linux, MacOS and Windows!
This question has been asked around several time. Many programs like Dropbox make use of some form of file system api interaction to instantaneously keep track of changes that take place within a monitored folder.
As far as my understanding goes, however, this requires some daemon to be online at all times to wait for callbacks from the file system api. However, I can shut Dropbox down, update files and folders, and when I launch it again it still gets to know what the changes that I did to my folder were. How is this possible? Does it exhaustively search the whole tree in search for updates?
Short answer is YES.
Let's use Google Drive as an example, since its local database is not encrypted, and it's easy to see what's going on.
Basically it keeps a snapshot of the Google Drive folder.
You can browse the snapshot.db (typically under %USER%\AppData\Local\Google\Drive\user_default) using DB browser for SQLite.
Here's a sample from my computer:
You see that it tracks (among other stuff):
Last write time (looks like Unix time).
checksum.
Size - in bytes.
Whenever Google Drive starts up, it queries all the files and folders that are under your "Google Drive" folder (you can see that using Procmon)
Note that changes can also sync down from the server
There's also Change Journals, but I don't think that Dropbox or GDrive use it:
To avoid these disadvantages, the NTFS file system maintains an update sequence number (USN) change journal. When any change is made to a file or directory in a volume, the USN change journal for that volume is updated with a description of the change and the name of the file or directory.
I'm looking for a solution to transfer files from one computer to another without any human interaction. I have a build server that needs to get the nightly build to another testing computer that evaluates the software and saves the results in a text file. When that is done, I need to get the text file back to the build server for the emailing and posting of the results. I've looked around a bit and found that it's not really possible to do this with forms for security reasons but this really isn't my area of expertise. I can't use a network share location because the network drives are not always in sync.
Some other ideas I had were running a ftp upload with the command line, having some kind of listen socket for both machines, and putting the file in a "to download" folder visible on a web server and notifying the other machine of what the filename is.
Are any of these better than the others or even possible?
Using FTP & PHP for your condition is very possible. Back then I created a page that has to scan through a network and download some images using FTP (I used VSFTPD) regularly.
So perhaps all you need to do is setup your FTP server, then create the php script to download your file, and run the script regularly using cron job at when your file is created + 5 or 10 minutes.
In our Silverlight business application, we need to cache very large files (100's MBs) into isolated storage. We distribute the files separately to be downloaded by users and then they can import those files into Isolated Storage through the application.
However, the Isolated Storage API seem to be very slow and it takes an hour to import about 500MB of data.
Given, that we're in a corporate environment where users trust us, I would like users to be able to copy the files directly into the physical location on their file system where Silverlight store files when using the API.
The location varies per OS, but that's ok. The problem however, is that Silverlight seems to store files in a somewhat cryptic way. If I go to my AppData\LocalLow\Microsoft\Silverlight\is, I can see some weirdly named folder that look like long Guid.
My question: is it possible to copy files directly in there, or is it going to upset Silverlight ?
From what I've been testing it will make stuff fail/act weird. We had some stuff we had to clear and even though we did delete the files to test how it worked the usedspace didn't drop. So there is some sort of register of which files are in IS and how big they are.
I think it would be paramount you find out why IS is so slow. Can you confirm its like that on all clients? Test some others. This should be brought up to Microsoft if it is the case. Possibly you can change your serailization schema and save smaller files? I would not advise trying to figure out Microsoft's temporary and volatile IS storage location.