I created a database model in django. But I would like to load initial data (in the form of a txt file, each row coresponding to a data) to this model. How can I achieve this?
I know how to load data to the mysql directly but not through django.
Django's documentation provides information for your use case, see [Providing initial data for models][1]
You will have to store your data as JSON/YAML/JSON.
You can dump existing data with python manage.py dumpdata
Related
I loaded data (almost 1-billion row data) from hdfs (Hadoop) to Apache Druid. Now, I am trying to export this data set as a CSV to my local. Is there any way to do this in Druid?
There is a download icon on the druid SQL. However, when you click it, it allows you to download the data up to which page you are on. I have soo many pages, so I cannot go through all pages to download all data.
You can POST a SQL query to the Query API and provide a resultFormat in your POST of csv.
https://druid.apache.org/docs/latest/querying/sql.html#responses
I'm interested in having a Table in my Laravel database, where I can read/write from.
However, I don't want it do be affected by php artisan migrate:fresh or php artisan migrate:refresh commands.
I want to use it as a custom storage for a dataset, which will be imported in the database and updated when needed but only be readable by Laravel.
the only thing that is come to my mind on the first place, is to create a route /create/customdbtable/ and insite the method use schema to create the table with all options, and then I think will not be touch by migration – Plamen Penchev
I am building a web app using nodeJS with an angular based frontend and a Firebase/AngularFire2 backend. I have a list of about 80 cities and couple of details about each of them that I need to display with checkboxes for the user.
Should I save them as a json object in a .json file on the server and call it, or just store it in my Real-time Database and query it? Are there any speed/memory benefits to either?
There are two scenarios :
1.Your task is search oriented. You have to query the data and manipulate it. Memory management is key issues for you. You want some complex searching methods on your data. Then go for the database.
2.Your task require whole data at a time. You don't need to worry about memory management. Then directly load the data from file. Obviously this method will save the connection making time with your database. It will work as simple as file streams. [suggested for your case]
I am using a combination of JSF,Servlets and Beans for a web application.The thing is I hit the database for some data and this data is populated on the user page in a chart manner using JFree chart.But the problem is,I dont want to hit the database each time,one query is enough to get all the data.So I want a logic which hits the database only once and gathers the data perhaps in a bean and then populates it according to the user request.Can this be done in my same application,should I be using Java Script to do this.I have the logic to populate the data but I dont have any idea where to put the Data Access Object method so that it executes only once.
You can use Spring cache-abstraction, and cache the data using EHcache
I have about 4000 records that I need to upload to Datastore.
They are currently in CSV format. I'd appreciate if someone would
point me to or explain how to upload data in bulk to GAE.
You can use the bulkloader.py tool:
The bulkloader.py tool included with
the Python SDK can upload data to your
application's datastore. With just a
little bit of set-up, you can create
new datastore entities from CSV files.
I don't have the perfect solution, but I suggest you have a go with the App Engine Console. App Engine Console is a free plugin that lets you run an interactive Python interpreter in your production environment. It's helpful for one-off data manipulation (such as initial data imports) for several reasons:
It's the good old read-eval-print interpreter. You can do things one at a time instead of having to write the perfect import code all at once and running it in batch.
You have interactive access to your own data model, so you can read/update/delete objects from the data store.
You have interactive access to the URL Fetch API, so you can pull data down piece by piece.
I suggest something like the following:
Get your data model working in your development environment
Split your CSV records into chunks of under 1,000. Publish them somewhere like Amazon S3 or any other URL.
Install App Engine Console in your project and push it up to production
Log in to the console. (Only admins can use the console so you should be safe. You can even configure it to return HTTP 404 to "cloak" from unauthorized users.)
For each chunk of your CSV:
Use URLFetch to pull down a chunk of data
Use the built-in csv module to chop up your data until you have a list of useful data structures (most likely a list of lists or something like that)
Write a for loop, iterating through each each data structure in the list:
Create a data object with all correct properties
put() it into the data store
You should find that after one iteration through #5, then you can either copy and paste, or else write simple functions to speed up your import task. Also, with fetching and processing your data in steps 5.1 and 5.2, you can take your time until you are sure that you have it perfect.
(Note, App Engine Console currently works best with Firefox.)
By using remote API and operations on multiple entities. I will show an example on NDB using python, where our Test.csv contains the following values separated with semicolon:
1;2;3;4
5;6;7;8
First we need to import modules:
import csv
from TestData import TestData
from google.appengine.ext import ndb
from google.appengine.ext.remote_api import remote_api_stub
Then we need to create remote api stub:
remote_api_stub.ConfigureRemoteApi(None, '/_ah/remote_api', auth_func, 'your-app-id.appspot.com')
For more information on using remote api have a look at this answer.
Then comes the main code, which basically does the following things:
Opens the Test.csv file.
Sets the delimiter. We are using semicolon.
Then you have two different options to create a list of entities:
Using map reduce functions.
Using list comprehension.
In the end you batch put the whole list of entities.
Main code:
# Open csv file for reading.
with open('Test.csv', 'rb') as file:
# Set delimiter.
reader = csv.reader(file, delimiter=';')
# Reduce 2D list into 1D list and then map every element into entity.
test_data_list = map(lambda number: TestData(number=int(number)),
reduce(lambda list, row: list+row, reader)
)
# Or you can use list comprehension.
test_data_list = [TestData(number=int(number)) for row in reader for number in row]
# Batch put whole list into HRD.
ndb.put_multi(test_data_list)
The put_multi operation also takes care of making sure to batch appropriate number of entities in a single HTTP POST request.
Have a look at this documentation for more information:
CSV File Reading and Writing
Using the Remote API in a Local Client
Operations on Multiple Keys or Entities
NDB functions
the later version of app engine sdk, one can upload using the appcfg.py
see appcfg.py