What is the correct way to load prebuilt SQlite databases with or without PouchDB in a React App - reactjs

Currently I'm writing a React App and struggling with a simple reading from a SQlite database.
Edit because of unclear question:
***The goal is to read from the database without any backend, because it needs to read from the database even when it is offline.
***I'm aiming for a ONE TIME file conversion, then just pouchdb queries offline. But I don't want to do it manually because there are around 6k+ registries.
***Or SQL queries from the browser without any APIs, but I need to support Internet Explorer, so WebSQL is not an option. I've tried sqlite3 library, but I can't make it work with Create React App.
The solution I tried was to use PouchDB for reading the file, but I'm coming to a conclusion that it is NOT possible to PRELOAD a SQlite file with PouchDB without using cordova (I'm not comfortable with it, I don't want any servers running), or even with some kind of adapter.
So is this the right way of doing things?
Is there any way that I would not loose my .db data, and have to convert it all of it manually?
Should I forget about supporting this features on IE?
Thanks :)

Try this:
sqlite3 example "DROP TABLE IF EXISTS some_table;";
sqlite3 example "CREATE TABLE IF NOT EXISTS some_table (id INTEGER PRIMARY KEY AUTOINCREMENT, anattr VARCHAR, anotherattr VARCHAR);";
sqlite3 example "INSERT INTO some_table VALUES (NULL, '1stAttr', 'AttrA');";
sqlite3 example "INSERT INTO some_table VALUES (NULL, '2ndAttr', 'AttrB');";
## Create three JSON fragment files
sqlite3 example ".output result_prefix.json" "SELECT '{ \"docs\": ['";
sqlite3 example ".output rslt.json" "SELECT '{ \"_id\": \"someTable_' || SUBSTR(\"000000000\" || id, LENGTH(\"000000000\" || id) - 8, 9) || '\", \"anattr\": \"' || anattr || '\", \"anotherattr\": \"' || anotherattr || '\" },' FROM some_table;";
sqlite3 example ".output result_suffix.json" "SELECT '] }'";
## strip trailing comma of last record
sed -i '$ s/.$//' rslt.json;
## concatenate to a single file
cat result_prefix.json rslt.json result_suffix.json > result.json;
cat result.json;
You should be able simply to paste the above lines onto the (unix) command line, seeing output:
{ "docs": [
{ "_id": "someTable_000000001", "anattr": "1stAttr", "anotherattr": "AttrA" },
{ "_id": "someTable_000000002", "anattr": "2ndAttr", "anotherattr": "AttrB" }
] }
If you have jq installed you can do instead ...
cat result.json | jq .
... obtaining:
{
"docs": [
{
"_id": "someTable_000000001",
"anattr": "1stAttr",
"anotherattr": "AttrA"
},
{
"_id": "someTable_000000002",
"anattr": "2ndAttr",
"anotherattr": "AttrB"
}
]
}
You'll find an example of how quickly to initialize PouchDB from JSON files in part 2 of the blog post Prebuilt databases with PouchDB.
So, if you have a CouchDB server available you can do the following;
export COUCH_DB=example;
export COUCH_URL= *** specify yours here ***;
export FILE=result.json;
## Drop database
curl -X DELETE ${COUCH_URL}/${COUCH_DB};
## Create database
curl -X PUT ${COUCH_URL}/${COUCH_DB};
## Load database from JSON file
curl -H "Content-type: application/json" -X POST "${COUCH_URL}/${COUCH_DB}/_bulk_docs" -d #${FILE};
## Extract database with meta data to PouchDB initialization file
pouchdb-dump ${COUCH_URL}/${COUCH_DB} > example.json
## Inspect PouchDB initialization file
cat example.json | jq .
Obviously you'll need some adaptations, but the above should give you no problems.

Since Couch/Pouch-DB are document-oriented DBs all records aka docs there are just JSON aka JS-objects. In my RN app when I met similar task I just put all docs I wanted to be "prepopulated" in PouchDB in an array of JS-objects, import it as module in my app and then write them during app init to PDB as necessarry docs. That's all prepopulation. How to export your SQL DB records to JSON - you decide, surely it depends on source DB structure and data logic you want to be in PDB.

Related

Gcp appengine app does not correctly connect to Cloud sql database

I have connection issue between my spring boot/kotlin app and database. I tried following guides what google has to offer and some Stackoverflow posts. I added client/admin sql permisions to users.
After following google guide https://github.com/spring-cloud/spring-cloud-gcp/tree/master/spring-cloud-gcp-samples/spring-cloud-gcp-sql-postgres-sample on this link and recreating everything as it is. I came to problem: deployed App does not connect to actual database, instead it uses data.sql and schema.sql objects from resources, also it does not do changes to actual database, but somewhere else, for example I am using this command to getdata:
#GetMapping("/getdata")
fun getTuples(): List<String>? {
return jdbcTemplate!!.queryForList("SELECT * FROM users").stream()
.map { m: Map<String?, Any?> -> m.values.toString() }
.collect(Collectors.toList())
}
result I get is the one provided in example:
0 "[luisao#example.com, Anderson, Silva]"
1 "[jonas#example.com, Jonas, Goncalves]"
2 "[fejsa#example.com, Ljubomir, Fejsa]"
the actual users table in Cloud database looks like this:
postgres=> select * from users;
email | first_name | last_name
--------------------+------------+-----------
luisao#example.com | Anderson | Silva
jonas#example.com | Jonas | Goncalves
fejsa#example.com | Ljubomir | Fejsa
kkskldk | sjndjfdf | skdskd
(4 rows)
first 3 rows are same just because I added them manually. App does not see DDL/DML changes in database, only those I do in data.sql or schema.sql. So the question is where could be a problem?
in this documentation https://cloud.spring.io/spring-cloud-gcp/multi/multi__spring_jdbc.html says that this dependency should do most of the stuff automaticly but somehow something goes wrong.
Could it be that spring boot/hikari creates virtual enviroment in appengine which blocks database connection and you connect and see only Mock data from data.sql files? Or problem lies in configuration?
My application.properties file:
server.error.include-message=always
spring.cloud.gcp.sql.database-name=[database name]
spring.cloud.gcp.sql.instance-connection-name=[Instance name]
spring.datasource.continue-on-error=true
spring.datasource.initialization-mode=always
spring.datasource.username=postgres
spring.datasource.password=[password]
spring.cloud.gcp.project-id=[project id]

How Do I Capture and Download Snowflake Query Results?

I'm using Snowflake on a Windows PC.
For example: https://<my_org>.snowflakecomputing.com/console#/internal/worksheet
I have a bunch of queries, the collective output of which I want to capture and load into a file.
Apart from running the queries one-at-a-time and using copy-and-paste to populate the file, is there a way I can run all the queries at once and have the output logged to a file on my PC?
There are many ways to achieve the high level outcome that you are seeking, but you have not provided enough context to know which would be best-suited to your situation. For example, by mentioning https://<my_org>.snowflakecomputing.com/console#/internal/worksheet, it is clear that you are currently planning to execute the series of queries through the Snowflake web UI. Is using the web UI a strict requirement of your use-case?
If not, I would recommend that you consider using a Python script (along with the Snowflake Connector for Python) for a task like this. One strategy would be to have the Python script serially process each query as follows:
Execute the query
Export the result set (as a CSV file) to a stage location in cloud storage via two of Snowflake's powerful features:
RESULT_SCAN() function
COPY INTO <location> command to EXPORT data (which is the "opposite" of the COPY INTO <table> command used to IMPORT data)
Download the CSV file to your local host via Snowflake's GET command
Here is a sample of what such a Python script might look like...
import snowflake.connector
query_array = [r"""
SELECT ...
FROM ...
WHERE ...
""",r"""
SELECT ...
FROM ...
WHERE ...
"""
]
conn = snowflake.connector.connect(
account = ...
,user = ...
,password = ...
,role = ...
,warehouse = ...
)
file_number = 0;
for query in query_array:
file_number += 1
file_name = f"{file_prefix}_{file_number}.csv.gz"
rs_query = conn.cursor(snowflake.connector.DictCursor).execute(query)
query_id = rs_query.sfqid # Retrieve query ID for query execution
sql_copy_into = f"""
COPY INTO #MY_STAGE/{file_name}
FROM (SELECT * FROM TABLE(RESULT_SCAN('{query_id}')))
DETAILED_OUTPUT = TRUE
HEADER = TRUE
SINGLE = TRUE
OVERWRITE = TRUE
"""
rs_copy_into = conn.cursor(snowflake.connector.DictCursor).execute(sql_copy_into)
for row_copy_into in rs_copy_into:
file_name_in_stage = row_copy_into["FILE_NAME"]
sql_get_to_local = f"""
GET #MY_STAGE/{file_name_in_stage} file://.
"""
rs_get_to_local = conn.cursor(snowflake.connector.DictCursor).execute(sql_get_to_local)
Note: I have chosen (for performance reasons) to export and transfer the files as zipped (gz) files; you could skip this by passing the COMPRESSION=NONE option in the COPY INTO <location> command.
Also, if your result sets are much smaller, then you could use an entirely different strategy and simply have Python pull and write the results of each query directly to a local file. I assumed that your result sets might be larger, hence the export + download option I have employed here.
You can use the SnowSQL client for this. See https://docs.snowflake.com/en/user-guide/snowsql.html
Once you get it configured, then you can make a batch file or similar that calls SnowSQL to run each of your queries and write the output to a file. Something like:
#echo off
>output.txt (
snowsql -q "select blah"
snowsql -q "select blah"
...
snowsql -q "select blah"
)

How to indicate the database in SparkSQL over Hive in Spark 1.3

I have a simple Scala code that retrieves data from the Hive database and creates an RDD out of the result set. It works fine with HiveContext. The code is similar to this:
val hc = new HiveContext(sc)
val mySql = "select PRODUCT_CODE, DATA_UNIT from account"
hc.sql("use myDatabase")
val rdd = hc.sql(mySql).rdd
The version of Spark that I'm using is 1.3. The problem is that the default setting for hive.execution.engine is 'mr' that makes Hive to use MapReduce which is slow. Unfortunately I can't force it to use "spark".
I tried to use SQLContext by replacing hc = new SQLContext(sc) to see if performance will improve. With this change the line
hc.sql("use myDatabase")
is throwing the following exception:
Exception in thread "main" java.lang.RuntimeException: [1.1] failure: ``insert'' expected but identifier use found
use myDatabase
^
The Spark 1.3 documentation says that SparkSQL can work with Hive tables. My question is how to indicate that I want to use a certain database instead of the default one.
use database
is supported in later Spark versions
https://docs.databricks.com/spark/latest/spark-sql/language-manual/use-database.html
You need to put the statement in two separate spark.sql calls like this:
spark.sql("use mydb")
spark.sql("select * from mytab_in_mydb").show
Go back to creating the HiveContext. The hive context gives you the ability to create a dataframe using Hive's metastore. Spark only uses the metastore from hive, and doesn't use hive as a processing engine to retrieve the data. So when you create the df using your sql query, its really just asking hive's metastore "Where is the data, and whats the format of the data"
Spark takes that information, and will run process against the underlying data on the HDFS. So Spark is executing the query, not hive.
When you create the sqlContext, its removing the link between Spark and the Hive metastore, so the error is saying it doesn't understand what you want to do.
I have not been able to implement the use databale command, but here is a workaround to use the desired database:
spark-shell --queue QUEUENAME;
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
val res2 = sqlContext.sql("select count(1) from DB_NAME.TABLE_NAME")
res2.collect()

Neo4j Failed to load csv from local disk (windows 7)

I am new to Neo4j and have been trying to load a CSV from my local disk, but without a success.
LOAD CSV WITH HEADERS FROM "file:C:/Neo4j/Persons.csv" AS csvLine
CREATE (p:Person { id: toInt(csvLine.id), name: csvLine.name })
I am getting the following response and error
Couldn't load the external resource at: file:C:/Neo4j/Persons.csv
Neo.TransientError.Statement.ExternalResourceFailure
Can you verify that the file is in the C:/Neo4j/ directory?
Could not be the perfect solutiuon but might be work for you.
You should try this one:
USING PERIODIC COMMIT
LOAD CSV WITH HEADERS
FROM "file:C:/Neo4j/Persons.csv" AS csvLine
CREATE (:PERSONS {id:csvLine.id, name:csvLine.name})
But you should take notice the headers from your Persons.csv file.
Imagine that your file got this header
id | name |
you must use this Cypher code, before the CREATE statment:
FIELDTERMINATOR "|"

How can I debug problems with warehouse creation?

When trying to create a warehouse from the Cloudant dashboard, sometimes the process fails with an error dialog. Other times, the warehouse extraction stays in a state of triggered even after hours.
How can I debug this? For example is there an API I can call to see what is going on?
Take a look inside the document inside the _warehouser database, and look for the warehouser_error_message element. For example:
"warehouser_error_message": "Exception occurred while creating table.
[SQL0670N The statement failed because the row size of the
resulting table would have exceeded the row size limit. Row size
limit: \"\". Table space name: \"\". Resulting row size: \"\".
com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-670,
SQLSTATE=54010, SQLERRMC=32677;;34593, DRIVER=4.18.60]"
The warehouser error message usually gives you enough information to debug the problem.
You can view the _warehouser document in the Cloudant dashboard or use the API, e.g.
export cl_username='<your_cloudant_account>'
curl -s -u $cl_username -p \
https://$cl_username.cloudant.com/_warehouser/_all_docs?include_docs=true \
| jq [.warehouse_error_code]

Resources