I created database using SQL in hive.
And I looked for the database using HDFS.
But I couldn't find database in HDFS.
In hive:
CREATE DATABASE practice
LOCATION '/user/hive/warehouse'/
Checking:
hdfs dfs -ls /user/hive/warehouse
There is nothing in warehouse.
In addition, I created a table in a specific database in hive.
But, using Hue, I could see the table in the default location.
I wanna insert the table into a specific database location.
CREATE TABLE prac (
id INT,
title STRING,
salary INT,
posted TIMESTAMP
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
LOCATION '/user/hive/warehouse/practice.db/prac';
I couldn't find the table prac in the database practice in Hue and HDFS.
How can I see the database in HDFS?
And I also wanna know how to see the table in the specific database location.
Try by specifying db name while creating hive table prac by default hive creates tables in default database.
Example:
hive> CREATE DATABASE practice LOCATION '/user/hive/warehouse/practice.db';
hive> CREATE TABLE `practice.prac` ( id INT, title STRING, salary INT, posted TIMESTAMP ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION '/user/hive/warehouse/practice.db/prac';
Try using below command:
CREATE DATABASE practice
LOCATION '/user/hive/warehouse/practice.db';
hive by default uses '/user/hive/warehouse/' directory to create databases under this location. So while creating database, If you don't provide the location it will pick the database location like this '/user/hive/warehouse/practice.db'.
You can choose any location over hdfs until you have read and write permission over that location.
Related
I am in the process of building a grocery list app with Azure.
Currently, I have an azure function that writes json files to blob storage. Here is what my current blob storage looks like:
When Azure data factory copies over the data to my SQL DB, this is what it looks like:
Here is the code that I am using to create my table:
-- Create a new table called 'groceryitems' in schema 'grocerylistapp'
-- Drop the table if it already exists
IF OBJECT_ID('grocerylistapp.groceryitems', 'U') IS NOT NULL
DROP TABLE grocerylistapp.groceryitems
GO
CREATE SCHEMA grocerylistapp
GO
-- Create the table in the specified schema
CREATE TABLE grocerylistapp.groceryitems
(
id INT IDENTITY(1,1), -- Primary Key column
epoch DATE,
[name] VARCHAR(50),
PRIMARY KEY CLUSTERED([id] ASC)
-- Specify more columns here
);
GO
1
Oddly enough, when I originally did this with Data Warehouse / Synapse, I didn't get this issue, so I am assuming it is something to do particularly with SQL DB.
I also want to add that it randomly decides to distort one row, if I had 3,4, or 5 items in blob storage, it would pick one row and distort it as shown in the screenshot. It is not always the same item.
Thank you.
Firstly, make sure you have chosen the json format file as Soure dataset.
Here're the steps I tested, and it works well.
Source dataset(json format) from Blob storage:
Import the cchema of the source dataset:
Sink dataset settings:
Mapping settings:
Run the pipeline:
Check the data in table grocerylistapp.groceryitems:
Hope this helps.
I'm trying to work out a specific way to copy all data from a particular table (let's call it opportunities) and copy it into a new table, with a timestamp of the date copied into the new table, for the sole purpose of generating historic data into a database hosted in Azure Data Warehousing.
What's the best way to do this? So far I've gone and created a duplicate table in the data warehouse, with an additional column called datecopied
The query I've started using is:
SELECT OppName, Oppvalue
INTO Hst_Opportunities
FROM dbo.opportunities
I am not really sure where to go from here!
SELECT INTO is not supported in Azure SQL Data Warehouse at this time. You should familiarise yourself with the CREATE TABLE AS or CTAS syntax, which is the equivalent in Azure DW.
If you want to fix the copy date, simply assign it to a variable prior to the CTAS, something like this:
DECLARE #copyDate DATETIME2 = CURRENT_TIMESTAMP
CREATE TABLE dbo.Hst_Opportunities
WITH
(
CLUSTERED COLUMNSTORE INDEX,
DISTRIBUTION = ROUND_ROBIN
)
AS
SELECT OppName, Oppvalue, #copyDate AS copyDate
FROM dbo.opportunities;
I should also mention that the use case for Azure DW is million and billions of rows with terabytes of data. It doesn't tend to do well at low volume, so consider if you need this product, a traditional SQL Server 2016 install, or Azure SQL Database.
You can write insert into select query like below which will work with SQL Server 2008 +, Azure SQL datawarehouse
INSERT INTO Hst_Opportunities
SELECT OppName, Oppvalue, DATEDIFF(SECOND,{d '1970-01-01'},current_timestamp)
FROM dbo.opportunities
what is the format of schema when we create a new table using Voltdb?
I'm a newbie. I have researched for a while and read the explanation in this https://docs.voltdb.com/UsingVoltDB/ChapDesignSchema.php
Please give me more detal about the schema format when I create a new table.
Another quesiton is What is the call flow of the system, since a request comes to the system until a response is create.
Which class/function does it go through in the system.
Since VoltDB is a SQL compliant database, you would create a new table in VoltDB just as you would create a new table in any other traditional relational database. For e.g.,
CREATE TABLE MY_TABLE (id INTEGER NOT NULL, name VARCHAR(10));
You can find all the SQL DDL statements that you can run on VoltDB here
1.Make file yourSchemaName.sql anywhere in the system. Suppose yourSchemaName.sql looks like this
CREATE TABLE Customer (
CustomerID INTEGER UNIQUE NOT NULL,
FirstName VARCHAR(15),
LastName VARCHAR (15),
PRIMARY KEY(CustomerID)
);
2.fire sqlcmd in CLI inside folder where you have installed the voltdB.
if you haven't set the path then you have to type /bin/sqlcmd.
After firing the command, a simple way to load schema in your voltdB database is by typing /path/to/yourSchemaName.sql; command inside the sqlcmd utility and the schema named yourSchemaName.sql will be imported inside the database.
VoltdB is relational database,so Now you can use all of sql database queries inside this database.
I have a table. For example:
CREATE TABLE temp AS SELECT * FROM range(20)
How i can create in sqlite a new database with one command with the table temp?
Since your in sqlite, in the command line just type:
sqlite3 datbase_name.db
This will create a new database called database_name.db, now to create your table to this database just type:
sqlite3 database_name.db "CREATE TABLE .......;"
Type your create table statement inside the quotes.
For more info, please take a look at their documentation.
I think that would depend on the DBMS but this may be what you're looking for.
COPY Command Syntax
"COPY {FROM database | TO database | FROM database TO database} {APPEND|CREATE|INSERT|REPLACE} destination_table [(column, column, column, ...)] USING query"
http://docs.oracle.com/cd/B19306_01/server.102/b14357/apb.htm
I have one table named "Staff" in access and also have this table(same name) in SQL 2008.
Both table have thousands of records. I want to merge records from the access table to sql table without affecting the existing records in sql. Normally, I just export using OCBC driver and that works fine if that table doesn't exist in sql server. Please advise. Thanks.
A simple append query from the local access table to the linked sql server table should work just fine in this case.
So, just drop in the first (from) table into the query builder. Then change the query type to append, and you are prompted for the append table name.
From that point on, just drop in the columns you want (do not drop in the PK column, as they need not be used nor transferred in this case).
You can also type in the sql directly in the query builder. Either way, you will wind up with something like:
INSERT INTO dbo_custsql
( ADMINID, Amount, Notes, Status )
SELECT ADMINID, Amount, Notes, Status
FROM custsql1;
This may help: http://www.red-gate.com/products/sql-development/sql-compare/
Or you could write a simple program to read from each data set and do the comparison, adding, updating, and deleting, etc.