I am building an angular 8 application and am storing JSON data in a list data type in DynamoDB. I can insert the records just fine and can query the table for the data but I'm having issues grabbing the data in the list data type.
Here is how it looks in a console log
I don't have any issues grabbing the String data values, only the nested data in the List data type.
If your issue is related parsing the objects returned from dynamo you can use the DynamoDB Converter
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/Converter.html#unmarshall-property
this will convert the returned dynamo record into a json record.
Also if you're using the sdk, consider also using the https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/DocumentClient.html where it will automatically convert dynamo records into json records.
Related
Using the REST connector in Azure Data Factory, I am trying to fetch the Facebook campaign details.
In the pipeline, I have a web activity followed by copy activity. In the mapping section, I can see only the three columns (Id, name, status) from the first array and not getting those columns listed inside the second array.
graph.facebook.com
data factory mapping
Is there a way to get the columns listed inside the array? I also tried creating a data flow taking the Json file as source and then used the flatten transformation, still I cannot see the columns related to campaigns. Any help is appreciated. Thanks again.
I tested and find that Data Factory will consider the first object/JSON array as the JSON schema.
If you can adjust the JSON data, then the "insights" can be recognized:
Schema:
If you can't, then the "insights" column will be missed:
In this case, there isn't a way to get all the columns listed inside the array.
HTH.
#Leon Yue, I found a way to do that.
Step 1: 1. Copy Facebook campaign data using the REST connector and save as JSON in Azure Blob.
Step 1: copy activity to extract FB data as JSON and save in Blob
Step 2: 2. Create a Data flow, considering the JSON file from Blob as the source.
Step 2: Data flow task
Step 3: Create a JSON schema and save it in your desktop with the insights array in the first row(which has all the column values) As per your previous comments, I created the JSON schema such that ADF will consider the first object/ JSON array as the JSON schema.
Step 4: In the Data flow - Source dataset, map the JSON schema using the 'Import schema' option from sample file.
Step 4: Import schema
Now you will be able to see all the columns from the array.
All columns
Flatten JSON
The "requirements" are something like this: as a user I want to upload a dataset in CSV, JSON, or other supported text formats and be able to do basic REST queries against it such as selecting all first names in the dataset or select the first 10 rows.
I'm struggling to think of the "best" way to store this data. While, off the bat I don't think this will generate millions of datasets, it seems generally bad to create a new table for every dataset for a user as, eventually, I would hit the inode limit. I could store as flat files in something like S3 that's cached but then it still does require opening and parsing the file to query it.
Is this a use case for the JSON type in Postgres? If not, what would be the "right" format and place to store this data?
I have a table in Cassandra, in which one column is a BLOB.
I wish to update only some values in that blob. Is that possible ?
Example :
String form of BLOB is let's say:
{"name":"ABC","rollNum": "1234"}
I want to make it as :
{"name":"ABC","rollNum": "1333"} with an CQL update query.
Originally this column gets update from my JAVA code where I send byte[] to be inserted in this BLOB column.
Now, I want to update just some fields without doing any type of select on this row.
You can't do this in general.
Cassandra as any other database does not know how to interpret your blob. You will
need to read, parse, update and save your blob again
use a map instead
use single fields - which will give the most performance
Apart from that, updates like you want to do can be archived in document databases like MongoDB.
I have successfully downloaded twitter data through flume directly into HBase table containing one column family and all of the data is stored in one column like this
hbase(main):005:0> scan 'tweet'
ROW
default00fbf898-6f6e-4b41-aee8-646efadfba46
COLUMN+CELL
column=data:pCol, timestamp=1454394077534, value={"extended_entities":{"media":[{"display_url":"pic.twitter.com/a7Mjq2daKZ","source_user_id":2987221847,"type":"photo"....
Now i want to access structs and arrays through HBase like we can access then in Hive. I have tried googling the issue but still clue less. Kindly Help
You can't query display_url , source_user_id or another json fields in hbase directly. You should use a document store nosql db like mongodb.
I'm trying an API that will list all values within all documents in my Cloudant database. I use '_all_docs' at the end of my API however this only shows the IDs and revs of my JSON documents, it does not show all values within the documents. Can an API show all the data within the documents, or should I be creating a 1 document with arrays (representing what would be rows on an SQL table)?
If you want to display the contents of the document append the following to the query string.
include_docs=true
You can also use this on Views and Search indexes to get the complete doc.