we are working on a datawarehouse project and need to pull data from our customer´s shopify database.
We want to analyse inventory stock quantity and stock value, also do market analysis, traffic analysis, top landing pages, conversions, revenues and others.
Shopify offers REST APIs for data pulling but not all our requirements can be fulfilled.
Our question are wether
there is a plan we can order, like shopify partner plus or AdminAPI that allows data pulling all data fields?
are there experiences in data pulling from shopify, deeper than the published REST API queries?
I am new to shopify requirements, that´s why my questions.
I am not sure that if it can solve your problem. Currently, our company uses an online tool called Acho Studio to retrieve Shopify data. It's like ETL tools that allow you to connect to Shopify and export data to somewhere. The difference is that the tool hosts a sever, so you can store data on it. Also, you can write SQL queries or apply some actions to do analysis.
This is the codes snippet i am using
# Prepare headers
headers = {}
headers["Content-Type"] = "application/json"
headers["X-Shopify-Access-Token"] = f"{access_token}"
# Define the shopify url
shop_host = f'https://{hostName}.myshopify.com'
orderEndpoint = f'{order_endpoint}'
url = f'{shop_host}/{orderEndpoint}'
params = {'status': 'any',
'limit': 250,
'created_at_min': '2022-01-01', 'created_at_max' : '2022-06-20'}
response = session.get(url=url, headers=headers, params=params)
orders = response.json()['orders'] # order data in json format
while 'next' in response.links:
try:
session = requests.Session()
response = requests.get(response.links['next']['url'], headers=headers)
orders.extend(response.json()['orders'])
except KeyError as e:
if response.links == 'next':
print('go next page')
else:
print('No more pages')
I am able to pull the Order data and other data such as refund, .etc
Related
In salesforce, I have a report that has more than 2000 records (40000 rows). When I am trying to get that report via API I am getting only 2000 rows. It seems there is a limit of 2,000 results that can be returned for a given request. I am using the below code to get the data.
URIBuilder builder = new URIBuilder(salesforceConnection.getInstanceUrl());
builder.setPath("/services/data/v39.0/analytics/reports/" + recordId);
final HttpGet get = new HttpGet(builder.build());
get.setHeader("Authorization", AppConstants.BEARER + salesforceConnection.getAccessToken());
final HttpResponse queryResponse = httpclient.execute(get);
Is there a way to query the remaining data?
This is a hard limit in the Salesforce REST API that we have also encountered.
Source on the REST API docs
I looked in vain for any update or method on the report API that would alter this.
My requirement is to pull data from an external api, first call returns only 100 records, but with a header information stating total pages and total records in remote database. I need to pull all this records at once and insert into my database, next call to the api should only pull new records in the remote database.
I am working with ASP.NET Core 3.0 and a SQL Server database.
public void GetReservation(int? pageNumber, int? pageSize)
{
using (HttpClient client = new HttpClient())
{
client.BaseAddress = new Uri("https://www.sitename.com/url");
MediaTypeWithQualityHeaderValue contentType = new
MediaTypeWithQualityHeaderValue("application/json");
client.DefaultRequestHeaders.Accept.Add(contentType);
HttpResponseMessage response = client.GetAsync($"/api/serviceurl?pageNumber=
{pageNumber}&pageSize={pageSize}").Result;
string stringData = response.Content.ReadAsStringAsync().Result;
List<Reservations> data = JsonConvert.DeserializeObject<List<Reservations>>(stringData);
var headerInfo = response.Headers.GetValues("X-Pagination").First();
XPaginationObject obj= JsonConvert.DeserializeObject<XPagination>(headerInfo);
// Insert into database code only enters first 100 page
}
}
headerinfo contains totalpages, totalrecords, currentpage, hasNext, hasprevious...
It looks to me you're almost there: just run this method in a loop until you have all the records.. But first you need to get the total number of pages.
What I would do is:
1: call the api with pagenumber 1 and size 0 so you receive a header.
2: Get the info from the header and loop over pages until you are done.
3. You will have to write your own logic for only getting the new reservations, for instance, store the last received page and record number so you can skip these the next time.
Does this answer your question?
P.S.: It could very well be possible that your data provider only allows getting 100 rows at a time. If this is the case you will have to loop over 100-record pages until you have received all the pages.
I have exported data from Google App Maker(2) into spreadsheet. The problem is, I have to do pivot table manually in the exported sheet in order to get the table such as in no(3). Image(1) show the field that I have in the datasource.
(1) Screenshot of what I have in my datasource
(2) Data exported from AppMaker into Spreadsheet
(3) Expected pivot table to be exported
Can anyone teach me how to query data from the existing datasource to produce another table such as in no(3) in Google App Maker directly, so that I just export that data directly to sheet and no need to do pivot manually after the data has been exported .. or is there anyway better than that anyone else know how to do it?
I would rather to go with some nice charts to layout these stats... but if you want to go Sheets way you'll need to do some scripting with Apps Script Sheets Advanced API.
// server side code
function addPivotTable_(spreadsheetId, pivotSourceDataSheetId, destinationSheetId) {
var requests = [{
// Request object goes here
}];
var response = Sheets.Spreadsheets.batchUpdate({'requests': requests}, spreadsheetId);
}
I would like to rewrite the example from the GAE djangoforms article to be show most up to date after submitting a form (e.g. when updating or adding a new entry) on Google App Engine using the High Replication Datastore.
The main recurring query in this article is:
query = db.GqlQuery("SELECT * FROM Item ORDER BY name")
which we will translate to:
query = Item.all().order('name') // datastore request
This query I would like to get the latest updated data from the high replication datastore after submitting the form (only in these occasions, I assume I can redirect to a specific urls after submission which just uses the query for the latest data and in all other cases I would not do this).
validating the form storing the results happens like:
data = ItemForm(data=self.request.POST)
if data.is_valid():
# Save the data, and redirect to the view page
entity = data.save(commit=False)
entity.added_by = users.get_current_user()
entity.put() // datastore request
and getting the latest entry from the datastore for populating a form (for editing) happens like:
id = int(self.request.get('id'))
item = Item.get(db.Key.from_path('Item', id)) // datastore request
data = ItemForm(data=self.request.POST, instance=item)
So how do I add entity groups/ancestor keys to these datastore queries to reflect the latest data after form submission. Please note, I don't want all queries to have the latest data, when populating a form (for editing) and after submitting a form.
Who can help me with practical code examples?
If it is in the same block, you have reference of the current intance.
Then once you put() it, you can get its id by:
if data.is_valid():
entity = data.save(commit=False)
entity.added_by = users.get_current_user()
entity.put()
id= entity.key().id() #this gives you inserted data id
Let's say I've got a SQL 2008 database table with lots of records associated with two different customers, Customer A and Customer B.
I would like to build a fat client application that fetches all of the records that are specific to either Customer A or Customer B based on the credentials of the requesting user, then stores the fetched records in a temporary local table.
Thinking I might use the MS Sync Framework to accomplish this, I started reading about row filtering when I came across this little chestnut:
Do not rely on filtering for security.
The ability to filter data from the
server based on a client or user ID is
not a security feature. In other
words, this approach cannot be used to
prevent one client from reading data
that belongs to another client. This
type of filtering is useful only for
partitioning data and reducing the
amount of data that is brought down to
the client database.
So, is this telling me that the MS Sync Framework is only a good option when you want to replicate an entire table between point A and point B?
Doesn't that seem to be an extremely limiting characteristic of the framework? Or am I just interpreting this statement incorrectly? Or is there some other way to use the framework to achieve my purposes?
Ideas anyone?
Thanks!
No, it is only a security warning.
We use filtering extensively in our semi-connected app.
Here is some code to get you started:
//helper
void PrepareFilter(string tablename, string filter)
{
SyncAdapters.Remove(tablename);
var ab = new SqlSyncAdapterBuilder(this.Connection as SqlConnection);
ab.TableName = "dbo." + tablename;
ab.ChangeTrackingType = ChangeTrackingType.SqlServerChangeTracking;
ab.FilterClause = filter;
var cpar = new SqlParameter("#filterid", SqlDbType.UniqueIdentifier);
cpar.IsNullable = true;
cpar.Value = DBNull.Value;
ab.FilterParameters.Add(cpar);
var nsa = ab.ToSyncAdapter();
nsa.TableName = tablename;
SyncAdapters.Add(nsa);
}
// usage
void SetupFooBar()
{
var tablename = "FooBar";
var filter = "FooId IN (SELECT BarId FROM dbo.GetAllFooBars(#filterid))";
PrepareFilter(tablename, filter);
}