I am using getdbt on redshift for data analytics operation. Can anyone please suggest, how to use --selector & --defer with "dbt run" commands.
What is the syntax ? What is the use of selectors.yml file?
Please share some examples.
Thanks
My interpretation of defer is a way to utilize the dbt cli to work with unbuilt or differential versions of the current & futures state defined versions of a model.
Example of why you may want to interact with that here: #2740 - Automating Non Regression Test
selectors being a relatively new feature, I also haven't seen much documentation to back this up but it is effectively a naming convention for a set of logical criteria (more than 1 tag, multiple directories, etc.)
I'd recommend this article in general for understanding the build path generation of a typical dbt run: How we made dbt runs 30% faster
From there, you can imagine that within a large project, there are huge interconnecting chains for each raw -> analytics ready transformation pipeline that you have.
We'll use Gitlab's open dbt project as an example.
Gitlab doesn't currently use selectors but they do make use of tags.
So they could build up a selectors.yml file using logical definitions like:
selectors.yml
selectors:
- name: sales_funnel
definition:
tag: salesforce
tag: sales_funnel
- name: arr
description: builds all arr models to current state + all upstream dependencies (zoho, zuora subscriptions, etc.)
default: true
definition:
tag: zuora_revenue
tag: arr
- name: month_end_process
description: builds reporting models about customer segments based on subscription activity for latest closed month
definition:
- union:
- method: fqn
value: rpt_available_to_renew_month_end
greedy: eager # default: will include all tests that touch selected model
- method: fqn
value: rpt_possible_to_churn_month_end
greedy: eager
Full list of valid selector definitions here: https://docs.getdbt.com/reference/node-selection/yaml-selectors#default
What that gives them the ability to do is on a cron job, via airflow, or some other orchestrator simply execute:
dbt run --selector month_end_process --full-refresh
And have confidence that the logical selection of models to run for that process is 100% accurately reproduced instead of another more fallible approach like assuming that all the models needed are in a single directory:
dbt run --models marts.finance.restricted_safe.reports --full-refresh
Architecturally, you likely won't need selectors until you get to the level of having multiple layers of tags and / or multiple layers of use-case directories to be mindful of within a single run.
Example: tags for the models' function, tags for the sources, tags for the bi/analyst consumers, tags for the materialization schedule, etc.
Related
Stuck at a trivial problem in Grails 3.1.5: Show the fields of a domain object, excluding one of them, including a transient property. Yes, this is my first Grails 3 project after many years with previous versions.
The generated show.gsp contains
<f:display bean="rfaPdffile"/>
This will include a field that may contain megabytes of XML. It should never be shown interactively. The display: false constraint is no longer in the docs, and seems to be silenty ignored.
Next I tried explicitly naming the fields:
<f:with bean="rfaPdffile">
<f:display property='fileName'/>
<f:display property='pageCount'/>
...
</f:with>
This version suprisingly displays the values without any markup whatsoever. Changing display to field,
<f:with bean="rfaPdffile">
<f:field property='fileName'/>
<f:field property='pageCount'/>
...
</f:with>
sort of works, but shows editable values. So does f:all.
In addition I tried adding other attributes to f:display: properties (like in f:table), except (like in f:all). I note in passing that those two attributes have different syntax for similar purposes.
In the Field plugin docs my use case is explicitly mentioned as a design goal. I must have missed something obvious.
My aim is to quickly throw together a prototype gui, postponing the details until later. Clues are greatly appreciated
If I understood you correctly, you want to have all bean properties included in the gsp but the one with the "megabytes of XML" should not be displayed to the user?
If that is the case you can do:
f:with bean="beanName"
f:field property="firstPropertyName"
f:field property="secondPropertyName"
And the one you don't wish to display:
g:hiddenField name="propertyName" value="${beanName.propertyName?}"
f:with
So list all the properties as f:field or f:display and put the one you don't wish to display in a g:hiddenField Grails tag
You can also try:
f:field property="propertyName"
widget-hidden="true"
but the Label is not hidden in this case.
Hope it helps
My own answer: "use the force, read the source". The f:display tag has two rather obvious bugs. I will submit a pull request as soon as I can.
Bugs aside, the documentation does not mention that the plugin may pick up the "scaffold" static property from the domain, if it has one. Its value should be a map. Its "exclude" key may define a list of property names (List of String) to be excluded. This probably works already for the "f:all" tag; bug correction is needed for the "f:display" tag.
My subjective impression is that the fields plugin is in a tight spot. It is intertwined with the Grails architecture, making it sensitive to changes in Grails internals. It is also required by the standard scaffolding plugin, making it very visible. Thus it needs constant attention from maintainers, a position not to be envied. Even now conventions for default constraints seem to have changed somewhere between Grails 3.0.9 and 3.1.7.
Performance of the fields plugin is sensitive to the total number of plugins in the app where it is used. It searches all plugins dynamically for templates.
For the wish list I would prefer stricter tag naming. The main tags should be verbs. There are two main actions, show and edit. For each action there are two main variants, single bean or multiple beans.
My answer is that at present (2 March 2017) there is no answer. I have searched the Net high and low. For the index (list) and create and edit views, the fields plugin works well enough. A certain field can be easily excluded from the create and edit views, relatively easily from the list view (by listing those that should show), and in no way I could find from the show view. This is such a common need that one would suspect it will be addressed soon. Also, easily showing derived values in the show view, like 'total' for an invoice. One can do that by adding an ordered list with a list item showing the value below the generated ordered list of values, but that is kind of a hack.
In some ways, the old way was easier. Yes, it generated long views, but they were generated and didn't have to be done by the programmer - just custom touches here and there.
I heard that scout eclipse can create table page - AbstractPageWithTable out of database table properties, without manually adding columns for it.
Is this true, and if it is is somebody knows how to do it?
Dynamic UI Components:
I am not sure I understand your question correctly. Can I rephrase it like this:
How can I add a column dynamically (without having to define them as inner class)?
Here some pointers on the Scout Forum:
Is possible to create a dynamic PageView and Form ?
How to create UI fields at runtime?
...
My personal opinion is that the scout philosophy is to have a declarative UI defined with inner classes. I am not sure I would recommend someone to use this dynamic approach...
Scaffoling for Eclipse Scout:
From your comment, I now understand your question like this:
Given an existing database, is it possible to generate some elements of the application like TablePages?
I am afraid there is no open-source tool doing this for the moment. In our company, we have some developers, having a series of scripts to do something like what you are describing.
There is a project called SAML (Scout Application Modeling Language) where the user interface is described with a DSL and the Scout entities are generated. This is more a prototype that was built for the Kepler version of scout (3.9). For the moment there is no active development on it.
I have also started some builder where you do something like:
SimpleApplicationBuilder app = SimpleApplicationBuilder.create("myapp", "simple.myapp", "");
IScoutFormBuilder formBuilder = ScoutBuilders.form("MyFirstForm", "simple.myapp.client.ui.forms", "simple.myapp.shared.services")
.propTitle(app.nlsKey("MyFormTitle"))
.propSubTitle(app.nlsKey("MyFormSubTitle"))
.propAskIfNeedSave(false)
.propDisplayHint(DisplayHint.DISPLAY_HINT_VIEW)
.addField(ScoutBuilders.groupBox("ContentGroupBox")
.propBorderVisible(false)
.addField(ScoutBuilders.stringField("MyStringField")
.propLabel(app.nlsKey("MyStringLabel"))
)
.addField(ScoutBuilders.tableField("MyTableField")
.propTableStatusVisible(true)
.addTable(ScoutBuilders.table("MyTable")
.propAutoResizeColumns(true)
.addColumn(ScoutBuilders.integerColumn("IdComlum")
.propDisplayable(false)
)
.addColumn(ScoutBuilders.stringColumn("NameComlum")
.propHeaderText(app.nlsKey("Name"))
)
.addColumn(ScoutBuilders.integerColumn("AgeComlum")
.propHeaderText(app.nlsKey("Age"))
)
)
)
);
app.addForm(formBuilder);
The idea is to have a Java model of the application source code. It should allows to be able to do modifications or to add new entities and then to generate or modify the code. My tool is not open source for the moment (it is not finished yet and I do not have enough time to work on it).
I definitively think that scaffolding for Eclipse Scout would be great. I have published a master thesis proposal to work with a student on this topic. I hope we will be able to move forward.
Here some pointers in the Eclipse Scout forum:
Scout usage from command line
Form Generator based on DB Table layout
Let me know if you are interested in those topics.
I'm currently making a bilingual Expression Engine 2.5.2 website. I'm using this technique to create the two langues, which works perfectly.
I have created a {country_code} global variable in the two index.php files which allows me to detect the current language.
Using this technique, I have no problems to get language-relative data when accessing an entry. My only concern is that I apparently have to privilege a language-specific "clean" URL.
Example entry:
{entry_id} = 123
{title} = My test article
{title_permalink} = my-test-article
{name_fr} = Mon article
{name_en} = My article
If I request http://www.example.com/index.php/en/blog/articles/my-test-article, I expect to to find, in english, "My article" using the template articles in the blog template group.
Everything is fine, but the french translation is accessible when requesting http://www.example.com/index.php/fr/blog/articles/my-test-article. The correct translation of the URL should be http://www.example.com/index.php/fr/blogue/articles/mon-article-test.
Anyone encountered a problem like this? Any solutions via extensions or modules?
I believe the Transcribe module solves this by both providing the ability to translate template group and template names, and having you create a separate entry for each language and piece of content in your site (hence, you have two separate URL titles). But that means buying into their entire methodology for a multi-lingual site.
Myself, I usually just stick to using the entry_id instead of the url_title, and live with the template names being in the primary language.
The best way I found to achieve this is by embedding templates with segment translations, duplicating template groups and duplicating channels.
In the blog/articles template:
{embed="shared/.head" segment_2_translation="blogue" segment_3_translation="articles"}
In the blogue/articles template:
{embed="shared/.head" segment_2_translation="blog" segment_3_translation="articles"}
In shared/.head template:
[...] {if lang == "fr"}English{if:else}Français{/if} [...]
And then you can create a Articles (FR) and a Articles (EN) channels, and each will have their unique URL titles. You can also add a relationship custom field for each channel to associate an entry with it's translation.
It feels messy, but it is the only way I could make it work without modules, plugins or whatnot.
I would like to translate my ExtJS application in different languages. My issue is that I'm using ExtJS MVC framework, and most of my JS files are downloaded dynamically by the framework itself.
The ideal solution (that I thought of) would be to have an extra option in the Ext.Loader (or in my Ext.app.Application) that would define the language to use, and depending on this to automatically download such file as "a.MyClass.fr.js" after loading my "a.MyClass.js" (which would contain an Ext.apply, overriding my string resources). That's probably not available in the ExtJS framework at the moment.
The alternative solution I can see, is to perform a trick on the server-side. First, a cookie would be created on the client, to set to the language. On the server-side, I could catch all the requests to JS files, then if a cookie is set (='fr' for example), I'd combine the requested JS file (MyClass.js) with its i18n's friend (MyClass.fr.js) dynamically on the server and return the result. That would work, but it's really tricky because it implies other things (caching...).
Maybe the best way is to implement the first behavior I described in the ExtJS framework myself...
What do you think? I'm looking for a really clean and neat way of doing it! Thanks :)
I recently struggled with the same problem.
Finding a clean way to do this was quite a challenge - most alternatives were either..
1) Duplicate your code base per locale (WTH)
2) Download localized files overriding each of your components (Maintenance hell? What about the poor translators?)
3) Use/generate a static file containing translations and refer to it (All languages are downloaded? Extra build step to generate it? How do you keep them in synch?)
I tried to get the best of all worlds and ended up with a utility class responsible for:
1) Loading the ExtJS translation files (which basically apply overrides to extjs base components)
2) Loading a locale specific property resourcebundle (specifying which locale to load) from the server.
3) Prototyping String with a translate() method which queries the loaded store (containing the message bundle from the server) and returns the translation based on the value of the string.
This is the gist of things:
Bundle & prototyping:
localeStore.load({
callback : function(records, operation, success) {
// Define translation function (NB! Must be defined before any components which want to use it.)
function translate() {
var record = localeStore.getById(this.valueOf()) ;
if(record === null) {
alert('Missing translation for: ' + this.valueOf()); // Key is not found in the corresponding messages_<locale>.properties file.
return this.valueOf(); // Return key name as placeholder
} else {
var value = record.get('value');
}
return value;
}
String.prototype.translate = translate;
callback.call(); // call back to caller(app.js / Ext.Application), loading rest of application
}
});
As an example from a view:
this.copyButton = Ext.create('Ext.button.Button', {
disabled: true,
text: 'DOCUMENT_LIBRARY_MENU_COPYTO_BUTTON'.translate(),
action: 'openCopyDialog'
});
Bundle on the server (mesages_en.properties):
DOCUMENT_LIBRARY_MENU_COPYTO_BUTTON=Copy file
etc..
Pros:
No-fuss code, 'Your_key'.translate() makes it easy to read and aware that this is a localized string
None/little maintenance overhead (Keeping an override file for each locale? Jesus..)
You only load the locale you need - not the whole shabang.
If you really want to, you could even have your own translation for the ExtJS locale files in the same bundle.
You could write unit tests to ensure that all bundles contain the same keys, thus avoiding orphaned translations later
Cons:
Synchronous - the store must be loaded before your main app starts. I solved this by adding a callback from the utility class which was called once all texts were loaded.
No real-time population of texts.. though I didn't want to make my users overload the server either :P
So far my approach has worked out pretty well for my requirements.
Site load isn't noticeably slower and the bundles (containing ~200 keys/values per bundle) measure out at ~10kb during load.
There is currently no solution so I decided to create my own hack/addon on the Ext.Loader. I uploaded the code on GitHub: https://github.com/TigrouMeow/extjs-locale-loader. It's exactly what I needed and I really hope it will help others as well!
You should first complete your development phase and build your project or use ext-all.js file to I18s translate your UI
see: http://docs.sencha.com/ext-js/4-0/#!/example/locale/multi-lang.html
The appropriate language modifier script (/ext/local/ext-lang-xxx.js) needs to be loaded after ext is loaded (including dynamically loaded classes). In the example above, I would have probably used Ext.Loader.loadScriptFile but they eval a downloaded one directly. The only other thing is that your classes need to be built in different languages or you just use variables and reference the lang-specific variable file.
you could also use a variable in the Loader paths:
var lang='fr';
Loader
{
paths:
{
'Ext': '.',
'My': './src/my_own_folder'+'/'+lang
}
A friend of mine is now building a web application with J2EE and Struts, and it's going to be prepared to display pages in several languages.
I was told that the best way to support a multi-language site is to use a properties file where you store all the strings of your pages, something like:
welcome.english = "Welcome!"
welcome.spanish = "¡Bienvenido!"
...
This solution is ok, but what happens if your site displays news or something like that (a blog)? I mean, content that is not static, that is updated often... The people that keep the site have to write every new entry in each supported language, and store each version of the entry in the database. The application loads only the entries in the user's chosen language.
How do you design the database to support this kind of implementation?
Thanks.
Warning: I'm not a java hacker, so YMMV but...
The problem with using a list of "properties" is that you need a lot of discipline. Every time you add a string that should be output to the user you will need to open your properties file, look to see if that string (or something roughly equivalent to it) is already in the file, and then go and add the new property if it isn't. On top of this, you'd have to hope the properties file was fairly human readable / editable if you wanted to give it to an external translation team to deal with.
The database based approach is useful for all your database based content. Ideally you want to make it easy to tie pieces of content together with their translations. It only really falls down for all the places you may want to output something that isn't out of a database (error messages etc.).
One fairly old technology which we find still works really well, is to use gettext. Gettext or some variant seems to be available for most languages and platforms. The basic premise is that you wrap your output in a special function call like so:
echo _("Please do not press this button again");
Then running the gettext tools over your source code will extract all the instances wrapped like that into a "po" file. This will contain entries such as:
#: myfolder/my.source:239
msgid "Please do not press this button again"
msgstr ""
And you can add your translation to the appropriate place:
#: myfolder/my.source:239
msgid "Please do not press this button again"
msgstr "s’il vous plaît ne pas appuyer sur le bouton ci-dessous à nouveau"
Subsequent runs of the gettext tools simply update your po files. You don't even need to extract the po file from your source. If you know you may want to translate your site down the line, then you can just use the format shown above (the underscored function) with all your output. If you don't provide a po file it will just return whatever you put in the quotes. gettext is designed to work with locales so the users locale is used to retrieve the appropriate po file. This makes it really easy to add new translations.
Gettext Pros
Doesn't get in your way while coding
Very easy to add translations
PO files can be compiled down for speed
There are libraries available for most languages / platforms
There are good cross platform tools for dealing with translations. It is actually possible to get your translation team set up with a tool such as poEdit to make it very easy for them to manage translation projects
Gettext Cons
Solves your site "furniture" needs, but you would usually still want a database based approach for your database driven content
For more info on gettext see this wikipedia page
They way I have designed the database before is to have an News-table containing basic info like NewsID (int), NewsPubDate (datetime), NewsAuthor (varchar/int) and then have a linked table NewsText that has these columns: NewsID(int), NewsText(text), NewsLanguageID(int). And at last you have a Language-table that has LanguageID(int) and LanguageName(varchar).
Then, when you want to show your users the news-page you do:
SELECT NewsText FROM News INNER JOIN NewsText ON News.NewsID = NewsText.NewsID
WHERE NewsText.NewsLanguageID = <<Session["UserLanguageID"]>>
That Session-bit is a local variable where you store the users language when they log in or enters the site for the first time.
Java web applications support internationalization using the java standard tag library.
You've really got 2 problems. Static content and dynamic content.
for static content you can use jstl. It uses java ResourceBundles to accomplish this. I managed to get a Databased backed bundle working with the help of this site.
The second problem is dynamic content.
To solve this problem you'll need to store the data so that you can retrieve different translations based on the user's Locale. (Locale includes Country and Language).
It's not trivial, but it is something you can do with a little planning up front.
#Auron
thats what we apply it to. Our apps are all PHP, but gettext has a long heritage.
Looks like there is a good Java implementation
Tag libraries are fine if you're using JSP, but you can also achieve I18N using a template-based technology such as FreeMarker.