Exist a way to encode parameters in flink-conf.yaml file?"
For example storing parameter like 'security.ssl.internal.keystore' in encrypted format.
Flink currently does not support to encrypt configuration values. If this is required, then please create a JIRA issue with this feature request.
What Flink supports is to not display secrets in the web ui if their key contains password or secret. See FLINK-14047 for more details.
Related
I am building a Camel application to read message from Confluent Kafka. The messages are in Avro format and added below route configuration to read the Avro messages using schema registry in Camel route. When I enable the valueDeserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer,
I am not getting any messages from Kafka topic. I tested the route with out schema registry and able to consume the message.
Route definition:
from("kafka:topic1?sslTruststoreLocation=<jks file>
&valueDeserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer
&brokers=host1:9092,host2:9092,host3:9092
&sslKeystoreType=JKS
&groupId=grp1
&allowManualCommit=true
&consumersCount=10
&sslKeyPassword=<password>
&autoOffsetReset=earliest
&sslKeystorePassword=<password>
&securityProtocol=SSL
&sslTruststorePassword=<password>
&sslEndpointAlgorithm=HTTPS
&maxPollRecords=10
&sslTruststoreType=JKS
&sslKeystoreLocation=<keystore_path>
&autoCommitEnable=false
&additionalProperties.schema.registry.url=https://localhost:8081
&additionalProperties.basic.auth.user.info=abc:xyz
&additionalProperties.basic.auth.credentials.source=USER_INFO");
Can you please let me know, what is wrong in above configuration for schema registry. I also tried with EndPointRouteBuilder and same issue. However the producer application which is also Camel based and uses the schema registry for publishing Avro messages is working fine.
I figured out the way to configure the basic auth with Confluent schema registry. We need to configure as below
from("kafka:topic1?sslTruststoreLocation=<jks file>
&valueDeserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer
&brokers=host1:9092,host2:9092,host3:9092
&sslKeystoreType=JKS
&groupId=grp1
&allowManualCommit=true
&consumersCount=10
&sslKeyPassword=<password>
&autoOffsetReset=earliest
&sslKeystorePassword=<password>
&securityProtocol=SSL
&sslTruststorePassword=<password>
&sslEndpointAlgorithm=HTTPS
&maxPollRecords=10
&sslTruststoreType=JKS
&sslKeystoreLocation=<keystore_path>
&autoCommitEnable=false
&additionalProperties.schema.registry.url=https://localhost:8081
&additional-properties[basic.auth.user.info]=abc:xyz
&additional-properties[basic.auth.credentials.source]=USER_INFO");
Note here, we need to use additional-properties for basic.auth.user.info and basic.auth.credentials.source as mentioned above.
My issue was that the schema registry password contained special characters, such +.
So I had to wrap the property in RAW as described in the documentation [1]
Given the above example, it would then result in:
&additional-properties[basic.auth.user.info]=RAW(abc:xyz+)
[1] https://camel.apache.org/manual/faq/how-do-i-configure-endpoints.html#HowdoIconfigureendpoints-Configuringparametervaluesusingrawvalues
I am using Apache Camel (v2.19.1) in a spring boot application. I want to monitor the application with JMX MBeans.
I want to find out the load for a particular 'route' from the class "org.apache.camel.management.mbean.ManagedRoute" attributes Load01, Load05 etc...
I get empty string as values for Load attributes. The statistics is already enabled.
Kindly help.
Thank you,
You need to turn this on. You can find details in the JMX documentation how to turn on load statistics: http://camel.apache.org/camel-jmx.html, eg set the loadStatisticsEnabled=true
In OSB Layer when the endpoint uri is changed, I need to alert the core group that the endpoint has changed and to review it. I tried SLA Alert rules but it does not have options for it. My question is, the endpoint uri should be saved somewhere in the underlying database. If so what is the schema and the table name to query it.
URI or in fact any other part of OSB artifact is not stored in relational database but rather kept in memory in it's original XML structure. It can be only accessed thru dedicated session management API. Interfaces you will need to use are part o com.bea.wli.sb.management.configuration and com.bea.wli.sb.management.query packages. Unfortunately it is not as straightforward as it sounds, in short, to extract URI information you will need to:
Create session instance(SessionManagementMBean)
Obtain ALSBConfigurationMBean instance that operates on SessionManagementMBean
Create Query object instance(BusinessServiceQuery) an run it on ALSBConfigurationMBean to get ref object to osb artifact of your interest
Invoke getServiceDefinition on your ref object to get XML service
definition
Extract URI from XML service definition with XPath
Downside of this approach is that you are basically pooling configuration each time you want to check if anything has changed.
More information including JAVA/WLST examples can be found in Oracle Fusion Middleware Java API Reference for Oracle Service Bus
There is also a good blog post describing OSB customization with WLST ALSB/OSB customization using WLST
The information about services and all its properties can be obtained via Java API. The API documentation contains sample code, so you can get it up and running quite quickly, see the Querying resources paragraph when following the given link.
We use the API to read the service (both proxy and business) configuration and for simple management.
As long as you only read the properties you do not need to handle management sessions. Once you change the values, you need to start a session and activate it once you are done -- a very similar approach to Service bus console.
Http Form adapter serves as an authentication service in my application. I have not implemented any application on the Identity Provider to get user inputs.
Therefore, on successful authentication, SP verifies the user's signature and redirects to the application. At my target Resource, I receive an open token. Is it still possible to utilize the open Token Jar to read the user attributes from OTK?
**Note: ** In Service Provider, I use open token Adapter.
Also, please let me know if there is any other possible way of getting the user attributes other than using the open token adapter/http form adapter.
Thanks.
There are numerous SP Adapters you can choose to use for your last mile integration with your application. The OpenToken Adapter is just one of them. If your application is in Java and you are using the SP OpenToken Adapter, then you would most likely use the Java OpenToken Agent implementation within your application to read the OTK (documented in the Java Integration Kit). If you look at the Add Ons list, there are actually 3 flavors of OTK Agents (.NET, Java and PHP from PingID. Ruby on Rails and Perl are available via respective Open Source repositories).
However, you are not limited to OpenToken Adapters. The Agentless Integration Kit is also very popular for SP/last-mile integration with PingFederate.
Unfortunately, the question is just too open ended for the Stackoverflow format. I would suggest talking to your Ping Identity Solution Architect who can help steer you in the right direction and ask the necessary follow-up questions on your use case.
If understand the question correctly, you desire attributes to be fulfilled that the web application can read and utilize. This starts with the SP Connection configuration. I am going to assume you are using Active Directory and already configured that data source along with the Password Credential Validator (PCV) for the HTML Form IdP Adapter. In the SP Connection you will need to extend the attribute contract to define the values to put into the SAML assertion and then use the Active Directory data source to fulfill the attributes. When the SAML assertion is received by the PingFederate SP role server, the SP Adapter maps the attribute values from the SAML assertion into the OpenToken. When your application receives the OpenToken, it can read the values.
I am looking at using the Camel crypto tool for processing PGP data but have a requirement that the password to the keys used be either encrypted in the configuration file or be sourced from a secure server elsewhere. Is this possible without generating my own PGP processor?
Yes see the security menu on the Apache Camel web site: http://camel.apache.org/security.html
There is a section about configuration security, where you can use camel-jasypt for that: http://camel.apache.org/jasypt.html
This allows you to store encrypted usernames / passwords etc in a .properties file, and then you can refer to these properties from Camel crypto, using Camel's property placeholder: http://camel.apache.org/using-propertyplaceholder.html