I need to read a configuration value early in the CreateHostBuilder(string[] args) method as shown in code below. But my problem is, I couldn't figure out how to get to the Configuration object in this place, this early in the process. How do I do that? Or is it possible to do?
public class Program
{
public static void Main(string[] args)
{
CreateHostBuilder(args).Build().Run();
}
public static IHostBuilder CreateHostBuilder(string[] args)
{
var builder = Host.CreateDefaultBuilder(args)
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.UseStartup<Startup>();
});
// Need to read a configuration setting from appsettings.json here, such as
// var value = Configuration["MyValue"];
...
}
}
Related
I'm fairly new to Flink and would be grateful for any advice with this issue.
I wrote a job that receives some input events and compares them with some rules before forwarding them on to kafka topics based on whatever rules match. I implemented this using a flatMap and found it worked well, with one downside: I was loading the rules just once, during application startup, by calling an API from my main() method, and passing the result of this API call into the flatMap function. This worked, but it means that if there are any changes to the rules I have to restart the application, so I wanted to improve it.
I found this page in the documentation which seems to be an appropriate solution to the problem. I wrote a custom source to poll my Rules API every few minutes, and then used a BroadcastProcessFunction, with the Rules added to to the broadcast state using processBroadcastElement and the events processed by processElement.
The solution is working, but with one problem. My first approach using a FlatMap would process the events almost instantly. Now that I changed to a BroadcastProcessFunction each event takes 60 seconds to process, and it seems to be more or less exactly 60 seconds every time with almost no variation. I made no changes to the rule matching logic itself.
I've had a look through the documentation and I can't seem to find a reason for this, so I'd appreciate if anyone more experienced in flink could offer a suggestion as to what might cause this delay.
The job:
public static void main(String[] args) throws Exception {
// set up the streaming execution environment
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
// read the input from Kafka
DataStream<KafkaEvent> documentStream = env.addSource(
createKafkaSource(getSourceTopic(), getSourceProperties())).name("Kafka[" + getSourceTopic() + "]");
// Configure the Rules data stream
DataStream<RulesEvent> ruleStream = env.addSource(
new RulesApiHttpSource(
getApiRulesSubdomain(),
getApiBearerToken(),
DataType.DataTypeName.LOGS,
getRulesApiCacheDuration()) // Currently set to 120000
);
MapStateDescriptor<String, RulesEvent> ruleStateDescriptor = new MapStateDescriptor<>(
"RulesBroadcastState",
BasicTypeInfo.STRING_TYPE_INFO,
TypeInformation.of(new TypeHint<RulesEvent>() {
}));
// broadcast the rules and create the broadcast state
BroadcastStream<RulesEvent> ruleBroadcastStream = ruleStream
.broadcast(ruleStateDescriptor);
// extract the resources and attributes
documentStream
.connect(ruleBroadcastStream)
.process(new FanOutLogsRuleMapper()).name("FanOut Stream")
.addSink(createKafkaSink(getDestinationProperties()))
.name("FanOut Sink");
// run the job
env.execute(FanOutJob.class.getName());
}
The custom HTTP source which gets the rules
public class RulesApiHttpSource extends RichSourceFunction<RulesEvent> {
private static final Logger LOGGER = LoggerFactory.getLogger(RulesApiHttpSource.class);
private final long pollIntervalMillis;
private final String endpoint;
private final String bearerToken;
private final DataType.DataTypeName dataType;
private final RulesApiCaller caller;
private volatile boolean running = true;
public RulesApiHttpSource(String endpoint, String bearerToken, DataType.DataTypeName dataType, long pollIntervalMillis) {
this.pollIntervalMillis = pollIntervalMillis;
this.endpoint = endpoint;
this.bearerToken = bearerToken;
this.dataType = dataType;
this.caller = new RulesApiCaller(this.endpoint, this.bearerToken);
}
#Override
public void open(Configuration configuration) throws Exception {
// do nothing
}
#Override
public void close() throws IOException {
// do nothing
}
#Override
public void run(SourceContext<RulesEvent> ctx) throws IOException {
while (running) {
if (pollIntervalMillis > 0) {
try {
RulesEvent event = new RulesEvent();
event.setRules(getCurrentRulesList());
event.setDataType(this.dataType);
event.setRetrievedAt(Instant.now());
ctx.collect(event);
Thread.sleep(pollIntervalMillis);
} catch (InterruptedException e) {
running = false;
}
} else if (pollIntervalMillis <= 0) {
cancel();
}
}
}
public List<Rule> getCurrentRulesList() throws IOException {
// call API and get rulles
}
#Override
public void cancel() {
running = false;
}
}
The BroadcastProcessFunction
public abstract class FanOutRuleMapper extends BroadcastProcessFunction<KafkaEvent, RulesEvent, KafkaEvent> {
protected final String RULES_EVENT_NAME = "rulesEvent";
protected final MapStateDescriptor<String, RulesEvent> ruleStateDescriptor = new MapStateDescriptor<>(
"RulesBroadcastState",
BasicTypeInfo.STRING_TYPE_INFO,
TypeInformation.of(new TypeHint<RulesEvent>() {
}));
#Override
public void processBroadcastElement(RulesEvent rulesEvent, BroadcastProcessFunction<KafkaEvent, RulesEvent, KafkaEvent>.Context ctx, Collector<KafkaEvent> out) throws Exception {
ctx.getBroadcastState(ruleStateDescriptor).put(RULES_EVENT_NAME, rulesEvent);
LOGGER.debug("Added to broadcast state {}", rulesEvent.toString());
}
// omitted rules matching logic
}
public class FanOutLogsRuleMapper extends FanOutRuleMapper {
public FanOutLogsJobRuleMapper() {
super();
}
#Override
public void processElement(KafkaEvent in, BroadcastProcessFunction<KafkaEvent, RulesEvent, KafkaEvent>.ReadOnlyContext ctx, Collector<KafkaEvent> out) throws Exception {
RulesEvent rulesEvent = ctx.getBroadcastState(ruleStateDescriptor).get(RULES_EVENT_NAME);
ExportLogsServiceRequest otlpLog = extractOtlpMessageFromJsonPayload(in);
for (Rule rule : rulesEvent.getRules()) {
boolean match = false;
// omitted rules matching logic
if (match) {
for (RuleDestination ruleDestination : rule.getRulesDestinations()) {
out.collect(fillInTheEvent(in, rule, ruleDestination, otlpLog));
}
}
}
}
}
Maybe you can give the complete code of the FanOutLogsRuleMapper class, currently the match variable is always false
Please let me know how to read properties file in device farm, since following code is not working. I have looked into various solutions but still device farm is not able to recognize properties file, although this code works fine locally
public abstract class AndroidCapabilities {
// protected static AppiumDriver<MobileElement> driver;
public static ExtentHtmlReporter reporter;
public static ExtentReports extent;
public static ExtentTest logger1;
// #Parameters("browser")
// #BeforeSuite
// public void setUp() throws MalformedURLException {
public static AndroidDriver<MobileElement> driver;
public abstract String getName();
#BeforeTest
public abstract void setUpPage();
#BeforeSuite
public void setUpAppium() throws MalformedURLException {
DesiredCapabilities capabilities = new DesiredCapabilities();
// capabilities.setCapability("device", "Android");
// capabilities.setCapability("platformName", "Android");
// capabilities.setCapability(MobileCapabilityType.BROWSER_NAME, "Chrome");
final String URL_STRING = "http://127.0.0.1:4723/wd/hub";
URL url = new URL(URL_STRING);
// Use a empty DesiredCapabilities object
driver = new AndroidDriver<MobileElement>(url, capabilities);
// Use a higher value if your mobile elements take time to show up
driver.manage().timeouts().implicitlyWait(35, TimeUnit.SECONDS);
}
public static Properties properties;
static {
properties = new Properties();
FileInputStream fis;
InputStream input;
try {
// fis = new FileInputStream(System.getProperty("user.dir") +
// "//src//test//resources//Properties//Android_OR.properties");
fis = (FileInputStream) Thread.currentThread().getContextClassLoader()
.getResourceAsStream("//Properties//Android_OR.properties");
System.out.println(properties.getProperty("url"));
properties.load(fis);
} catch (IOException e) {
e.printStackTrace();
}
}
#BeforeTest
public void createReport() {
reporter = new ExtentHtmlReporter("./extent.html");
extent = new ExtentReports();
extent.attachReporter(reporter);
}
#AfterTest
public void flush() throws IOException {
extent.flush();
// reporter.setAppendExisting(true);
}
#AfterSuite
public void closeApplication() {
driver.quit();
Reporter.log("===Session End===", true);
}
}
Faced the same issue. Instead of using System.getProperty, try this:
// pass fileName as "Android_OR.properties"
public Properties loadProperty(String fileName) {
Properties prop = new Properties();
try (InputStream input = Base.class.getClassLoader().getResourceAsStream(filePath)) {
if (input == null) {
System.out.println("Sorry, unable to find config.properties");
return prop;
}
// load a properties file from class path, inside static method
prop.load(input);
} catch (IOException ex) {
ex.printStackTrace();
}
return prop;
}
I am trying to test my Camel Routes using CamelTestSupport. I have my routes defined in a class like this
public class ActiveMqConfig{
#Bean
public RoutesBuilder route() {
return new SpringRouteBuilder() {
#Override
public void configure() throws Exception {
from("activemq:{{push.queue.name}}").to("bean:PushEventHandler?method=handlePushEvent");
}
};
}
}
And my test class look like this
#RunWith(SpringRunner.class)
public class AmqTest extends CamelTestSupport {
#Override
protected RoutesBuilder createRouteBuilder() throws Exception {
return new ActiveMqConfig().route();
}
#Override
protected Properties useOverridePropertiesWithPropertiesComponent() {
Properties properties = new Properties();
properties.put("pim2.push.queue.name", "pushevent");
return properties;
}
protected Boolean ignoreMissingLocationWithPropertiesComponent() {
return true;
}
#Mock
private PushEventHandler pushEventHandler;
#BeforeClass
public static void setUpClass() throws Exception {
BrokerService brokerSvc = new BrokerService();
brokerSvc.setBrokerName("TestBroker");
brokerSvc.addConnector("tcp://localhost:61616");
brokerSvc.setPersistent(false);
brokerSvc.setUseJmx(false);
brokerSvc.start();
}
#Override
protected JndiRegistry createRegistry() throws Exception {
JndiRegistry jndi = super.createRegistry();
MockitoAnnotations.initMocks(this);
jndi.bind("pushEventHandler", pushEventHandler);
return jndi;
}
#Test
public void testConfigure() throws Exception {
template.sendBody("activemq:pushevent", "HelloWorld!");
Thread.sleep(2000);
verify(pushEventHandler, times(1)).handlePushEvent(any());
}}
This is working perfectly fine. But I have to set the placeholder {{push.queue.name}} using useOverridePropertiesWithPropertiesComponent function. But I want it to be read from my .yml file.
I am not able to do it. Can someone suggest.
Thanks
Properties are typically read from .properties files. But you can write some code that read the yaml file in the useOverridePropertiesWithPropertiesComponent method and put them into the Properties instance which is returned.
Thank Claus.
I got it working by doing this
#Override
protected Properties useOverridePropertiesWithPropertiesComponent() {
YamlPropertySourceLoader loader = new YamlPropertySourceLoader();
try {
PropertySource<?> applicationYamlPropertySource = loader.load(
"properties", new ClassPathResource("application.yml"),null);
Map source = ((MapPropertySource) applicationYamlPropertySource).getSource();
Properties properties = new Properties();
properties.putAll(source);
return properties;
} catch (IOException e) {
LOG.error("Config file cannot be found.");
}
return null;
}
suppose I want to mock a method with the following signature:
public A foo(A a)
I want foo to be mocked in a way that it returned what it received (that is the same instance a)
I tried unsuccessfully the following:
Capture<A> capture = new Capture();
expect(myclass.foo(capture)).andReturn(capture.getValue());
This does not work because the capture is still empty when getValue() is called.
Any ideas?
Thanks
public class A {
public <A> A foo(A a) {
return null;
}
public static void main(String[] args) throws Exception {
A mock = createNiceMock(A.class);
expect(mock.foo(anyObject())).andAnswer(new IAnswer<Object>() {
#Override
public Object answer() throws Throwable {
return EasyMock.getCurrentArguments()[0];
}
}).anyTimes();
replay(mock);
System.out.println(mock.foo("1"));
System.out.println(mock.foo(2L));
}
}
Prints out:
1
2
I am developing an application which uses WindowsFormsApplicationBase to enforce Single Instance. I get the following error when calling a method on a Remote object. It works fine if I don't use Single Instance approach.
System.Runtime.Remoting.RemotingException: Authentication failure ---> System.IO.IOException: Unable to read data from the transport connection: The connection was closed.
at System.Net.Security.NegoState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.NegotiateStream.AuthenticateAsClient(NetworkCredential credential, String targetName, ProtectionLevel requiredProtectionLevel, TokenImpersonationLevel allowedImpersonationLevel)
at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.CreateAuthenticatedStream(Stream netStream, String machinePortAndSid)
Here is my Code:
public class EntryPoint
{
[STAThread]
public static void Main(string[] args)
{
SingleInstanceManager sim = new SingleInstanceManager();
sim.Run(args);
}
}
public class SingleInstanceManager : WindowsFormsApplicationBase
{
private App app;
public SingleInstanceManager()
{
IsSingleInstance = true;
}
protected override bool OnStartup(Microsoft.VisualBasic.ApplicationServices.StartupEventArgs eventArgs)
{
app = new App();
app.InitializeComponent();
app.Run();
return false;
}
protected override void OnStartupNextInstance(StartupNextInstanceEventArgs eventArgs)
{
base.OnStartupNextInstance(eventArgs);
app.Activate();
}
}
This is how I am invoking the Remoting object:
public Hashtable GetData(string[] arg1, string[] arg2)
{
IDataProvider dataProvider = (IDataProvider )Activator.GetObject(typeof(IDataProvider ), "tcp://.....");
Hashtable data = dataProvider.GetData(arg1, arg2);
return data;
}
Thanks in advance.
I found the solution myself.
I used the following to implement single instance(http://www.ai.uga.edu/mc/SingleInstance.html).
[STAThread]
static void Main() // args are OK here, of course
{
bool ok;
m = new System.Threading.Mutex(true, "YourNameHere", out ok);
if (! ok)
{
MessageBox.Show("Another instance is already running.");
return;
}
Application.Run(new Form1()); // or whatever was there
GC.KeepAlive(m); // important!
}