I need to pass a File to a class for unit tests. The class requires a File specifically, and can't be modified - it just reads it. (So I don't want to try to mock out that class or modify it - just pass it the File it needs.)
What's the recommended way to do this?
I put the file in src/test/resources/... and just passed that entire path in, and, since the test is run in the project root dir, this works. But this seems quite wrong.
UPDATE: To emphasize - the class needs a File object, not an InputStream or anything else.
this.getClass().getResourceAsStream(fileName) is recommended way to read files from resources
If you need to look up something on the classpath and have it returned as a File, try this:
import java.net.URL
import java.io.File
object Test extends App {
val fileUrl: URL = getClass.getResource("Test.class")
val file : File = new File(fileUrl.toURI())
println(s"File Path: ${file.getCanonicalPath}")
}
It more or less uses all Java classes, but it works. In my example, the Test object is finding its own compiled classfile, but you can obviously change it to whatever you want.
The JVM doesn't have a concept of getting a classpath resource as a File because resources don't necessarily correspond to files. (Often they're contained in jars.)
Your test could start by getting an InputStream and copying it into a temporary file.
Related
I am new to PyFlink. I have done the official training exercise in Java: https://github.com/apache/flink-training
However, the project I am working on must use Python as a programming language. I want to know if it is possible to write a data generator using the "SourceFunction". In older PyFlink versions this was possible, using Jython: https://nightlies.apache.org/flink/flink-docs-release-1.7/dev/stream/python.html#streaming-program-example
In newer examples the dataframe contains a finite set of data, which is never extended. I have not found any example of a data generator in PyFlink, e.g. https://github.com/apache/flink-training/blob/master/common/src/main/java/org/apache/flink/training/exercises/common/sources/TaxiRideGenerator.java
I am not sure which functionality the interfaces Source and SinkFunction provide. Can it be used somehow in python or can it only be used in combination with other pipelines or jar files? It looks like the methods "run()" and "cancel()" are not implemented and thus it cannot be used like some other classes, by overloading.
If it can not be used in Python, are there any other ways to use it? Someone may provide an easy example.
If it is not possible to use it, are there any other ways to write a data generator in OOP style? Take this example: https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/datastream_tutorial/ There the split() method is used to separate the stream. Basically, I want to do this by an extra class and just extending the stream, which was done in the Java TaxiRide example via "ctx.collect()". I am trying to avoid using Java, another framework for the pipeline, and Jython. It would be nice to get a short example code, but I appreciate any tips and advice.
I tried to use SourceFunction directly, but as already mentioned, I think this is a completely wrong way, resulting in an error: AttributeError: 'DataGenerator' object has no attribute '_get_object_id'
class DataGenerator(SourceFunction):
def __init__(self):
super().__init__(self)
self._num_iters = 1000
self._running = True
def run(self, ctx):
counter = 0
while self._running and counter < self._num_iters:
ctx.collect('Hello World')
counter += 1
def cancel(self):
self._running = False
Solution:
After looking in some older code using the classes Source and SinkFunction, I came to a solution. Here a kafka connector written in Java is used. The python code can be taken as an example of how to use pyflink's Source and SinkFuntion.
I have only written an example for the SourceFunction:
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.datastream import SourceFunction
from pyflink.java_gateway import get_gateway
class TaxiRideGenerator(SourceFunction):
def __init__(self):
java_src_class = get_gateway().jvm.org.apache.flink.training.exercises.common.sources.TaxiRideGenerator
java_src_obj = java_src_class()
super(TaxiRideGenerator, self).__init__(java_src_obj)
def show(ds, env):
# this is just a little helper to show the output of the pipeline
ds.print()
env.execute()
def streaming():
# arm the flink ExecutionEnvironment
env = StreamExecutionEnvironment.get_execution_environment()
env.set_parallelism(1)
taxi_src = TaxiRideGenerator()
ds = env.add_source(taxi_src)
show(ds, env)
if __name__ == "__main__":
streaming()
The second line in the class init was hard to find. I had expected to get an object in the first line.
You have to create a jar file after building this project.
I have entered the path until I see the folder "org":
$ cd flink-training/flink-training/common/build/classes/java/main
flink-training/common/build/classes/java/main$ ls
flink-training/common/build/classes/java/main$ org
flink-training/common/build/classes/java/main$ jar cvf flink-training.jar org/apache/flink/training/exercises/common/**/*.class
Copy the jar file to the pyflink/lib folder, normally under your python environment, e.g. flinkenv/lib/python3.8/site-packages/pyflink/lib. Then start the script.
I have the two following methods and I am using them to store a special value locally and be able to access it on application restart:
(Store value locally:)
private void SaveSet(string key, string value)
{
ISharedPreferences prefs = PreferenceManager.GetDefaultSharedPreferences(this);
ISharedPreferencesEditor prefEditor = prefs.Edit();
prefEditor.PutString(key, value);
// editor.Commit(); // applies changes synchronously on older APIs
prefEditor.Apply(); // applies changes asynchronously on newer APIs
}
(Read it again:)
private string RetrieveSet(string key)
{
ISharedPreferences prefs = PreferenceManager.GetDefaultSharedPreferences(this);
return prefs.GetString(key, null);
}
This works perfectly. Now is it possible to access and edit this Shared Preferences externally? Unfortunately, I cannot find any file when searching in folder
Phone\Android\data\com.<company_name>.<application_name>\files
nor anywhere else. I want / try to edit this value from my computer, after connecting the phone to it. Is this possible?
Alternatively: Can anyone maybe show me how to create a new file in the given path above, write/read it programmatically and how it stays there, even if application is closed / started again? So I can then edit this file with my computer anyhow?
I tried it with the following code, but unfortunately it doesn't work / no file is created or at least i cannot see it in the given path above:
//"This code snippet is one example of writing an integer to a UTF-8 text file to the internal storage directory of an application:"
public void SaveValueIntoNewFile(int value)
{
var backingFile = Path.Combine(System.Environment.GetFolderPath(System.Environment.SpecialFolder.Personal), "newFile.txt");
using (var writer = System.IO.File.CreateText(backingFile))
{
writer.WriteLine(value.ToString());
}
}
Would be very happy about every answer, thanks in advance and best regards
What you're looking for is where Android stores the Shared Preference file for applications that make use of it's default PreferenceManager.
I'd refer to this SO post which answers your question pretty well
SharedPreferences are stored in an xml file in the app data folder,
i.e.
/data/data/YOUR_PACKAGE_NAME/shared_prefs/YOUR_PREFS_NAME.xml
or the default preferences at:
/data/data/YOUR_PACKAGE_NAME/shared_prefs/YOUR_PACKAGE_NAME_preferences.xml
SharedPreferences added during runtime are not stored in the Eclipse
project.
Note: Accessing /data/data/ requires superuser
privileges
A simple method is to use Android Device Monotor,you can open it by clicking Tools--> android-->Android Device Monotor...
For example:
The path in my device is as follows:
/data/data/YOUR_PACKAGE_NAME/shared_prefs/YOUR_PACKAGE_NAME_preferences.xml
And we notice three buttons in the upper right corner of the picture.
The first one is used toPull a file from the device,the second one is used to Push a file onto the device,and the last one is used to delete the preferences.xml file.
So we can pull the preferences.xml file from current device to our computer and edit it as we want, and then push the updated preferences.xml to the folder again.Then we will get the value of preferences.xml file .
I would like to monitor all of the files in a given directory for changes, ie an updated timestamp. This use case seems natural for Camel using the file component, but I can't seem to find a way to configure this behavior.
A uri like:
file:/some/directory
will consume the files in the provided directory but will delete them.
A uri like:
file:/some/directory?noop=true
consumes each file once when it is added or when the route is started.
It's surprising that there isn't an option along the lines of
consumeOnChange=true
Is there a straightforward way to monitor file changes and not delete the file after consuming?
You can do this by setting up the idempotentKey to tell Camel how a file is considered changed. For example if the file size changes, or its timestamp changes etc.
See more details at the Camel file documentation at: https://camel.apache.org/components/latest/file-component.html
See the section Avoiding reading the same file more than once (idempotent consumer). And read about idempotent and idempotentKey.
So something alike
from("file:/somedir?noop=true&idempotentKey=${file:name}-${file:size}")
Or
from("file:/somedir?noop=true&idempotentKey=${file:name}-${file:modified}")
You can read here about the various ${file:xxx} tokens you can use: http://camel.apache.org/file-language.html
Setting noop to true will result in Camel setting idempotent=true as well, despite the fact that idempotent is false by default.
Simplest solution to monitor files would be:
.from("file:path?noop=true&idempotent=false&delay=60s")
This will monitor changes to all files in the given directory every one minute.
This can be found in the Camel documentation at: http://camel.apache.org/file2.html.
I don't think Camel supports that specific feature but with the existent options you can come up with a similar solution of monitoring a directory.
What you need to do is set a small delay value to check the directory and maintain a repository of the already read files. Depending on how you configure the repository (by size, by filename, by a mix of them...) this solution would be able to provide you information about news files and modified files. As a caveat it would be consuming the files in the directory very often.
Maybe you could use other solutions different from Camel like Apache Commons VFS2 (I wrote a explanation about how to use it for this scenario: WatchService locks some files?
I faced the same problem i.e. wanted to copy updated files also (along with new files). Below is my configuration,
public static void main(String[] a) throws Exception {
CamelContext cc = new DefaultCamelContext();
cc.addRoutes(createRouteBuilder());
cc.start();
Thread.sleep(10 * 60 * 1000);
cc.stop();
}
protected static RouteBuilder createRouteBuilder() {
return new RouteBuilder() {
public void configure() {
from("file://D:/Production"
+ "?idempotent=true"
+ "&idempotentKey=${file:name}-${file:size}"
+ "&include=.*.log"
+ "&noop=true"
+ "&readLock=changed")
.to("file://D:/LogRepository");
}
};
}
My testing steps:
Run the program and it copies few .log files from D:/Production to D:/LogRepository and then continues to poll D:/Production directory
I opened a already copied log say A.log from D:/Production (since noop=true nothing is moved) and edited it with some editor tool. This doubled the file size and save it.
At this point I think Camel is supposed to copy that particular file again since its size is modified and in my route definition I used "idempotent=true&idempotentKey=${file:name}-${file:size}&readLock=changed". But camel ignores the file.
When I use TRACE for logging it says "Skipping as file is already in progress...", but I did not find any lock file in D:/Production directory when I editted and saved the file.
I also checked that camel still ignores the file if I replace A.log (with same name but bigger size) in D:/Production directory from outside.
But I found, everything is working as expected if I remove noop=true option.
Am I missing something?
If you want monitor file changes in camel, use file-watch component.
Example -> RECURSIVE WATCH ALL EVENTS (FILE CREATION, FILE DELETION, FILE MODIFICATION):
from("file-watch://some-directory")
.log("File event: ${header.CamelFileEventType} occurred on file ${header.CamelFileName} at ${header.CamelFileLastModified}");
You can see the complete documentation here:
Camel file-watch component
I have an application that needs to load an add-on in the form of a dll. The dll needs to take its configuration information from a configuration (app.config) file. I want to dynamically find out the app.config file's name, and the way to do this, as I understand , is AppDomain.CurrentDomain.SetupInformation.ConfigurationFile
However, since it is being hosted INSIDE a parent application, the configuration file that is got from the above piece of code is (parentapplication).exe.config. I am not able to load another appdomain inside the parent application but I'd like to change the configuration file details of the appdomain. How should I be going about this to get the dll's configuration file?
OK, in the end, I managed to hack something together which works for me. Perhaps this will help;
Using the Assembly.GetExecutingAssembly, from the DLL which has the config file I want to read, I can use the .CodeBase to find where the DLL was before I launched a new AppDomain for it. The *.dll
.config is in that same folder.
Then have to convert the URI (as .CodeBase looks like "file://path/assembly.dll") to get the LocalPath for the ConfigurationManager (which doesn't like Uri formatted strings).
try
{
string assemblyName = Assembly.GetExecutingAssembly().GetName().Name;
string originalAssemblyPath = Path.GetDirectoryName(Assembly.GetExecutingAssembly().CodeBase);
Uri uri = new Uri(String.Format("{0}\\{1}.dll", originalAssemblyPath, assemblyName));
string dllPath = uri.LocalPath;
configuration = ConfigurationManager.OpenExeConfiguration(dllPath);
}
catch { }
So I want to test one of my Functions in my Web Project, but it's not actually connected to anything in the project yet (someone else is working on that part). The Function takes in an "ID" field, goes off and does some queries and gets some data, performs some calculations on it, and then writes a bunch of lines to a FileStream and returns that stream. I pretty much just want to test it by having it write the file to my own computer locally, and working with that file directory after the Function completes.
So my question is mainly:
1) How do I call this Function just for testing purposes so I can test all the queries/calculations/File writes, etc without it being connected to another part of the application just yet.
2) How can I change the 'Return fs' for the FileStream to write to my own computer locally to view the file that has been written.
Thanks guys!
To make your function testable you need to isolate all your dependencies and replace them in your test with stubs mocks. You can achieve this by wrappers around the file system classes and making sure your data layers classes have interfaces. With this your code could look like:
public class Something
{
IDataProvider provider;
IFileSystem fileSystem;
public Something(IDataProvider provider, IFileSystem fileSystem)
{
this.provider = provider;
this.fileSystem = fileSystem;
}
void DoThing(int id)
{
// make database call to get data
var data = provider.GetData(id);
fileSystem.Write("someFilePath",data);
}
}
With this you can write a test as such (in this casing using Moq like syntax):
void SomeTest()
{
var mockDataProvider = new Mock<IDataProvider>();
var mockFileSystem = new Mock<IFileSystem>();
var something = new Something(mockDataProvider.Object, mockFileSystem.Object);
var data = "someData";
mockDataProvider.Setup(x => x.GetData(5)).Return(data);
DoThing(5);
mockFileSystem.Verify(x => x.Write("someFilePath",data);
}
You need to read up on Unit Testing as this solves your problem in so many ways - it would also introduce you to dependency injection and mocking, which would be a great way to handle your problem.
Here is an overview...
Set up your class so it accepts the data-access and file-writer in the constructor. You can then pass in mock or stub version of the data access and file writer so you don't physically need to connect to a database or write to the file system to test your code.
In the "real world" you pass in the genuine data access and file writer.
In "test world" you use something such as MOQ or Rhino Mocks to create a pretend version of the data access, this means you can predict what will come back from the data access every time you test as it isn't the real database, it's some data you have prepared. You can also create a pretend file-writer that doesn't actually need to write a real file.
You can then test your class in isolation.
Dependency Injection:
http://msdn.microsoft.com/en-us/magazine/cc163739.aspx
Moq
http://code.google.com/p/moq/