I have a class similar to this one:
public static class Stats implements Serializable {
private static final long serialVersionUID = 1L;
#Persistent(serialized = "true", defaultFetchGroup="true")
private Map<String, Integer> requests;
public Stats() {
requests = new HashMap<String, Integer>();
}
}
However, Appengine complains that HashMap is not a supported property type.
How then it is possible to store a "HashMap" with Appengine? Is there any other workaround?
You might wanna take a look at EmbeddedEntity. It's not on the official Javadoc yet but will be soon. See 1.6.6 SDK pre-release announcement on AE group.
Another solution could be to serialize your map into a Blob property, e.g.
ByteArrayOutputStream out = new ByteArrayOutputStream();
ObjectOutputStream writer;
try {
writer = new ObjectOutputStream(out);
writer.writeObject(requests);
writer.close();
Blob requestsMapAsBlob = new Blob(out.toByteArray());
} catch (Exception e) {
// TODO: handle exceptions
}
You can then restore the map from that blob with readObject()
It looks like you're using JDO. See the supported JDO collection types. You could consider whether your data could be represented using one of those supported collection types instead, or you could store your HashMap as a serialized field.
Related
I am curious about how to use this User Configuration option in Flink Jobmanager UI. Is there any way that my application.conf values should be exposed via flink environment and displayed in User configuration. I did not find much documentation regarding this User Configuration online.
If someone has any Idea about it, let me know.
Thanks.
This section of the UI is populated with the GlobalJobParameters that are set via ExecutionConfig#setGlobalJobParameters.
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// assemble a map of values (e.g., from 'args', a file on the classpath or the jar manifest)
Map<String, String> data = ...
env.getConfig().setGlobalJobParameters(new MetaData(data));
// rest the job
...
}
// a trivial mapper around an existing map
private static class MetaData extends ExecutionConfig.GlobalJobParameters {
private final Map<String, String> data;
private MetaData(Map<String, String> data) {
this.data = data;
}
#Override
public Map<String, String> toMap() {
return data;
}
}
I want to use a non serializable object in stream.map() like this
stream.map { i =>
val obj = new SomeUnserializableClass()
obj.doSomething(i)
}
It is very inefficient, because I create many SomeUnserializableClass instance. Actually, it can be created only once in each worker.
In Spark, I can use mapPartition to do this. But in flink stream api, I don't known.
If you are dealing with a non serializable class what I recommend you is to create a RichFunction. In your case a RichMapFunction.
A Rich operator in Flink has a open method that is executed in the taskmanager just one time as initializer.
So the trick is to make your field transient and instantiate it in your open method.
Check below example:
public class NonSerializableFieldMapFunction extends RichMapFunction {
transient SomeUnserializableClass someUnserializableClass;
#Override
public void open(Configuration parameters) throws Exception {
super.open(parameters);
this.someUnserializableClass = new SomeUnserializableClass();
}
#Override
public Object map(Object o) throws Exception {
return someUnserializableClass.doSomething(o);
}
}
Then your code will looks like:
stream.map(new NonSerializableFieldMapFunction())
P.D: I'm using java syntax, please adapt it to scala.
I am defining certain variables in one java class and i am accessing it with a different class so as to filter the stream for unique elements. Please refer code to understand the issue better.
The problem i am facing is this Filter function doesn't work well and fails to filter unique events. I doubt the variable is shared among different threads and it is the cause!? Please suggest another method if this is not the correct way to do it. Thanks in advance.
**ClassWithVariables.java**
public static HashMap<String, ArrayList<String>> uniqueMap = new HashMap<>();
**FilterClass.java**
public boolean filter(String val) throws Exception {
if(ClassWithVariables.uniqueMap.containsKey(key)) {
Arraylist<String> al = uniqueMap.get(key);
if(al.contains(val) {
return false;
} else {
//Update the hashmap list(uniqueMap)
return true;
}
} else {
//Add to hashmap list(uniqueMap)
return true;
}
}
The correct way to de-duplicate a stream involves partitioning the stream by the key, so that all elements containing the same key will be processed by the same worker, and using flink's managed, keyed state mechanism so that the state is fault-tolerant and re-scalable. Here's a sample implementation:
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.addSource(new EventSource())
.keyBy(e -> e.key)
.flatMap(new Deduplicate())
.print();
env.execute();
}
public static class Deduplicate extends RichFlatMapFunction<Event, Event> {
ValueState<Boolean> seen;
#Override
public void open(Configuration conf) {
ValueStateDescriptor<Boolean> desc = new ValueStateDescriptor<>("seen", Types.BOOLEAN);
seen = getRuntimeContext().getState(desc);
}
#Override
public void flatMap(Event event, Collector<Event> out) throws Exception {
if (seen.value() == null) {
out.collect(event);
seen.update(true);
}
}
}
This could also be implemented as a RichFilterFunction, btw. But note that if you have an unbounded key space, the state being used will grow indefinitely until you run out of heap, or space on the disk, depending on which of Flink's state backends you choose. If this is an issue, you might want to set up a state retention policy via State Time-to-Live.
Note also that sharing state between different parts of a Flink pipeline isn't possible. You need to turn things inside-out compared to what might seem normal, and bring the event stream to the state, rather than fetching it.
Is it possible to make an application using Spring DATA with common code that supports both RDMS and Nosql(MongoDb) as back-end data store.It should support either one of them at one point of time and it should be configurable.
I have just pushed a new Spring-Data project named spring-data-gremlin which aims to do exactly this. It uses JPA annotations to map to any Tinkerpop blueprints graph database (OrientDB, TitanDB, etc). This means that switching between RDBMS and nosql graph databases should be a matter of configuration for any Spring-Data-JPA project.
Note: The project is in early stages of development and therefore not all JPA annotations are implemented yet.
I don't know for sure for MongoDB but we currently have projects configured with Spring Data JPA and Spring Data Neo4J simultaneously. I can't think of any obstacles why you could not make this work with Spring Data JPA and Spring Data MongoDB.
Be aware of transaction management: as far as I know MongoDB does not support transactionability so any kind of writing to both data sources can not be done as atom operation. If this is not an issue, you're good to go.
Our example snippet:
<neo4j:config storeDirectory="${neo4j.storeDirectory}"
base-package="app.model.neo4j" />
<neo4j:repositories base-package="app.neo4j.repo" />
<tx:annotation-driven transaction-manager="neo4jTransactionManager" />
And Spring Data JPA in Configuration annotated class:
#Configuration
#EnableJpaRepositories(value = "app.dao", entityManagerFactoryRef = "emf", transactionManagerRef = "tm")
#ComponentScan("app")
#EnableTransactionManagement
public class ConfigDao {
protected final String PROPERTY_DB_MODEL_PACKAGESTOSCAN = "db.model.packagesToScan";
protected final String PROPERTY_DB_DRIVER_CLASSNAME = "db.driver.className";
protected final String PROPERTY_DB_URL = "db.url";
protected final String PROPERTY_DB_USERNAME = "db.username";
protected final String PROPERTY_DB_PASSWORD = "db.password";
protected final String PROPERTY_DB_ADDITIONAL_DDL = "hibernate.hbm2ddl.auto";
protected final String PROPERTY_DB_ADDITIONAL_DIALECT = "hibernate.dialect";
protected final String PROPERTY_DB_ADDITIONAL_EMF_NAME = "hibernate.ejb.entitymanager_factory_name";
#Bean
public DataSource dataSource() {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName(PROPERTY_DB_DRIVER_CLASSNAME);
dataSource.setUrl(PROPERTY_DB_URL);
dataSource.setUsername(PROPERTY_DB_USERNAME);
dataSource.setPassword(PROPERTY_DB_PASSWORD);
return dataSource;
}
#Bean
public PlatformTransactionManager transactionManager() {
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setEntityManagerFactory(entityManagerFactory().getObject());
return transactionManager;
}
#Bean
public EntityManager entityManager() {
return entityManagerFactory().getObject().createEntityManager();
}
#Bean
public LocalContainerEntityManagerFactoryBean entityManagerFactory() {
LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
em.setDataSource(dataSource());
em.setPackagesToScan(PROPERTY_DB_MODEL_PACKAGESTOSCAN);
JpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
em.setJpaVendorAdapter(vendorAdapter);
em.setJpaProperties(additionalJpaProperties());
return em;
}
#Bean
protected Properties additionalJpaProperties() {
Properties properties = new Properties();
properties.setProperty(PROPERTY_DB_ADDITIONAL_DDL);
properties.setProperty(PROPERTY_DB_ADDITIONAL_DIALECT);
properties.setProperty(PROPERTY_DB_ADDITIONAL_EMF_NAME);
return properties;
}
}
Hope it helps.
I'm developing a modular app using prism in SL3, one of the modules is responsible for persisting the application settings in the isolated storage (so that when you open the app next time, you continue where you were). It works perfectly, except that I don't like the way dependencies are wired now.
I want to have a type-agnostic settings manager that has a generic store and then I add custom data from each module, some thing like this:
AppSettings["OpenForEditEmployees"] = new List<EmployeeDTO>();
AppSettings["ActiveView"] = ViewsEnum.Report;
I have implemented this part, but serialising that dictionary to xml proved to be harder than I suspected. I was wondering if there is an easy way to serialise a Dictionary<string, object> into XML.
Since you are using a Dictionary, the regular XmlSerializer won't work, you can serialize using DataContractSerializer.
These 2 static classes will handle all of your serialization/deserialization needs for string representation of xml in silverlight (and any .NET)
You will need a reference to System.Runtime.Serialization for the DataContractSerializer
public static void SerializeXml<T>(T obj, Stream strm)
{
DataContractSerializer ser = new DataContractSerializer(typeof(T));
ser.WriteObject(strm, obj);
}
public static T DeserializeXml<T>(Stream xml)
{
DataContractSerializer ser = new DataContractSerializer(typeof(T));
return (T)ser.ReadObject(xml);
}
and if you would rather use JSON, you can add a reference to the System.ServiceModel.Web assembly and use this version instead.
public static void SerializeJson<T>(T obj, Stream strm)
{
DataContractJsonSerializer ser = new DataContractJsonSerializer(typeof(T));
ser.WriteObject(strm, obj);
}
public static T DeserializeJson<T>(Stream json)
{
DataContractJsonSerializer ser = new DataContractJsonSerializer(typeof(T));
return (T)ser.ReadObject(json);
}
Have you looked at json.net
http://json.codeplex.com/
It's not XML but it does a great job with serialization.
And, works great in Silverlight.