I am using akka-grpc to generate client bindings. They usually have the form of
func[A, B](in: Source[A]) : Source[B],
i.e. they consume a Source[A] and offer a Source[B].
Now, I want to turn func into a Flow[A, B] to use them with akka-stream.
The solution is:
def SourceProcessor[In, Out](f : Source[In, NotUsed] => Source[Out, NotUsed]): Flow[In, Out, NotUsed] =
Flow[In].prefixAndTail(0).flatMapConcat { case (Nil, in) => f(in) }
It uses prefixAndTail to highjack the underyling Source.
Related
I want to create a sink in akka streams which is made up of many operations.
e.g map, filter, fold and then sink.
The best I can do at the moment is the following.
I don't like it because I have to specify broadcast even though I am only letting a single value through.
Does anyone know a better way of doing this?
def kafkaSink(): Sink[PartialBatchProcessedResult, NotUsed] = {
Sink.fromGraph(GraphDSL.create() { implicit b =>
import GraphDSL.Implicits._
val broadcast = b.add(Broadcast[PartialBatchProcessedResult](1))
broadcast.out(0)
.fold(new BatchPublishingResponseCollator()) { (c, e) => c.consume(e) }
.map(_.build())
.map(a =>
FunctionalTesterResults(sampleProjectorConfig, 0, a)) ~> Sink.foreach(new KafkaTestResultsReporter().report)
SinkShape(broadcast.in)
})
}
One key point to remember with akka-stream is that any number of Flow values plus a Sink value is still a Sink.
A couple of examples demonstrating this property:
val intSink : Sink[Int, _] = Sink.head[Int]
val anotherSink : Sink[Int, _] =
Flow[Int].filter(_ > 0)
.to(intSink)
val oneMoreSink : Sink[Int, _] =
Flow[Int].filter(_ > 0)
.map(_ + 4)
.to(intSink)
Therefore, you can implement the map and filter as Flows. The fold that you are asking about can be implemented with Sink.fold.
I have 2 flows like the following:
val aToSeqOfB: Flow[A, Seq[B], NotUsed] = ...
val bToC: Flow[B, C, NotUsed] = ...
I want to combine these into a convenience method like the following:
val aToSeqOfC: Flow[A, Seq[C], NotUsed]
So far I have the following, but I know it just ends up with C elements and not Seq[C].
Flow[A].via(aToSeqOfB).mapConcat(_.toList).via(bToC)
How can I preserve the Seq in this scenario?
Indirect Answer
In my opinion your question highlights one of the "rookie mistakes" that is common when dealing with akka streams. It is usually not good organization to put business logic within akka stream constructs. Your question indicates that you have something of the form:
val bToC : Flow[B, C, NotUsed] = Flow[B] map { b : B =>
//business logic
}
The more ideal scenario would be if you had:
//normal function, no akka involved
val bToCFunc : B => C = { b : B =>
//business logic
}
val bToCFlow : Flow[B,C,NotUsed] = Flow[B] map bToCFunc
In the above "ideal" example the Flow is just a thin veneer on top of normal, non-akka, business logic.
The separate logic can then simply solve your original question with:
val aToSeqOfC : Flow[A, Seq[C], NotUsed] =
aToSeqOfB via (Flow[Seq[B]] map (_ map bToCFunc))
Direct Answer
If you cannot reorganize your code then the only available option is to deal with Futures. You'll need to use bToC within a separate sub-stream:
val mat : akka.stream.Materializer = ???
val seqBToSeqC : Seq[B] => Future[Seq[C]] =
(seqB) =>
Source
.apply(seqB.toIterable)
.via(bToC)
.to(Sink.seq[C])
.run()
You can then use this function within a mapAsync to construct the Flow you are looking for:
val parallelism = 10
val aToSeqOfC: Flow[A, Seq[C], NotUsed] =
aToSeqB.mapAsync(parallelism)(seqBtoSeqC)
I know there are more elaborate ways to achieve this in Java, but Groovy should have a concise way to do the same as per http://groovy.codehaus.org/Looping
Class Currency.groovy
class Currency {
String name
double rate
}
CurrencyController
def select(){
List<Currency> selectedCurrencies = Currency.getAll(params.currencies)
selectedCurrencies.eachWithIndex { obj, i -> obj.rate = update(obj.name)};
[selectedCurrencies:selectedCurrencies]
}
def update(String sym){
return sym
}
The above code throws:
No signature of method: currencychecker.CurrencyController$_$tt__select_closure12.doCall() is applicable for argument types: (currencychecker.Currency)
Thanks to #dmahapatro, the issue was that I was using an iterator variable obj[i], even though obj itself is the iterated object. The rest is correct!
I experimented with selectCurrencies.each as well instead of selectCurrencies.eachWithIndex however the right one in this case is eachWithIndex
I have an array of maps and the maps and I would like to find the maximum date in the array of maps and I think I'm heading down a non-scala path because I'm not sure how to wire the pieces of this question together.
Is there a better way of doing this? I'm concerned that I need to assume things like casting the value to a Date for comparison, but that is what's in the Map and the map includes other data types also (so Map[String, Object] is what I have)
val df = new SimpleDateFormat("yyyy-MM-dd")
def omap = List(Map("date" -> df.parse("2013-08-01")), Map("date" -> df.parse("2013-02-01"), "otherkey" -> "nothing special"), Map("date" -> df.parse("2013-01-01")))
omap.max(new Ordering[Map[String, Object]] {
def compare(x: Map[String, Object], y: Map[String, Object]) = x.get("date").get.asInstanceOf[Date] compareTo y.get("date").get.asInstanceOf[Date]
})
The code seems to work, but I feel like I'm missing a more scala like way of doing this.
This little one liner works, but it will throw an exception if there is no date in any map:
omap.flatMap(map => map.get("date").collect({case d:Date => d})).max
Here's a safer version, but you have to provide a default date:
val defaultDate = new Date()
omap.map(map => map.get("date").collect({case d:Date => d}))
.foldLeft(defaultDate)((default, od) => od.fold(default)( d => if (d.compareTo(default) > 0) d else default))
If you have a minimal number of types, you should probably make them explicit with something like Either. But I'll assume you have too many for that.
Then, the more canonical way to get your value is something like this (to replace your omap.max):
val defaultDate = df.parse("1970-01-01")
def dateFrom(m: Map[String, Object]) =
m.get("date").collect{ case d: Date => d }.getOrElse(defaultDate)
omap.maxBy(dateFrom)
This protects you from missing entries by grouping everything missing into the earliest possible date.
If you don't have any sensible default date, then
def dateFrom(m: Map[String, Object]) = m.get("date").collect{ case d: Date => d }
omap.max(new Ordering[Map[String, Object]] {
def compare(x: Map[String, Object], y: Map[String, Object]) = {
(dateFrom(x), dateFrom(y)) match {
case (Some(a), Some(b)) => a compareTo b
case (_, None) => -1
case _ => 1
}
}
})
to shuffle all dateless maps to the end.
omap.maxBy{ _("date").asInstanceOf[Date] }
I'm struggling with Slick's lifted embedding and mapped tables. The API feels strange to me, maybe just because it is structured in a way that's unfamiliar to me.
I want to build a Task/Todo-List. There are two entities:
Task: Each task has a an optional reference to the next task. That way a linked list is build. The intention is that the user can order the tasks by his priority. This order is represented by the references from task to task.
TaskList: Represents a TaskList with a label and a reference to the first Task of the list.
case class Task(id: Option[Long], title: String, nextTask: Option[Task])
case class TaskList(label: String, firstTask: Option[Task])
Now I tried to write a data access object (DAO) for these two entities.
import scala.slick.driver.H2Driver.simple._
import slick.lifted.MappedTypeMapper
implicit val session: Session = Database.threadLocalSession
val queryById = Tasks.createFinderBy( t => t.id )
def task(id: Long): Option[Task] = queryById(id).firstOption
private object Tasks extends Table[Task]("TASKS") {
def id = column[Long]("ID", O.PrimaryKey, O.AutoInc)
def title = column[String]("TITLE")
def nextTaskId = column[Option[Long]]("NEXT_TASK_ID")
def nextTask = foreignKey("NEXT_TASK_FK", nextTaskId, Tasks)(_.id)
def * = id ~ title ~ nextTask <> (Task, Task.unapply _)
}
private object TaskLists extends Table[TaskList]("TASKLISTS") {
def label = column[String]("LABEL", O.PrimaryKey)
def firstTaskId = column[Option[Long]]("FIRST_TASK_ID")
def firstTask = foreignKey("FIRST_TASK_FK", firstTaskId, Tasks)(_.id)
def * = label ~ firstTask <> (Task, Task.unapply _)
}
Unfortunately it does not compile. The problems are in the * projection of both tables at nextTask respective firstTask.
could not find implicit value for evidence parameter of type
scala.slick.lifted.TypeMapper[scala.slick.lifted.ForeignKeyQuery[SlickTaskRepository.this.Tasks.type,justf0rfun.bookmark.model.Task]]
could not find implicit value for evidence parameter of type scala.slick.lifted.TypeMapper[scala.slick.lifted.ForeignKeyQuery[SlickTaskRepository.this.Tasks.type,justf0rfun.bookmark.model.Task]]
I tried to solve that with the following TypeMapper but that does not compile, too.
implicit val taskMapper = MappedTypeMapper.base[Option[Long], Option[Task]](
option => option match {
case Some(id) => task(id)
case _ => None
},
option => option match {
case Some(task) => task.id
case _ => None
})
could not find implicit value for parameter tm: scala.slick.lifted.TypeMapper[Option[justf0rfun.bookmark.model.Task]]
not enough arguments for method base: (implicit tm: scala.slick.lifted.TypeMapper[Option[justf0rfun.bookmark.model.Task]])scala.slick.lifted.BaseTypeMapper[Option[Long]]. Unspecified value parameter tm.
Main question: How to use Slick's lifted embedding and mapped tables the right way? How to I get this to work?
Thanks in advance.
The short answer is: Use ids instead of object references and use Slick queries to dereference ids. You can put the queries into methods for re-use.
That would make your case classes look like this:
case class Task(id: Option[Long], title: String, nextTaskId: Option[Long])
case class TaskList(label: String, firstTaskId: Option[Long])
I'll publish an article about this topic at some point and link it here.