

# Scala API Extensions

In order to keep a fair amount of consistency between the Scala and Java APIs, some of the features that allow a high-level of expressiveness in Scala have been left out from the standard APIs for both batch and streaming.

If you want to _enjoy the full Scala experience_ you can choose to opt-in to extensions that enhance the Scala API via implicit conversions.

To use all the available extensions, you can just add a simple `import` for the DataSet API



```
import org.apache.flink.api.scala.extensions._
```



or the DataStream API



```
import org.apache.flink.streaming.api.scala.extensions._
```



Alternatively, you can import individual extensions _a-là-carte_ to only use those you prefer.

## Accept partial functions

Normally, both the DataSet and DataStream APIs don’t accept anonymous pattern matching functions to deconstruct tuples, case classes or collections, like the following:



```
val data: DataSet[(Int, String, Double)] = // [...] data.map {
  case (id, name, temperature) => // [...]
  // The previous line causes the following compilation error:
  // "The argument types of an anonymous function must be fully known. (SLS 8.5)" }
```



This extension introduces new methods in both the DataSet and DataStream Scala API that have a one-to-one correspondence in the extended API. These delegating methods do support anonymous pattern matching functions.

#### DataSet API

| Method | Original | Example |
| --- | --- | --- |
| **mapWith** | **map (DataSet)** | 



```
data.mapWith {
  case (_, value) =&gt; value.toString
}
```



 |
| **mapPartitionWith** | **mapPartition (DataSet)** | 



```
data.mapPartitionWith {
  case head #:: _ =&gt; head
}
```



 |
| **flatMapWith** | **flatMap (DataSet)** | 



```
data.flatMapWith {
  case (_, name, visitTimes) =&gt; visitTimes.map(name -&gt; _)
}
```



 |
| **filterWith** | **filter (DataSet)** | 



```
data.filterWith {
  case Train(_, isOnTime) =&gt; isOnTime
}
```



 |
| **reduceWith** | **reduce (DataSet, GroupedDataSet)** | 



```
data.reduceWith {
  case ((_, amount1), (_, amount2)) =&gt; amount1 + amount2
}
```



 |
| **reduceGroupWith** | **reduceGroup (GroupedDataSet)** | 



```
data.reduceGroupWith {
  case id #:: value #:: _ =&gt; id -&gt; value
}
```



 |
| **groupingBy** | **groupBy (DataSet)** | 



```
data.groupingBy {
  case (id, _, _) =&gt; id
}
```



 |
| **sortGroupWith** | **sortGroup (GroupedDataSet)** | 



```
grouped.sortGroupWith(Order.ASCENDING) {
  case House(_, value) =&gt; value
}
```



 |
| **combineGroupWith** | **combineGroup (GroupedDataSet)** | 



```
grouped.combineGroupWith {
  case header #:: amounts =&gt; amounts.sum
}
```



 |
| **projecting** | **apply (JoinDataSet, CrossDataSet)** | 



```
data1.join(data2).
  whereClause(case (pk, _) =&gt; pk).
  isEqualTo(case (_, fk) =&gt; fk).
  projecting {
    case ((pk, tx), (products, fk)) =&gt; tx -&gt; products
  }

data1.cross(data2).projecting {
  case ((a, _), (_, b) =&gt; a -&gt; b
}
```



 |
| **projecting** | **apply (CoGroupDataSet)** | 



```
data1.coGroup(data2).
  whereClause(case (pk, _) =&gt; pk).
  isEqualTo(case (_, fk) =&gt; fk).
  projecting {
    case (head1 #:: _, head2 #:: _) =&gt; head1 -&gt; head2
  }
}
```



 |

#### DataStream API

| Method | Original | Example |
| --- | --- | --- |
| **mapWith** | **map (DataStream)** | 



```
data.mapWith {
  case (_, value) =&gt; value.toString
}
```



 |
| **mapPartitionWith** | **mapPartition (DataStream)** | 



```
data.mapPartitionWith {
  case head #:: _ =&gt; head
}
```



 |
| **flatMapWith** | **flatMap (DataStream)** | 



```
data.flatMapWith {
  case (_, name, visits) =&gt; visits.map(name -&gt; _)
}
```



 |
| **filterWith** | **filter (DataStream)** | 



```
data.filterWith {
  case Train(_, isOnTime) =&gt; isOnTime
}
```



 |
| **keyingBy** | **keyBy (DataStream)** | 



```
data.keyingBy {
  case (id, _, _) =&gt; id
}
```



 |
| **mapWith** | **map (ConnectedDataStream)** | 



```
data.mapWith(
  map1 = case (_, value) =&gt; value.toString,
  map2 = case (_, _, value, _) =&gt; value + 1
)
```



 |
| **flatMapWith** | **flatMap (ConnectedDataStream)** | 



```
data.flatMapWith(
  flatMap1 = case (_, json) =&gt; parse(json),
  flatMap2 = case (_, _, json, _) =&gt; parse(json)
)
```



 |
| **keyingBy** | **keyBy (ConnectedDataStream)** | 



```
data.keyingBy(
  key1 = case (_, timestamp) =&gt; timestamp,
  key2 = case (id, _, _) =&gt; id
)
```



 |
| **reduceWith** | **reduce (KeyedStream, WindowedStream)** | 



```
data.reduceWith {
  case ((_, sum1), (_, sum2) =&gt; sum1 + sum2
}
```



 |
| **foldWith** | **fold (KeyedStream, WindowedStream)** | 



```
data.foldWith(User(bought = 0)) {
  case (User(b), (_, items)) =&gt; User(b + items.size)
}
```



 |
| **applyWith** | **apply (WindowedStream)** | 



```
data.applyWith(0)(
  foldFunction = case (sum, amount) =&gt; sum + amount
  windowFunction = case (k, w, sum) =&gt; // [...] )
```



 |
| **projecting** | **apply (JoinedStream)** | 



```
data1.join(data2).
  whereClause(case (pk, _) =&gt; pk).
  isEqualTo(case (_, fk) =&gt; fk).
  projecting {
    case ((pk, tx), (products, fk)) =&gt; tx -&gt; products
  }
```



 |

For more information on the semantics of each method, please refer to the [DataSet](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/batch/index.html) and [DataStream](//ci.apache.org/projects/flink/flink-docs-release-1.7/dev/datastream_api.html) API documentation.

To use this extension exclusively, you can add the following `import`:



```
import org.apache.flink.api.scala.extensions.acceptPartialFunctions
```



for the DataSet extensions and



```
import org.apache.flink.streaming.api.scala.extensions.acceptPartialFunctions
```



The following snippet shows a minimal example of how to use these extension methods together (with the DataSet API):



```
object Main {
  import org.apache.flink.api.scala.extensions._
  case class Point(x: Double, y: Double)
  def main(args: Array[String]): Unit = {
    val env = ExecutionEnvironment.getExecutionEnvironment
    val ds = env.fromElements(Point(1, 2), Point(3, 4), Point(5, 6))
    ds.filterWith {
      case Point(x, _) => x > 1
    }.reduceWith {
      case (Point(x1, y1), (Point(x2, y2))) => Point(x1 + y1, x2 + y2)
    }.mapWith {
      case Point(x, y) => (x, y)
    }.flatMapWith {
      case (x, y) => Seq("x" -> x, "y" -> y)
    }.groupingBy {
      case (id, value) => id
    }
  }
}
```



