Apache Flink is an open source system for fast and versatile data analytics in clusters. Flink supports batch and streaming analytics,
in one system. Analytical programs can be written in concise and elegant APIs in Java and Scala.
Apache Flink is an open source system for expressive, declarative, fast, and efficient data analysis. Flink combines the scalability and programming flexibility of distributed MapReduce-like platforms with the efficiency, out-of-core execution, and query optimization capabilities found in parallel databases.
```scala
caseclassWordWithCount(word:String,count:Int)
valtext=env.readTextFile(path)
Learn more about Flink at http://flink.incubator.apache.org/
valcounts=text.flatMap{_.split("\\W+")}
.map{WordWithCount(_,1)}
.groupBy("word")
.sum("count")
counts.writeAsCsv(outputPath)
```
Flink is highlighted by some unique features:
* Hybrid batch/streaming runtime that supports batch processing and data streaming programs.
* Custom memory management to guarantee efficient, adaptive, and highly robust switching between in-memory and data processing out-of-core algorithms.
* Flexible and expressive windowing semantics for data stream programs
* Built-in program optimizer that chooses the proper runtime operations for each program
* Custom type analysis and serialization stack for high performance
## Build Apache Flink
Learn more about Flink at http://flink.incubator.apache.org/
### Build From Source
## Building Apache Flink from Source
#### Requirements
* Unix-like environment (We use Linux, Mac OS X, Cygwin)
* git
* Maven (at least version 3.0.4)
...
...
@@ -27,8 +44,46 @@ mvn clean package -DskipTests # this will take up to 5 minutes
Flink is now installed in `flink-dist/target`
## Developing Flink
The Flink committers use the IntelliJ IDE and Eclipse to develop the Flink codebase.
Minimal requirements for an IDE are:
* Support for Java and Scala (also mixed projects)
* Support for Maven with Java and Scala
### IntelliJ
The IntelliJ IDE supports Maven out of the box and offers a plugin for Scala development.
We recommend using Scala IDE 3.0.3, based on Eclipse Kepler. While this is a slightly older version,
we found it to be the verstion that works most robustly for a complex project like Flink.
Further details, and a guide to newer Scala IDE versions can be found in the
[How to setup Eclipse](https://github.com/StephanEwen/incubator-flink/blob/master/docs/internal_setup_eclipse.md) docs.
**Note:** Before following this setup, make sure to run the build from the command line once
(`mvn clean package -DskipTests`, see above)
1. Download the Scala IDE (preferred) or install the plugin to Eclipse Kepler. See [here]([How to setup Eclipse](apache/incubator-flink/blob/master/docs/internal_setup_eclipse.md)
for download links and instructions.
2. Add the "macroparadise" compiler plugin to the Scala compiler.
Open "Window" -> "Preferences" -> "Scala" -> "Compiler" -> "Advanced" and put into the "Xplugin" field the path to
the *macroparadise* jar file (typically "/home/*-your-user-*/.m2/repository/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar").
Note: If you do not have the jar file, you probably did not ran the command line build.
4. During the import, Eclipse will ask to automatically install additional Maven build helper plugins.
5. Close the "flink-java8" project. Since Eclipse Kepler does not support Java 8, you cannot develop this project.
## Support
Don’t hesitate to ask!
Please contact the developers on our [mailing lists](http://flink.incubator.apache.org/community.html#mailing-lists) if you need help.
...
...
@@ -38,7 +93,8 @@ Please contact the developers on our [mailing lists](http://flink.incubator.apac
## Documentation
The documentation of Apache Flink is located on the website: http://flink.incubator.apache.org or in the `docs/` directory of the source code.
The documentation of Apache Flink is located on the website: [http://flink.incubator.apache.org](http://flink.incubator.apache.org)
or in the `docs/` directory of the source code.
## Fork and Contribute
...
...
@@ -50,6 +106,5 @@ This article describes [how to contribute to Apache Flink](http://flink.incubato
## About
Apache Flink is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
Apache Flink is an open source project of The Apache Software Foundation (ASF).
The Apache Flink project originated from the [Stratosphere](http://stratosphere.eu) research project.
Alternatively, you can download Eclipse Kepler from [https://eclipse.org/downloads/packages/release/Kepler/SR2](https://eclipse.org/downloads/packages/release/Kepler/SR2)
and manually add the Scala and Maven plugins by plugin site at [http://scala-ide.org/download/prev-stable.html](http://scala-ide.org/download/prev-stable.html).
* Either use the update site to install the plugin ("Help" -> "Install new Software")
* Or download the [zip file](http://download.scala-ide.org/sdk/helium/e38/scala211/stable/update-site.zip), unpack it, and move the contents of the
"plugins" and "features" folders into the equally named folders of the Eclipse root directory
## Eclipse Scala IDE 4.0.0
**NOTE: From personal experience, the use of the Scala IDE 4.0.0 performs worse than previous versions for complex projects like Flink.**
**Version 4.0.0 does not handle mixed Java/Scala projects as robustly and it frequently raises incorrect import and type errors.**
*Note:* Before following this setup, make sure to run the build from the command line once
(`mvn clean package -DskipTests`)
1. Download the Scala IDE: [http://scala-ide.org/download/sdk.html](http://scala-ide.org/download/sdk.html)
3. While importing the Flink project, the IDE may ask you to install an additional maven build helper plugin.
4. After the import, you need to set the Scala version of your projects to Scala 2.10 (from the default 2.11).
To do that, select all projects that contain Scala code (marked by the small *S* on the project icon),
right click and select "Scala -> Set the Scala Installation" and pick "2.10.4".
Currently, the project to which that is relevant are "flink-runtime", "flink-scala", "flink-scala-examples",
"flink-streaming-example", "flink-streaming-scala", "flink-tests", "flink-test-utils", and "flink-yarn".
5. Depending on your version of the Scala IDE, you may need to add the "macroparadise" compiler plugin to the
Scala compiler. Open "Window" -> "Preferences" -> "Scala" -> "Compiler" -> "Advanced" and put into the "Xplugin" field
the path to the *macroparadise* jar file (typically "/home/*-your-user-*/.m2/repository/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar")
6. In order to compile the "flink-java-8" project, you may need to add a Java 8 execution environment.
See [this post](http://stackoverflow.com/questions/25391207/how-do-i-add-execution-environment-1-8-to-eclipse-luna)
for details.
---
*This documentation is maintained by the contributors of the individual components.
We kindly ask anyone that adds and changes components to eventually provide a patch
or pull request that updates these documents as well.*
