The Flink Job Lifecycle

Overview

When a Flink job is executed, it is sent to the Flink cluster where it will pass through multiple possible stages in its lifecycle. The exact flow depends on what commands are sent to the job, and whether or not it encounters any errors. This video will introduce the lifecycle of a Flink job as well as some of the commands that can be used to move the job through different lifecycle states.

Topics:

• Running a Job
• Finishing a Job
• Canceling a Job
• Stopping a Job
• Resuming a Job
• Failing Jobs
• Recovery Strategies

Code

A Simple Job

public static void main(String[] args) throws Exception {
	StreamExecutionEnvironment env = StreamExecutionEnvironment
		.getExecutionEnvironment();

	env.fromElements(1,2,3,4,5).print();

	env.execute();
}

Running a Job

$ flink run $JAR_FILE

$ flink run -c mypackage.MyClass $JAR_FILE

$ flink run --detached $JAR_FILE

Canceling a Job

$ flink cancel $JOB_ID

Stopping a Job

$ flink stop --savepointPath $SAVEPOINT_FOLDER $JOB_ID

Resuming a Job

$ flink run --fromSavepoint $SAVEPOINT_FILE $JAR_FILE

Setting a Restart Strategy

env.setRestartStrategy(RestartStrategies.fixedDelayRestart(
        3, // number of restart attempts
        Time.of(10, TimeUnit.SECONDS) // delay
));

Resources

• Flink Job Scheduling
• Flink Command Line Interface