oozie workflow and coordinator

1. oozie workflow lifecycle

 

필요한 jar 파일은 application 디렉토리밑 /lib에 포함 시키면 자동으로 실행시 포함됨

sharedlib 디렉토리에 새로운 jar파일을 올리고 싶을때 (4.1.0 이후)

   oozie-setup.sh  sharelib create -fs FS_URI [ -locallib  SHARED_LIBRARY ]

   oozie-setup.sh  sharelib create -fs hdfs://namenode01:8020  -locallib /tmp/share/lib

sharelib  상태  update 

   oozie admin -share libupdate

active sharelib 상태 체크

   oozie admin -shareliblist

   oozie admin -shareliblist pig

   oozie admin -shareliblist hive

oozie 런쳐는 mapreduce job에 대해 user code 수행시 overload 및 overwhelm을 방지하는 기능을 제공하나 job이 끝날때까지 hadoop task slot를 1개 차지 하고 있음

* fs, distcp, ssh, sub-workflow는 oozie 서버에 의해서 관리되며 oozie launcher는 사용하지 않음.

oozie action	type (동작방식)
MapReduce	Asynchronous
Java	Asynchronous
Pig	Asynchronous
Filesystem	Synchronous
Sub-Workflow	Synchronous
Hive	Asynchronous
DistCp	Asynchronous
Email	Synchronous
Shell	Asynchronous
SSH	Synchronous
Sqoop	Asynchronous

* fork and join

2. coordinator lifecycle

3. bundles

   collection of coordinator jobs

4. oozie 상태 체크

   1) oozie command line tool

      export OOZIE_URL=http://localhost:11000/oozie

      bin/oozie admin -oozie http://localhost:11000/oozie -status

oozie jobs

oozie info

oozie admin

oozie admin -queuedump

      oozie job -info 0000001-14234234123423-oozie-test-W

      oozie job –suspend 0000001-14234234123423-oozie-test-W

      oozie job –resume 0000001-14234234123423-oozie-test-W

      oozie job –kill 0000001-14234234123423-oozie-test-W

      oozie job -change

      workflow job list 출력

      oozie jobs 

      oozie jobs -jobtype coordinator

      oozie jobs -jobtype=bundle

      oozie jobs -len 20 -filter status=RUNNING

      oozie jobs -bulk bundle=test

      

   2) oozie http rest api  (json-based and utf8)

      oozie client 명령어에서 debug 옵션을 붙이면 http rest api 호출하는 정확한 방법이 나옴

      oozie job -info 0000001-14234234123423-oozie-test-W -debug

      GET http://localhost:11000/oozie/v2/job/0000001-14234234123423-oozie-test-W?show=info

      /versions

      /v2/admin

      /v2/job

      /v2/jobs

      /v2/sla

      curl "http://localhost:11000/oozie/versions"

      curl "http://localhost:11000/oozie/v2/admin/status"  | python -m json.tool

      (json.tool를 사용하면 사용자가 보기 좋게 json를 표시함)

3) oozie java client api

   import java.util.Properties;

   import org.apache.oozie.client.OozieClient;

   import org.apache.oozie.client.WorkflowJob;

   import org.apache.oozie.client.WorkflowAction;

    OozieClient  test =   new OozieClient(“http://t:11000/oozie");

    Properties conf = test.createConfiguration();

    conf.setProperty(OozieClient.APP_PATH, "hdfs://nn:8020/user/test/test-app");

    conf.setProperty("jobTracker", "test:8032");

    conf.setProperty("inputDir", "/user/joe/input");

    conf.setProperty("outputDir", "/user/joe/output");

    String jobId = test.run(conf);

   

    while (test.getJobInfo(jobId).getStatus() == Workflow.Status.RUNNING) {

              System.out.println("Workflow job running ...");

              Thread.sleep(30 * 1000);

    }

 

    System.out.println("Workflow job completed ...");

    System.out.println(test.getJobInfo(jobId));

    커버로스 설정시 아래 spnego를 사용하여 설정함

    AuthOozieClient myOozieClient = new

    AuthOozieClient(“http://t:11000/oozie");