Masa's blog

How to use tcpdump and rtmpdump to download NHK for School ;P

_ First, you need a computer that connect to Internet directly and start tcpdump to capture network packets by doing something like:

tcpdump -s 0 -i eth0 -v -w capture.cap

And then, start browser(firefox etc) and play movie in NHK for School. After few minutes, stop tcpdump with pressing CTRL-C and convert capture data to text format by doing something like:

tcpdump -s 0 -v -XX -r capture.cap >capture.cap.txt

And see capture.cap.txt and determine some parameters for rtmpdump.

$ less capture.cap.txt

22:57:35.268804 IP foo.bar.ne.jp.57403 > xx.xx.xx.xx.http: P 3576:3681(105) ack 3460 win 273 <nop,nop,timestamp 1603797886 2111353824>
        0x0000:  0017 1088 75cf 000a e4be cc43 0800 4500  ....u......C..E.
        0x0010:  009d 4eb3 4000 4006 ba7c 7ac5 c22e ca20  ..N.@.@..|z.....
        0x0020:  2a17 e03b 0050 f687 5a36 fc3b 5a6f 8018  *..;.P..Z6.;Zo..
        0x0030:  0111 60e1 0000 0101 080a 5f98 037e 7dd8  ..`......._..~}.
        0x0040:  b3e0 0800 010d 0000 5214 0100 0000 0200  ........R.......
        0x0050:  0470 6c61 7900 0000 0000 0000 0000 0502  .play...........
        0x0060:  002c 6d70 343a 6461 732f 4430 3030 3534  .,mp4:das/D00054    <- string for '-y' parameter
        0x0070:  3032 2f44 3030 3031 3233 3435 3637 5f30  02/D0001234567_0    <-
        0x0080:  3030 3030 5f56 5f30 3030 2e66 3476 0000  0000_V_000.f4v..    <-
        0x0090:  0000 0000 0000 0000 c08f 4000 0000 0000  ..........@.....
        0x00a0:  c200 0300 0000 0100 004e 20              .........N.

... snip ...

22:57:53.628365 IP foo.bar.ne.jp.41867 > xx.xx.xx.xx.1935: P 2986:3508(522) ack 2897 win 137 <nop,nop,timestamp 1603816246 2111372216>
        0x0000:  0017 1088 75cf 000a e4be cc43 0800 4500  ....u......C..E.
        0x0010:  023e eb26 4000 4006 1c68 7ac5 c22e ca20  .>.&@.@..hz.....
        0x0020:  2a17 a38b 078f 9292 ada1 ff2c 1bc8 8018  *..........,....
        0x0030:  0089 375c 0000 0101 080a 5f98 4b36 7dd8  ..7\......_.K6}.
        0x0040:  fbb8 8444 ac8e f8dc 402f cc3e 90c1 146c  ...D....@/.>...l
        0x0050:  c46b 34cb 5983 bbe2 8036 e94c c844 3597  .k4.Y....6.L.D5.
        0x0060:  ed11 b8aa 66b7 2bfe 7cab 4b10 4aa6 64a5  ....f.+.|.K.J.d.
        0x0070:  2746 26f7 9086 9bd9 bec0 add2 ac11 74f3  'F&...........t.
        0x0080:  981c 52dc 8528 e7b3 ff81 1890 1642 ba3c  ..R..(.......B.<
        0x0090:  3ffa 6698 a0f8 8a89 ce3a 0300 0000 0001  ?.f......:......
        0x00a0:  a314 0000 0000 0200 0763 6f6e 6e65 6374  .........connect
        0x00b0:  003f f000 0000 0000 0003 0003 6170 7002  .?..........app.
        0x00c0:  000e 666c 7639 2f5f 6465 6669 6e73 745f  ..flv9/_definst_
        0x00d0:  0008 666c 6173 6856 6572 0200 104c 4e58  ..flashVer...LNX
        0x00e0:  2031 312c 322c 3230 322c 3530 3800 0673  .11,2,202,508..s
        0x00f0:  7766 5572 6c02 004c 6874 7470 3a2f 2f77  wfUrl..Lhttp://w
        0x0100:  7777 2e6e 686b 2e6f 722e 6a70 2f73 6368  ww.nhk.or.jp/sch
        0x0110:  6f6f 6c2f 7061 7274 7332 3031 352f 706c  ool/parts2015/pl
        0x0120:  6179 6572 2f73 c377 662f 706c 6179 6572  ayer/s.wf/player
        0x0130:  2e73 7766 3f6e 633d 3134 3431 3839 3332  .swf?nc=14418932
        0x0140:  3437 3736 3300 0574 6355 726c 0200 2472  47763..tcUrl..$r    <- string for '-r' and '-a' parameter
        0x0150:  746d 703a 2f2f 666c 7639 2e6e 686b 2e6f  tmp://flv9.nhk.o    <-
        0x0160:  722e 6a70 2f66 6c76 392f 5f64 6566 696e  r.jp/flv9/_defin    <-
        0x0170:  7374 5f00 0466 7061 6401 0000 0c63 6170  st_..fpad....cap    <-
        0x0180:  6162 696c 6974 6965 7300 406d e000 0000  abilities.@m....
        0x0190:  0000 000b 6175 6469 6f43 6f64 6563 7300  ....audioCodecs.
        0x01a0:  40ab ee00 0000 00c3 0000 0b76 6964 656f  @..........video
        0x01b0:  436f 6465 6373 0040 6f80 0000 0000 0000  Codecs.@o.......
        0x01c0:  0d76 6964 656f 4675 6e63 7469 6f6e 003f  .videoFunction.?
        0x01d0:  f000 0000 0000 0000 0770 6167 6555 726c  .........pageUrl
        0x01e0:  0200 4a68 7474 703a 2f2f 7777 7732 2e6e  ..Jhttp://www2.n
        0x01f0:  686b 2e6f 722e 6a70 2f73 6368 6f6f 6c2f  hk.or.jp/school/
        0x0200:  6d6f 7669 652f 636c 6970 2e63 6769 3f64  movie/clip.cgi?d
        0x0210:  6173 5f69 643d 4430 3030 3132 3334 3536  as_id=D000123456
        0x0220:  375f 3030 3030 3026 c370 3d62 6f78 000e  7_00000&.p=box..
        0x0230:  6f62 6a65 6374 456e 636f 6469 6e67 0000  objectEncoding..
        0x0240:  0000 0000 0000 0000 0009 0100            ............

And now, you can download streaming movie from NHK for School into file.flv by doing something like:

rtmpdump -r rtmp://flv9.nhk.or.jp/ -y mp4:das/D0005402/D0001234567_00000_V_000.f4v -a flv9/_definst_ -o file.flv

Good luck ;)

get_nhk_for_school.sh(script to download nhk for school)

#! /bin/sh
#
#  1. tcpdump -s 0 -i eth0 -v -w capture.cap
#  2. browse movie that you want
#  3. stop tcpdump with CTRL-C
#  4. ./get_nhk_for_school.sh capture.cap
#
function funcFindUrl (){
        strings $1 |\
        egrep -1 "^play$|^tcUrl$" |\
        awk 'BEGIN{
                tcUrlSw = 0;
                tcUrl = "";
                playSw = 0;
        }
        {
                if ($0 == "tcUrl"){
                        tcUrlSw = 1;
                }else if($0 == "play"){
                        playSw = 1;
                }else{
                        if (tcUrlSw == 1){
                                gsub(/^./,"");
                                tcUrl = $0;
                                tcUrlSw = 0;
                        }else if (playSw == 1){
                                gsub(/^./,"");
                                if (tcUrl != ""){
                                        printf("%-s/%-s\n",tcUrl,$0);
                                }
                                playSw = 0;
                        }
                }
        }' |\
        sort |\
        uniq
}

for i in `funcFindUrl $1`
do
        echo == $i ==
        output=`basename $i`
        rtmpdump -r "$i" -o ${output}
done