so i had to find a new host for the podcast thanks to dreamhost being complete dicks and terminating my entire account. (sigh)

so i decided that since the podcast was unavailable for a few days (hope no one noticed), it would be a good time to analyze day 1 web traffic in a fun and interesting way.

all web servers log requests to a file. each request is a single line in the logfile that looks something like this

127.0.0.1 - - [04/Jun/2014:21:00:48 -0700] "GET /ARCHIVE_-_0000-00-00_-_Show_Name_With_DJ_Name_From_Location.mp3 HTTP/1.1" 200 140398275 "-" "iTunes/11.2.1 (Macintosh; OS X 10.9.3) AppleWebKit/537.75.14"


there is a log entry for every whole or partial content request. every time you visit a website, for every image you see, for every js, css, etc file that loads is an entry. on busy sites this log file can become huge. the log file i used for this demonstration has 22,577 lines. each line (see above) can tell you a few things:

the ip address of the person making the request
time stamp
request type and request
http response code
bytes transferred
user agent

unfortunately, the new hosting provider only allows access to the raw apache logs for the current day. i learned this while emailing support for a few hours. apparently they rotate the logs at 9pm pst every day… and guess what, its not really rotated (sigh). once 9pm rolls around, the log file is deleted/emptied. so theres only 1 log file. no historical data whatsoever. i guess since i configured the vhost after 9pm on june 3rd, thus never creating a file in my home directory. so as you can see, i missed out. the data that i REALLY, REALLY wanted to have, which was the FIRST 24 hours, 86’d.

ok.. so now.. set a crontab to grab the file just before they delete it, otherwise you loose, and its gone forever.

55 20 * * * scp user@host:~/logs/podcast.access.log /var/log/apache2/podcast.access.log >> /dev/null 2>&1

and now, with a wonderful tool called logstalgia combined with ffmpeg and some command line-fu

cat /var/log/apache2/podcast.access.log | logstalgia -1920x1080 -g Archives,ARCHIVE,99 --paddle-mode pid --update-rate 1 --output-framerate 60 --output-ppm-stream - - | ffmpeg -f image2pipe -r 60 -c:v ppm -s 1920x1080 -pix_fmt yuv420p -i - -crf 1 -c:v h264 -pix_fmt yuv420p -f mp4 outfile.mp4

and if you want real time…

tail -F -q /var/log/apache2/podcast.access.log | logstalgia -1920x1080 -g Archives,ARCHIVE,99 --paddle-mode pid --update-rate 1 --output-framerate 60 --output-ppm-stream - - | ffmpeg -f image2pipe -r 60 -c:v ppm -s 1920x1080 -pix_fmt yuv420p -i - -crf 1 -c:v h264 -pix_fmt yuv420p -f flv rtmp://videoservice.com/live/secretkey

i will also add that due to storage limits on the new host, only the archives from 2013 Q4 and this year (2014) are on the podcast. this would be much different otherwise =]

and.. the timescale is non-linear! (because i like to fiddle)

Date | Hits | Bandwidth
6/04 | 20,034 | 2365.35 GB (2.3TB)
6/05 | 12,702 | 1528.78 GB (1.5TB)

T: 2 | 32,736 | 3894.13 GB (3.8TB)

and heres one using logs from a more well established instance of apache.

the goal:

use ffmpeg to stream audio from shoutcast and a video loop to ustream.tv, justin.tv, twitch.tv, youtube, wowza, flash media server, etc

because ffmpeg -loop 1 seems to only support images, export your loop video file as an image sequence (jpeg,png,etc)

so lets get started. in this example lets output to wowza (mpegts)

ffmpeg -i http://shoutcast.domain.com:port -re -loop 1 -pattern_type glob -i 'live/*.jpg' -c:v libx264 -b:v 3000k -pix_fmt yuvj420p -bsf h264_mp4toannexb -profile:v main -level 41 -g 20 -c:a libfdk_aac -b:a 192k -f mpegts udp://IP:10000?pkt_size=1316

breakdown

-r 30 will read the image sequence at 30 fps
-loop 1 loops a single image or image sequence
-pattern_type glob allows usage of *
-i '/path/to/loop00*.jpg' path to your image(s) sequence
-i http://shoutcast.domain.com:port host and port of shoutcast server
-f flv use filetype flv for output
rtmp://ustream.tv/url/key stream url and key provided by ustream.tv, justin.tv, twitch.tv, youtube, wowza, flash media server, etc

console output

ffmpeg version 2.1.3 Copyright (c) 2000-2013 the FFmpeg developers
built on Jan 21 2014 17:05:51 with gcc 4.6 (Ubuntu/Linaro 4.6.3-1ubuntu5)
configuration: --prefix=/usr/local --extra-ldflags=-L/usr/local/lib --enable-openssl --enable-openal --enable-libxvid --enable-libx264 --enable-libwavpack --enable-libvorbis --enable-libvo-aacenc --enable-libtheora --enable-libssh --enable-librtmp --enable-libopenjpeg --enable-libopencv --enable-libmp3lame --enable-avisynth --enable-libaacplus --enable-libbluray --enable-libfaac --enable-libfdk-aac --enable-avresample --enable-gray --enable-nonfree --enable-version3 --enable-gpl
libavutil 52. 48.101 / 52. 48.101
libavcodec 55. 39.101 / 55. 39.101
libavformat 55. 19.104 / 55. 19.104
libavdevice 55. 5.100 / 55. 5.100
libavfilter 3. 90.100 / 3. 90.100
libavresample 1. 1. 0 / 1. 1. 0
libswscale 2. 5.101 / 2. 5.101
libswresample 0. 17.104 / 0. 17.104
libpostproc 52. 3.100 / 52. 3.100
Input #0, mp3, from 'http://relay1.dubstep.fm:80':
Duration: N/A, start: 0.000000, bitrate: 192 kb/s
Stream #0:0: Audio: mp3, 44100 Hz, stereo, s16p, 192 kb/s
Input #1, image2, from 'live/*.jpg':
Duration: 00:00:24.00, start: 0.000000, bitrate: N/A
Stream #1:0: Video: mjpeg, yuvj420p(pc), 1280x720 [SAR 72:72 DAR 16:9], 25 fps, 25 tbr, 25 tbn, 25 tbc
[libx264 @ 0xa0b2960] using SAR=1/1
[libx264 @ 0xa0b2960] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 0xa0b2960] profile Main, level 4.1
Output #0, mpegts, to 'udp://IP:10000?pkt_size=1316':
Metadata:
encoder : Lavf55.19.104
Stream #0:0: Video: h264 (libx264), yuvj420p, 1280x720 [SAR 1:1 DAR 16:9], q=-1--1, 3000 kb/s, 90k tbn, 25 tbc
Stream #0:1: Audio: aac (libfdk_aac), 44100 Hz, stereo, s16, 192 kb/s
Stream mapping:
Stream #1:0 -> #0:0 (mjpeg -> libx264)
Stream #0:0 -> #0:1 (mp3 -> libfdk_aac)
Press [q] to stop, [?] for help
frame= 7482 fps= 25 q=12.0 size= 127133kB time=00:05:10.12 bitrate=3358.2kbits/s

to stream to ustream.tv just change to -f flv rtmp://ustream.tv/url/key and so on for the other services.

you may use any encoder you like as long as its supported by ustream.tv, justin.tv, twitch.tv, youtube, wowza, flash media server, etc

and for a little fun

when -loop 1 is set ffmpeg seems to read the file on every loop cycle, which means once ffmpeg is running and looping your image sequence, you can edit any of the individual image files and ffmpeg will stream the updated frames/images on the next loop cycle. this means you can edit/change individual frames in real time. (fight club lol)

if you want to use a single image file, for example -loop 1 -i image.png, you can then open with photoshop and make changes to layers, opacity, etc. just save the file and ffmpeg will stream it out!

note that if your using glob *.jpg for an image sequence, you can only make changes to files that were present at the time ffmpeg was executed. so just dropping another 100 images in the same directory and hoping that *.jpg will pick them up wont work, sorry :/

from what i can tell ffmpeg really hasnt been utilized in this particular way, but is good for radio station owners that want to take advantage of all available delivery methods.

(edit 04.19.14 – i have since bagged the still image, or looping video for something a bit sexier)

http://www.dsfm.tv

cheers =]

DL

Here is my remix of The Widdler – Ghost Train. Enjoy!

The reason project file for this tune has been changed beyond the point of recovering what this tune was… I rendered it only twice, so these are the only 2 versions of this track that will ever exist.

Render 1

Render 2

Download

01 – Ghosts
02 – Greyfold
03 – Butterflies (Widdler Remix)
04 – Tibet
05 – Inside
06 – The Moon
07 – Dojo Nights
08 – 1252
09 – Sleepless
10 – Listening To Space
11 – I Don’t Need You (DopeLabs Remix)
12 – Trentemoller – Moan Radio Slaves (Widdler Remix)
13 – Come Home
14 – December
15 – Lady Dub (VIP)
16 – Empty
17 – Surpass – Loosing You (Widdler Remix)
18 – Positive Vibes
19 – Grino
20 – Sia – Breathe Me (Widdler Remix)