Node.js Project to Add Proxy (IP Addresses) to HLS Live Video Streaming (.m3u8) Files in Browser Using Javascript

 

Basic Functionality:

  • proxy .m3u8 files, and the video segments (.ts files) they internally reference
  • to all proxied files:
    • add permissive CORS response headers
  • to .m3u8:
    • modify contents such that URLs in the playlist will also pass through the proxy

Advanced Features:

  • inject custom HTTP headers in all outbound proxied requests
  • prefetch video segments (.ts files)
  • use a hook function to conditionally decide which video segments to prefetch
  • use a hook function to conditionally redirect URLs in the playlist (before they’re modified to pass through the proxy)

Benefits:

  • any video player (on the LAN) can access the proxied video stream
    • including Chromecast
  • prefetch and caching of video segments ahead-of-time makes playback of the video stream very stable
    • solves buffering problems
  • the proxy can easily be configured to bypass many of the security measures used by video servers to restrict access:
    • CORS response headers (to XHR requests)
      • used by web browsers to enforce a security policy that limits which website(s) may access the content
    • HTTP request headers
      • Origin and Referer are often inspected by the server
        • when these headers don’t match the site hosting the content, a 403 Forbidden response is returned (in lieu of the requested data)
    • restricted access to encryption keys
      • often times the encrypted video segments (.ts files) are readily available, but the encryption keys are well protected
        • if the keys can be obtained from another source, then a hook function can be used to redirect only those URL requests

URL Format:

 

 

URL Format:

  • [example Javascript]: construction of URL to HLS Proxy for video stream
  • [example Javascript]: construction of URL to HLS Proxy for video stream w/ “Referer” request header
  • [example Bash]: construction of URL to HLS Proxy for video stream
  • [example Bash]: construction of URL to HLS Proxy for video stream w/ “Referer” request header
notes:
  • adding a file extension to the base64 encoded video URL is optional
    • doing so allows video clients to recognize that the URL being requested from HLS Proxy is an HLS video stream
high-level tools that automate this task:

Installation and Usage: Globally

How to: Install:

How to: Run the server(s):

Options:

  • –tls is a flag to start HTTPS proxy, rather than HTTP
    • used as shorthand to automatically configure the following options:
      • –tls-cert
      • –tls-key
      • –tls-pass
    • the values assigned to these options enable the use of a self-signed security certificate that is included in both the git repo and npm package, within the directory:
  • –host must be an IP address of the server on the LAN (so Chromecast can proxy requests through it)
    • ex: 192.168.0.100
    • used to modify URLs in .m3u8 files
    • when this option is not specified:
      • the list of available network addresses is determined
      • if there are none, ‘localhost’ is used silently
      • if there is only a single address on the LAN, it is used silently
      • if there are multiple addresses:
        • they are listed
        • a prompt asks the user to choose (the numeric index) of one
  • –port is the port number that the server listens on
    • ex: 8080
    • used to modify URLs in .m3u8 files
    • when this option is not specified:
      • HTTP proxy binds to: 80
      • HTTPS proxy binds to: 443
  • –req-headers is the filepath to a JSON data Object containing key:value pairs
    • each key is the name of an HTTP header to send in in every outbound request
  • –origin is the value of the corresponding HTTP request header
  • –referer is the value of the corresponding HTTP request header
  • –useragent is the value of the corresponding HTTP request header
  • –header is a single name:value pair
    • this option can be used multiple times to include several HTTP request headers
    • the pair can be written:
      • “name: value”
      • “name=value”
      • “name = value”
  • –req-options is the filepath to a JSON data Object
  • –req-insecure is a flag to override the following environment variable to disable certificate validation for secure https requests:
  • –req-secure-honor-server-cipher-order is a flag to set the following key in the request options Object to configure the context for secure https requests:
    • {honorCipherOrder: true}
  • –req-secure-ciphers is the value to assign to the following key in the request options Object to configure the context for secure https requests:
    • {ciphers: value}
  • –req-secure-protocol is the value to assign to the following key in the request options Object to configure the context for secure https requests:
    • {secureProtocol: value}
  • –req-secure-curve is the value to assign to the following key in the request options Object to configure the context for secure https requests:
    • {ecdhCurve: value}
  • –hooks is the filepath to a CommonJS module that exports a single JSON Object
    • each key is the name of a hook function
    • each value is the implementation of the corresponding Function
    • hook function signatures:
      • "redirect": (url) => new_url
        • conditionally redirect the URLs encountered in .m3u8 files before they are modified to pass through the proxy
      • "prefetch": (url) => boolean
        • conditionally decide whether to prefetch video segments on a per-URL basis
        • return value must be a strict boolean type (ie: true or false)
        • otherwise, the default behavior supersedes
          • to only prefetch .ts files
      • "prefetch_segments": (prefetch_urls, max_segments, is_vod, seg_duration_ms, perform_prefetch) => new_prefetch_urls
        • conditionally filter the list of video segment URLs that are pending prefetch, when more than –max-segments are contained in an HLS manifest
        • inputs:
          • prefetch_urls
            • array of string video segment URLs
          • max_segments
            • integer that denotes the max length of the return value
          • is_vod
            • boolean that indicates whether the HLS manifest is for video-on-demand
              • if true:
                • the video is not a live stream
                • the HLS manifest is complete and contains URLs for every video segment that would be needed to play the entire stream from start to finish
          • seg_duration_ms
            • integer that represents the duration (ms) of each video segment in the HLS manifest
          • perform_prefetch
            • function that accepts an array of string video segment URLs, and immediately begins to prefetch all corresponding segments
        • return value:
          • array of string video segment URLs that is a subset of prefetch_urls
            • can be emtpy (ex: when using perform_prefetch)
        • pre-conditions:
          • the length of prefetch_urls is > max_segments
        • post-conditions:
          • the length of the return value array is <= max_segments
  • –prefetch is a flag to enable the prefetch and caching of video segments
    • when .m3u8 files are downloaded and modified inflight, all of the URLs in the playlist are known
    • at this time, it is possible to prefetch the video segments (.ts files)
    • when the video segments (.ts files) are requested at a later time, the data is already cached (in memory) and can be returned immediately
  • –max-segments is the maximum number of video segments (.ts files) to hold in each cache
    • this option is only meaningful when –prefetch is enabled
    • a cache is created for each unique HLS manifest URL
      • all of the video segments (.ts files) associated with each distinct video stream are stored in isolation
    • when any cache grows larger than this size, the oldest data is removed to make room to store new data
    • when this option is not specified:
      • default value: 20
  • –cache-timeout is the maximum number of seconds that any segment cache can remain unused before its contents are cleared (to reduce wasted space in memory)
    • this option is only meaningful when –prefetch is enabled
    • when this option is not specified:
      • default value: 60
  • –cache-key sets the type of string used to represent keys in the cache hashtable when logged
    • this option is only meaningful when –prefetch is enabled
    • scope:
      • v0.16.0 and earlier
        • keys in the cache hashtable used this string representation
      • v0.16.1 and later
        • keys in the cache hashtable are full URLs
          • the data structure to cache video segments (.ts files) was updated
          • each unique HLS manifest is associated with a distinct FIFO list that holds –max-segments
          • when a video segment is requested
            • the proxy needs to search every FIFO list for a match
            • when keys in the cache hashtable lose fidelity, collisions can occur and the wrong video segment can be returned
            • full URLs are unique and guarantee correct behavior
    • 0 (default):
      • sequence number of .ts file w/ .ts file extension (ex: “123.ts”)
        • pros:
          • shortest type of string
          • makes the log output easiest to read
        • cons:
          • in the wild, I’ve encountered video servers that assign each .ts file a unique filename that always terminate with the same static sequence number
            • this is a strange edge case, but this option provides an easy workaround
    • 1:
      • full filename of .ts file
    • 2:
      • full URL of .ts file
  • -v sets logging verbosity level:
    • -1:
      • silent
    • 0 (default):
      • show errors only
    • 1:
      • show an informative amount of information
    • 2:
      • show technical details
    • 3:
      • show an enhanced technical trace (useful while debugging unexpected behavior)
  • –acl-whitelist restricts proxy server access to clients at IP addresses in whitelist
    • ex: "192.168.1.100,192.168.1.101,192.168.1.102"
  • –tls-cert is the filepath to a security certificate to use for HTTPS
  • –tls-key is the filepath to the private key for the –tls-cert security certificate
  • –tls-pass is the filepath to a text file containing the security passphrase for the –tls-key private key
    • optional, not required when the –tls-key private key was created without a security passphrase

Examples:

  1. print help
    hlsd --help

  2. print version
    hlsd --version

  3. start HTTP proxy at default host:port
    hlsd

  4. start HTTP proxy at default host and specific port
    hlsd --port "8080"

  5. start HTTP proxy at specific host:port
    hlsd --host "192.168.0.100" --port "8080"

  6. start HTTPS proxy at default host:port
    hlsd --tls

  7. start HTTPS proxy at specific host:port
    hlsd --tls --host "192.168.0.100" --port "8081"

  8. start HTTPS proxy at default host:port and send specific HTTP headers
    hlsd --tls --req-headers "/path/to/request/headers.json"

  9. start HTTPS proxy at default host:port and enable prefetch of 10 video segments
    hlsd --tls --prefetch --max-segments 10

  10. start HTTPS proxy using a non-generic security certificate
    hlsd --tls-cert "/path/to/cert.pem" --tls-key "/path/to/key.pem" --tls-pass "/path/to/pass.phrase"


Installation and Usage: Working with a Local git Repo

How to: Install:

How to: Run the server(s):

Options:

Examples:

  1. print help
    npm start -- --help

  2. start HTTP proxy at specific host:port
    npm start -- --host "192.168.0.100" --port "8080"

  3. start HTTPS proxy at specific host:port
    npm start -- --host "192.168.0.100" --port "8081" --tls

  4. start HTTP proxy at default host:port with escalated privilege
    npm run sudo -- --port "80"

  5. start HTTPS proxy at default host:port with escalated privilege
    npm run sudo -- --port "443" --tls

  6. start HTTP proxy at specific port and send custom “Referer” request header for specific video stream

  1. start HTTP proxy at specific port and send custom request headers
  1. start HTTPS proxy at specific port and send custom request headers
  1. start HTTPS proxy at specific port and send custom request headers

Observations:

  • when playing the proxied HLS video stream in an HTML5 player in a Chromium web browser (ex: THEOplayer)
    • if the page hosting the HTML5 video player is served from HTTPS:
      • when running only the HTTP proxy server:
        • the XHR requests from the player to the HTTP proxy server raise a security warning (insecure content)
        • the XHR requests get elevated to HTTPS, which are unanswered (since the HTTPS proxy server isn’t running)
      • when running only the HTTPS proxy server:
        • the XHR requests from the player to the HTTPS proxy server will silently fail
        • this is because the HTTPS proxy server is using a self-signed security certificate
        • this certificate needs to be (temporarily) allowed
        • once it is, the video stream works perfectly
          • to allow the certificate:
            • browse to a URL hosted by the proxy server ( example )
            • you should see the warning: NET::ERR_CERT_AUTHORITY_INVALID Your connection is not private
            • click: Advanced
            • click: Proceed to 127.0.0.1 (unsafe)
            • done
  • when playing the proxied HLS video stream on a Chromecast
    • the HTTP proxy server works perfectly
    • the HTTPS proxy server doesn’t begin playback
      • not sure why..
      • probably has something to do with the Chromecast’s browser security policies
      • a more respectable security certificate (ie: more expensive) would probably fix it

Summary of (Rarely) Observed OpenSSL Connection Errors:

  • error:
    ssl3_check_cert_and_algorithm:dh key too small

    1. attempted fix:
      --req-secure-ciphers "AES128-SHA"
  • error:
    SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

    1. attempted fix:
      --req-secure-protocol "SSLv3_method"

      • result:
        Error: SSLv3 methods disabled
      • issue:
    2. attempted fix:
      --req-secure-curve "auto"

Other Projects:

(directly related, but very loosely coupled)
  • Webcast-Reloaded
    • consists of 2 parts:
      1. a Chromium web browser extension (.crx)
        • on each browser tab, it’s silently watching the URL of all outbound requests
        • every requested URL matching a regex pattern that identifies it to be a video file is displayed in the modal window that toggles open when the extension’s icon is clicked
        • links in this modal window open to URLs of component #2
      2. a static website
        • there is a selection of several HTML5 videos players
          • each is better at some things and worse at others
          • each integrates with a different Chromecast receiver app
        • there is a page to help redirect the intercepted video URL through a local instance of HLS Proxy

Other Projects:

(unrelated, but somewhat similar in scope and purpose)
  • Streamlink
    • notes:
      • this project has way more features, and is way more polished
      • though its main purpose is to transcode online video with ffmpeg and pipe the output into another program, it can be configured to not load a video player and instead start a web server
      • it can strongly support individual websites through single-purpose plugins
      • it can also support streams via direct URLs
        • using URLs from the wild will have mixed results, since cookies and headers and authentication aren’t being managed by any plugin
    • docs:
    • binaries:
      • Windows portable
        • minimum system requirements:
          • Windows 7 SP1
          • .NET Framework 4.5
    • usage test:
      • streamlink --player-external-http --player-external-http-port 8080 --default-stream best --http-ignore-env --http-no-ssl-verify --url "https://XXX/video.m3u8"
    • usage test result:

Legal:

 

https://github.com/warren-bank/HLS-Proxy

 

Leave a Reply